CC based semantic highlight

[Alpha]

I have begun exploration of full semantic highlighting, provided by the knowledge from CC's parser.  The attached patch is able to show colors, however it is very temperamental.  Also, if you type anything into the editor, it will break.
The colors were chosen at random; suggestions for these would be appreciated.

This code currently uses some operations which are potentially unsafe.  You have been warned.

[ollydbg]

Great work. Look at the screen shot, it is so beautiful.
I'm going to test it right now.

[ollydbg]

Testing result: sometimes, C::B hangs.

[mistar]

Have a look also on my plugin shouldn't hang
http://forums.next.codeblocks.org/index.php/topic,16249.msg120205.html#msg120205

[ollydbg]

Have a look also on my plugin shouldn't hang
http://forums.next.codeblocks.org/index.php/topic,16249.msg120205.html#msg120205

Both you and Alpha did good job, thanks, will take time to test those.

@Alpha, about the hang, I think the locker(there are many lockers in the CC code), but from my point of view, we should remove mostly of them.

TokenTree are the shared resource, sometimes, the worker thread (Parserthread are adding tokens to it), in the other time, the main GUI thread(like the one you implement the semantic highlight should query the TokenTree). I think they should not happen concurrently. I mean if the parser is working, the TokenTree should not accessed by main thread. Either main GUI thread or worker thread can access the TokenTree.

[p2rkw]

I see it's in trunk now: http://sourceforge.net/p/codeblocks/code/8920/
but as I can see it's impossible to disable this feature.
My proposition is to detect whenever third keyword set have exactly the same colour as default keyword set: if those colours are equal then all token processing inside CodeCompletion::UpdateEditorSyntax is unnecessary.

[oBFusCATed]

Alpha:
Do you plan to fix this issue:

Code Select Expand


class Test
{
    void func()
    {
        int a = var;
    }
    int var;
};

void func()
{
    int var;
}

Hint: the issue is that the var is highlighted everywhere, but it shouldn't inside the func.

If not please revert your patch!

[Alpha]

The one that is in trunk is actually this.  But I will add an enable/disable checkbox now.

Do you plan to fix this issue: [...]

Impossible with the current method of implementation (see other thread).  Just disable it, if it bothers you, when I get the checkbox uploaded.

[oBFusCATed]

Impossible with the current method of implementation (see other thread).  Just disable it, if it bothers you, when I get the checkbox uploaded.

I'll just revert it, don't worry. There is no point in having a fake feature in trunk. If you want it in do it properly, please.

@other devs: please state your opinion here, I won't revert if for a couple of days.

[ollydbg]

C++ is too complex and its grammar is context dependant, the way Alpha use was set the identifier to a group, then let the scintilla to colorize the group in the same color. I think we can follow what the way clang sematic highlight plugin use, I mean write our own lexer for scintilla, colorize each piece of text by the token type. The token type info can be queryed from the tokentree. But this may still cause error, because our parser does not do sematic check.

EDIT: It looks that it is impossible to query every text info from the tokentree, our tokentree only hold the variable definition info, but for other variable usage(reference), it need the AI() function, this will make the custom lexer much slower.

[MortenMacFly]

@other devs: please state your opinion here, I won't revert if for a couple of days.

Well I like and in fact use it although I realise its not working 100%. Maybe this should be turned off by default and an explanation in the settings points out that this is not meant to work perfectly.

[oBFusCATed]

Well I like and in fact use it although I realise its not working 100%. Maybe this should be turned off by default and an explanation in the settings points out that this is not meant to work perfectly.

For sure it should be turned off by default if it stays.
But the problem is that this is a half feature - broken and hard to extend.

The thing is that the semantic part of the name of the feature implies that it uses the CC and so it should work 100% of the time (ignoring CC parser bugs).
But this is not the case. And we have a prototype (I've not looked at it, so I cannot tell how good it is) that does better job.

Also this feature is almost the same as the broken highlight for stl stuff, but the brokenness is a bit more hidden.

I know that implementing this properly is hard, because of the need to modify scintilla, but I'm sure it is doable.
Also the semantic highlight should not be part of the CC plugin! It should be part of the SDK or it should be separated in a plugin.

[Alpha]

It is off by default now.  If you would like to revert it completely, go ahead, however from previous discussions of this patch, it sounded as though there are at least several people who do find it useful in its current (flawed) state.

Also the semantic highlight should not be part of the CC plugin! It should be part of the SDK or it should be separated in a plugin.

Agreed, but do we yet have a way for a seperate plugin to access the data in CC's parser?

[oBFusCATed]

It is off by default now.  If you would like to revert it completely, go ahead, however from previous discussions of this patch, it sounded as though there are at least several
people who do find it useful in its current (flawed) state.

OK, but what do you do when they ask you to highlight the member functions differently and then the local variables?

Agreed, but do we yet have a way for a seperate plugin to access the data in CC's parser?

Nope, but we should start from somewhere otherwise we will never have it.

BTW: The current implementation is quite laggy and thus quite annoying. Something happens but I don't know when.

p.s. Anyone else from the devs which wants to express opinion. Currently we are 1:1.

[MortenMacFly]

p.s. Anyone else from the devs which wants to express opinion. Currently we are 1:1.

Express opinion on what exactly?

For semantic highlight and access to CC internals my preferences are to use clang, maybe through Eran's engine. I think we can never be better than that - its amazing how fast the code base and functional support grows. So having a nice abstraction layer for clang will serve way more than providing interfaces to access (our) CC internals IMHO...