CC: Expensive calls on opening new files

[dmoore]

When opening a new file, the following gets called:

Code Select Expand


void CodeCompletion::OnEditorActivated(CodeBlocksEvent& event)
{
    EditorBase* eb = event.GetEditor();
    if (IsAttached() && m_InitDone)
    {
        m_NativeParsers.OnEditorActivated(eb);
        m_FunctionsParsingTimer.Start(1000, wxTIMER_ONE_SHOT); // one second delay should be ok
    }

    event.Skip();
}


which eventually results in this call

Code Select Expand


bool Parser::ParseBufferForFunctions(const wxString& buffer)
{
    ParserThreadOptions opts;
    opts.wantPreprocessor = m_Options.wantPreprocessor;
    opts.useBuffer = true;
    opts.bufferSkipBlocks = true;
    opts.handleFunctions = true;
    ParserThread* thread = new ParserThread(this,
                                            buffer,
                                            false,
                                            opts,
                                            m_pTempTokens);
    return thread->Parse();
}


on the main thread??

this is really time consuming for big files...


also note this hook gets called hundreds if not thousands of times when opening a file (should shut down the event propagation in the sdk until the text has been inserted?):

Code Select Expand


void CodeCompletion::EditorEventHook(cbEditor* editor, wxScintillaEvent& event)
{
    ConfigManager* cfg = Manager::Get()->GetConfigManager(_T("code_completion")); //EXPENSIVE??

    if (!IsAttached() ||
        !m_InitDone)// ||
        !cfg->ReadBool(_T("/use_code_completion"), true))  //cfg-Read... EXPENSIVE??
    {
        event.Skip();
        return;
    }
...


[dmoore]

I patched cbeditor to restrict editorhook calls during file opens. CC reports only ~ 200 events with patch (vs ~700,000 unpatched) when loading a 9mb file. patch: http://developer.berlios.de/patch/index.php?func=detailpatch&patch_id=2072&group_id=5358

this roughly halves the time until you see the editor spring up (at which time function parsing begins on the main thread - i haven't attempted to fix this...)

[dmoore]

PS: please move if this doesn't belong here (the thread stuff is more relevant to this sub-forum, perhaps)

[rickg22]

dmoore: Please share more! I need to know everything.

OK, first of all: The "m_FunctionsParsingTimer.Start(1000, wxTIMER_ONE_SHOT);" was added so Code Completion would wait at least some time before starting to parse. If a file was added to the project, the timer would be reset and the countdown would start from -1000 ms again. When all files had finished opening, the timer would advance to 0 and parsing would begin. So yes, basically it was a way to delay the parsing until the project had been finished.

However, I didn't know anything about the hooks. What tests exactly did you do to realize this was called thousands of times?

Update: I added a debug log to cbEditor::OnScintillaEvent(wxScintillaEvent& event). It displays a message along with a cumulative number. The number i got was 17242 effective calls during project opening. You're right.

But I got a better solution than your patch. It's called ProjectManager::IsBusy(). Returns true when the project is loading/closing.

[rickg22]

What do you know, it works! It only gets 17 effective calls during project opening.

I'll do something similar to code compleition's OnEditorActivated. Hats off for your discovery.

Update: Guess what, biplab had already fixed that part. I'm just cleaning it up.

[dmoore]

yes, please clean up as you see fit (would it be possible to switch off the event responder altogether the save the function calls?)

unfortunately, I can't say I used any special tool to find the problem other than C::B itself (lots of find in files and find implementation of). I know that on Linux you can use callgrind (part of valgrind) to produce call statistics, but I'm not familiar enough with that tool or the C::B code to use it effectively (you have to have an idea of what code should be called thousands of times and what shouldn't).

The main reason I was able find the problem came from playing around with big files and from recent stuff I learned while playing with scintilla (adding the php lexer and my wild goose chase with wxIdleEvents). Scintilla is very event driven, so it was just a case of figuring out which events were creating lags or conflicts... there may still be more issues that callgrind might be able to unearth.

It also helps that you guys have writen very clear modular code! (despite wxwidgets macro heavy implementation)

back to the Function Parsing code - delaying till all project files are open is good, but then it appears that the parsing occurs on the main thread, which blocks the UI until it's done, which kind of defeats the purpose of waiting...

and next problem: after the editor loses then regains focus, the function parser appears to reparse the whole file if you click on the class/function browser drop down. should only happen if file is modified right? (unless the data is released when the file loses focus?)

[Biplab]

unfortunately, I can't say I used any special tool to find the problem other than C::B itself (lots of find in files and find implementation of). I know that on Linux you can use callgrind (part of valgrind) to produce call statistics, but I'm not familiar enough with that tool or the C::B code to use it effectively (you have to have an idea of what code should be called thousands of times and what shouldn't).

I'll post a Callgrind log tonight (Now it's morning for me), too, if I can set it up properly.

back to the Function Parsing code - delaying till all project files are open is good, but then it appears that the parsing occurs on the main thread, which blocks the UI until it's done, which kind of defeats the purpose of waiting...

and next problem: after the editor loses then regains focus, the function parser appears to reparse the whole file if you click on the class/function browser drop down. should only happen if file is modified right? (unless the data is released when the file loses focus?)

These problems needs attention.

[rickg22]

back to the Function Parsing code - delaying till all project files are open is good, but then it appears that the parsing occurs on the main thread, which blocks the UI until it's done, which kind of defeats the purpose of waiting...

Actually the parsing isn't that slow. It's just one file. What makes it so slow is the event handling related to it. But if you wish, we can [ EDIT: at a *LATER* stage*, hope that clears it up, Don Corleone ;-) ] fix that, too. Volunteers? (I have thread-phobia )

and next problem: after the editor loses then regains focus, the function parser appears to reparse the whole file if you click on the class/function browser drop down. should only happen if file is modified right? (unless the data is released when the file loses focus?)

Yes, I noticed that. Curiously, this was the same redundant behavior that I *just* fixed in TODO list. So I'm changing it so it will parse *ALL* the open files at once, and just switch from the results whenever you change the active file.

[mandrav]

I must admit I 'm kind of scared with some of the things I read here...
I 'm not saying you don't have a point: you may very well be right.

BUT...

I 'm afraid of such optimizations at such a late stage by people who are not very familiar with the particular code. And don't take this as an insult please. You all remember the time when CC was responsible for every bad thing in C::B. We don't want to go back there, do we?

All I 'm saying is that if you want to "play" around with the code and try to optimize things you 're very welcome to do so. But in a separate branch of the code. I don't want the risk of ruining what we already have, even if it's not performing optimally.

So, if it's a new branch you need created, just ask me and I will create one for you.
Thanks again for your efforts .

[rickg22]

Yiannis: Don't worry about the thread problems! I'm not touching THAT thing (eew :-P ). It was just a suggestion, I have to admit sometimes I get misinterpreted (ah, remember the old times? ). I said "we" as "the team" because I don't want to meddle with multithreading. I still have open sores about that  ;-)

However, about prefetching the function lists, it's a simple change, I already tried in the todo list, and it worked wonders (that was BEFORE i had discovered the huge memory leak).

Rest assured, I'm not letting the multithreaded creature go rampant again  :lol:

[dmoore]

rick: let me know when you have finished making your planned changes (if you make them). after that I might release the multithreaded beast again in my local copy  :lol:

[David Perfors]

Yiannis: Don't worry about the thread problems! I'm not touching THAT thing (eew :-P ). It was just a suggestion, I have to admit sometimes I get misinterpreted (ah, remember the old times? ). I said "we" as "the team" because I don't want to meddle with multithreading. I still have open sores about that  ;-)

Yes I remember the old times :lol: but it seems that you have learned a little bit of it (How do I know? you have just edited you're own text )

[rickg22]

[

back to the Function Parsing code - delaying till all project files are open is good, but then it appears that the parsing occurs on the main thread, which blocks the UI until it's done, which kind of defeats the purpose of waiting...

and next problem: after the editor loses then regains focus, the function parser appears to reparse the whole file if you click on the class/function browser drop down. should only happen if file is modified right? (unless the data is released when the file loses focus?)

These problems needs attention.

Good news everybody, I patched that part. Now you can switch files as fast as you want, and the files won't get re-parsed.
It's in revision 4163.

[dmoore]

good work rick. 

[dmoore]

But I got a better solution than your patch. It's called ProjectManager::IsBusy(). Returns true when the project is loading/closing.

what about opening a file when no project is loaded? I assume that projectmanager won't be IsBusy() during that time? (could be wrong about this)