Revamping Code Completion with little changes

[rickg22]

I got an idea that might just work

The reason we're having so many problems with code completion under linux is because of the parserthread class, right?

That's because the ParserThread carries all the information to parse a file.

Well, my idea is to SEPARATE this "parsing information" from the thread, encapsulating it into a single object: ParserJob. The parserthread will be just a container.

This way, if we want to parse a small string, we don't need to create a new thread to parse. We'll just create a ParserJob, and ask it to parse.

Furthermore, with this model, we can create the Parser jobs in a queue ( ParseQueue ) and we'll feed them to whatever ParserThread is currently running. To handle an include, it'll just create a new parser job and add it to the queue (no meddling with wxwidgets events!  ). And when the ParserThread finishes parsing, it just searches for a job which will parse the next file.

This way, we don't have to create 100 threads and delete them! (I've ran it under linux gdb, the threads aren't deleted instanty, so you can imagine)

And I bet parsing will be a lot faster!

What do you think guys? Yiannis?

[mandrav]

Sounds nice

Yiannis.

[me22]

How about we have a user-settable number of ParserThreads?  This is actually a concurrant-able task, so SMP machines should be able to take advantage of that...

[grv575]

If you're going to go this route (which I think is a good idea), then why not use thread pools?  The way this works is you hand a job/task to a threadpool object which handles allocating a new thread if needed or using an existing thread if enough are already allocated.  So you just tell the pool to execute tasks.  It scales well because the pool uses a configurable limit on the # of threads in the pool.  So once this limit is reached, it queues the task and tasks in the queue run on existing threads in that case.  Anyone know if thread pools are part of standard c++ libs or what would need to be included/written?  But this is the standard/clean way to do this type of thing these days.  That's about all I know about the subject, so... anyone know more about using them (have only used thread pools in .net).

[mandrav]

If you're going to go this route (which I think is a good idea), then why not use thread pools?  The way this works is you hand a job/task to a threadpool object which handles allocating a new thread if needed or using an existing thread if enough are already allocated.  So you just tell the pool to execute tasks.  It scales well because the pool uses a configurable limit on the # of threads in the pool.  So once this limit is reached, it queues the task and tasks in the queue run on existing threads in that case.  Anyone know if thread pools are part of standard c++ libs or what would need to be included/written?  But this is the standard/clean way to do this type of thing these days.  That's about all I know about the subject, so... anyone know more about using them (have only used thread pools in .net).

This is what it currently does, sort of.
There is a stack (or list - can't recall now) where each new thread is put into.
After that, when each thread ends, it pops as many threads from the stack as needed to reach the number of allowed running threads.

Yiannis.

[grv575]

Heh, but that's like exhausting resources first, and then trying to fix the damage later.

[mandrav]

Right, hence this thread

Yiannis.

[zieQ]

Oh yes, I want a true wxwidget thread pool implementation in C::B core, so as to use it for compiler engine too :roll: (it will speed up compilation with gcc for multiprocs machines as asked in another thread)

Here is a java implementation that we could inspire from:
http://www.informit.com/articles/article.asp?p=30483&seqNum=6

[zieQ]

And here is a helpful discussion on performance:
www.scl.ameslab.gov/Publications/ Brett/CCCTFinal-color.pdf

[mandrav]

OK, ok

I will add a thread-pool implementation in the core SDK.

Yiannis.

[rickg22]

Just... be careful...

Anyway you might like subclassing my managedthread class for the threadpool. This way we can make all the threads abort immediately (almost) by setting a single flag.

[zieQ]

Humm forget what I have said: no thread poll for compilation since wxExecute deals with processes and not threads. However, a thread pool could be useful. Be careful, it should be a singleton (one instance).

[mandrav]

However, a thread pool could be useful. Be careful, it should be a singleton (one instance).

Why a singleton? You could have many different pools (if needed). Why restrict yourself to just one global pool?
One pool for code-completion, one for compiler, etc.

And the compiler could use thread pooling. Each wxExecute could be executed in a thread pool worker class...

Yiannis.

[rickg22]

And don't forget Search-in-files and TODO list!

OK, but if the code completion is going to use the thread pool, how long will I have to wait before the pool's ready so I can make the changes to CC? Or should I begin the changes and later we can switch to using the pool?

[mandrav]

And don't forget Search-in-files and TODO list!

OK, but if the code completion is going to use the thread pool, how long will I have to wait before the pool's ready so I can make the changes to CC? Or should I begin the changes and later we can switch to using the pool?

I said I 'm doing it. Stop being impatient
I guess it 'll take a day or two.

Yiannis.