Parallel builds

[IvanBatrakov]

Hello developers,

function int CompilerGCC::DoRunQueue() is not parallel due to it is Queue
and having QUAD CPU it still uses only ONE CPU.

There is workspace with 4 or more projects. Needed to run build of workspace parallelly, not single process with single queue as it is now.

Needed true Parallel builds.

[oBFusCATed]

Patches welcome

I've some ideas, but I've no time for implementing them, not enough desire

[IvanBatrakov]

needed info on how to create patches

(development environment: Ubuntu 10.04, Editor: CodeBlocks compiling CodeBlocks to become perfect)

[Jenna]

Did you try to increase the number of parallel builds in "Settings... -> Compiler and debugger -> Build options -> Number of processe for parallel builds" ?
In older versions of C::B it was on the "Other options"-tab instead of the "Build options"-tab.

[Alpha]

needed info on how to create patches

Creating a patch.

[IvanBatrakov]

yes, number in Settings was changed to 1000, it helps to make parallel build of ONE project in workspace but other projects have to wait first project in queue.

So 3 CPU cores are sleeping at 0% CPU usage almost constantly.

All projects are without dependencies between them, only common files used like header files included.

Also dependencies of header files included to project even shown in file tree in project do not work at all.
As solution manually change .CPP file in project then build is ok.

[Jenna]

I use 3 processes on my core2duo and both processors are used to compile.

[IvanBatrakov]

Quantity of projects in workspace helps to understand situation.

Yes, parallel build works for ONE project in workspace but other projects are waiting ONE build to be ended.

It is queue, not parallel build and code in function int CompilerGCC::DoRunQueue() shows it.

[thomas]

Ok, but who cares? The point in doing parallel builds is reducing build times. The current "non-parallel" parallel build does that  by running more than one compiler instance on a build-target scope. It works, and it is robust.

A "true parallel" build will likely not be any faster, and possibly be slower due to caching effects, since unrelated files are compiled concurrently and will less likely share the same headers and their object files will be less likely found in caches at link time. It will also be less robust.

[Alpha]

A "true parallel" build will likely not be any faster, and possibly be slower due to caching effects, since unrelated files are compiled concurrently and will less likely share the same headers and their object files will be less likely found in caches at link time.

There is a situation in which I would find this method parallel compiling to be faster.  When I have tested various modifications to the core of a program (for example, Code::Blocks), many of the extra plugins must be re-linked to the sdk, but not necessarily re-compiled (if they do not include a modified header).  When running on multiple cores, this results in only a single core being used for the majority of the time.

[thomas]

Have you ever looked at what happens during linking? It's 99% seeking and waiting for IO, and the CPU hardly ever gets to do something. Running several links in parallel is pouring oil into the fire.

[oBFusCATed]

A "true parallel" build will likely not be any faster, and possibly be slower due to caching effects, since unrelated files are compiled concurrently and will less likely share the same headers and their object files will be less likely found in caches at link time. It will also be less robust.

It will, because linking c++ projects is slow and single core and during linking time, some object files could be complied on the other cores.
And c++ compilation is not IO bound. Someone should play with it and see if there is any benefit.

There is a situation in which I would find this method parallel compiling to be faster.  When I have tested various modifications to the core of a program (for example, Code::Blocks), many of the extra plugins must be re-linked to the sdk, but not necessarily re-compiled (if they do not include a modified header).  When running on multiple cores, this results in only a single core being used for the majority of the time.

I doubt parallel linking will help here. What we should do is remove the external link dependencies in the C::B project, because they are not needed.

[Alpha]

I doubt parallel linking will help here. What we should do is remove the external link dependencies in the C::B project, because they are not needed.

You are correct here, however, I had been giving Code::Blocks as an example.  If I am linking with a static library and have only changed the implementation of a function (in the static library), a re-link is required.

[IvanBatrakov]

Migrating from Windows to Linux platform needed replacement of Visual Stuido to CodeBlocks.

Visual Stuido builds the same workspace at least 10-20 times faster.

Already made changes to CodeBlocks editor behaivour similar to Visual Studio and even more features to work comfortable and faster.

But without parallel builds migration to CodeBlocks is impossible due to slow down development.

[thomas]

Well, feel free to submit a patch. I'm just telling you that I don't believe making it more parallel will make anything faster, except maybe in some very contrieved example cases. But again, feel free to experiment. If you really manage to make my builds faster, I will not complain.

Visual Studio compiles a lot faster because MSVC is a much faster (and lower quality) compiler than gcc. This is more true under Windows than under Linux too, because the GNU-style absurd number of process creations is not very efficient. It is not uncommon that for compiling a single source, a chain of 5 or 6 executables is launched. This somewhat less of a problem under Linux, since it is not that painfully devastative, but it is much more disturbing under Windows. Visual Studio only ever loads the compiler once, for the entire workspace.

Also, in particular GNU ld is the most terrible thing on earth, which is why link times often dominate the build process by far.

Though you can use an alternative linker, which is even officially supported via plugins in the mean time, if that is a problem. There exist some which are an order of magnitude faster, too.

On my development systems, compile times under Windows are bound by both process creation and IO. Under Linux, they are bound by IO alone. No matter how many parallel builds I start (the sweet spot is 5 for me), CPU usage never comes even close to 100%.
Linking using ld literally causes millions of seeks on moderately large projects. Solid state and ram drives help, but do not make the problem go away. As such, IO is a dominating factor, and my guess is that adding more seeks does not improve the situation.