Skip to content

Enable parallel builds by default.#70

Merged
thatch merged 1 commit intocodrsquad:mainfrom
thatch:thatch/parallel-build
Mar 4, 2026
Merged

Enable parallel builds by default.#70
thatch merged 1 commit intocodrsquad:mainfrom
thatch:thatch/parallel-build

Conversation

@thatch
Copy link
Member

@thatch thatch commented Mar 2, 2026

This roughly halves the time taken to build on a 12-core MBP to make it easier to iterate. I picked 3.10 arbitrarily but this affects all make invocations by default. One can override it with cpu_count=1 for a particular call if there are problems.

before change

INFO Compiling cpython took 6 minutes 26 seconds
INFO Tar build/cpython-3.10.19/ppp-marker/3.10.19 -> dist/cpython-3.10.19-macos-arm64.tar.gz
INFO Overall compilation took 9 minutes 56 seconds
portable-python build 3.10.19  321.52s user 120.27s system 74% cpu 9:56.96 total

after change

INFO Compiling cpython took 3 minutes 42 seconds
INFO Tar build/cpython-3.10.19/ppp-marker/3.10.19 -> dist/cpython-3.10.19-macos-arm64.tar.gz
INFO Overall compilation took 4 minutes 51 seconds
.venv/bin/portable-python build 3.10.19  327.72s user 141.55s system 160% cpu 4:52.06 total

@thatch thatch requested a review from zsimic March 2, 2026 14:35
@thatch
Copy link
Member Author

thatch commented Mar 2, 2026

The more I think about it, -j0 is nonsensical; why don't we make that the "auto" default, and forbid negative numbers entirely? I'm not sure when I'd ever need almost-all-the-cores.

Copy link
Contributor

@zsimic zsimic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice speed up!

@thatch
Copy link
Member Author

thatch commented Mar 4, 2026

More results, this time GCC on Linux. A big chunk of the time is spent on two link steps, so most of the win comes from being able to do those two concurrently.

Before After Speedup
12-thread Xeon (HT) - Archlinux 29m56s 11m46s 2.54x
12-thread Xeon (HT) - Ubuntu 22m42s 10m6s 2.25X
40-thread Xeon (HT) - Archlinux 31m47s 12m16s 2.59x

@zsimic
Copy link
Contributor

zsimic commented Mar 4, 2026

Excellent bump! Let's merge it. We can also change/review this to be "use all CPUs" by default, with just a boolean to turn it off as well... You're right, the granularity on how many CPUs to use is not really needed, it was just mimicking -j in a way. So we could change this from cpu_count=None to parallel=True, I'm down for the simplification if you think it'll help future readers.

@thatch thatch force-pushed the thatch/parallel-build branch from 7cf9feb to 763b4fa Compare March 4, 2026 15:32
@thatch thatch merged commit 69e3a93 into codrsquad:main Mar 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants