tl;dr: comparison tables
|runner||cpu||OS||build||no cache install||cached install|
|MoeSzyslak||Intel i9 (2019, 2.3 Ghz 8-Core)||MacOS||19.7s||20.4s||6.7s|
|runner||cpu||OS||production build||uncached production build||cached install|
|MoeSzyslak||Intel i9 (2019, 2.3 Ghz 8-Core)||MacOS||32.2s||30s||12.6s|
Because folks have been thinking about upgrading to an M1 Max Mac OS machine for development, I wanted to share some of my own benchmarks as I don’t use MacOS, and wanted to show folks that there are cheaper alternatives to the Apple Ecosystem, if you’re after speed (hopefully – depending on what this thread brings to light!).
So, I’m going to be testing with two OSS projects:
And what we’ll be measuring is:
- install hyperfine
- disable scripts (you should do this anyway for security)
These settings also effect what npm/yarn/etc do after installing deps, which can dramatically affect results
npm config set ignore-scripts=true
yarn config set ignore-scripts true
NOTE: all of these are “high level” benchmarks, involving the whole system, CPU, RAM, disk, etc.
- RAM: 32GB
- CPU: 11th Gen Intel Core i5-1135G7 @ 2.4GHz × 8
- OS: Ubuntu 22.04
- Disk: WD Black SN850 NVMe
Make sure you have your system monitor utility open and verify that you’re not hitting any sort of RAM limit or are running out of available CPUs usage. Hitting swap, or being thrown in some de-prioritized CPU queue can greatly negatively affect the results.
(I had to re-do some tests due to hitting swap )
If you only have 32GB of RAM, make sure you have at least 10GB free to run these benches (to safely avoid hitting swap)
git clone https://github.com/emberjs/ember.js.git cd ember.js git checkout 55ffe6326
❯ hyperfine --runs 5 \ --setup 'yarn && yarn build' \ 'yarn build' # .......... Benchmark 1: yarn build Time (mean ± σ): 15.713 s ± 0.956 s [User: 24.479 s, System: 1.805 s] Range (min … max): 14.817 s … 16.831 s 5 runs
the initial build (done during setup), builds out a cache for the build, so all subsequent builds, and what this is measuring is mostly a “no-op” build where we determine that nothing has changed and all parts of the build should be 100% cache-hits.
❯ hyperfine --runs 5 \ --prepare 'yarn cache clean && rm -r node_modules' \ 'yarn install' # .......... Benchmark 1: yarn install Time (mean ± σ): 22.225 s ± 0.362 s [User: 20.772 s, System: 15.475 s] Range (min … max): 21.858 s … 22.771 s 5 runs
❯ hyperfine --runs 5 \ --setup 'yarn' \ --prepare 'rm -r node_modules' \ 'yarn install' # .......... Benchmark 1: yarn install Time (mean ± σ): 5.829 s ± 1.072 s [User: 6.563 s, System: 6.748 s] Range (min … max): 4.685 s … 6.815 s 5 runs
git clone https://github.com/NullVoxPopuli/limber.git cd limber git checkout 6acac0f
This projects uses turbo to aggressively cache, so we’ll be crafting the commands to avoid turbo’s cache, and focusing on the build of the “app”. For building the app, the setup phase will build the dependent libraries, and the time spent on those is omitted from the measurements.
❯ hyperfine --runs 5 \ --setup 'pnpm install && pnpm build' \ 'pnpm run --filter=limber build' # .......... Benchmark 1: pnpm run --filter=limber build Time (mean ± σ): 26.891 s ± 2.141 s [User: 34.278 s, System: 2.934 s] Range (min … max): 24.883 s … 30.121 s 5 runs
because this app uses embroider, we can delete
/tmp/embroider to get a less-cache build
❯ hyperfine --runs 5 \ --setup 'pnpm install && pnpm build' \ --prepare 'rm -r /tmp/embroider' \ 'pnpm run --filter=limber build' # .......... Benchmark 1: pnpm run --filter=limber build Time (mean ± σ): 28.967 s ± 2.583 s [User: 34.861 s, System: 2.869 s] Range (min … max): 26.493 s … 32.511 s 5 runs
I don’t know how to clear the cache in pnpm
❯ hyperfine --runs 5 \ --setup 'pnpm install' \ --prepare 'rm -r node_modules' \ 'pnpm install' # .......... Benchmark 1: pnpm install Time (mean ± σ): 4.817 s ± 0.035 s [User: 5.645 s, System: 3.617 s] Range (min … max): 4.785 s … 4.863 s 5 runs
NOTE: at the time of writing / measuring / testing,
- ember.js is still a v1 addon
- limber consumes a lot of v1 addons
over time as more things move to v2 addons, these benches will improve on the same hardware, changing no other variables.