Improve test speed with ember exam and github actions

Testing
1

Hello community!

i’m currently testing out ember-exam. i could get the the tests to run faster by splitting them up in multiple browser with:

ember exam -s --split=4 --parallel

but i would like to get those test parallelization in gihub action as well, which does not have a browser.

dos anybody have an idea how that could work?

i tried:

ember exam --split=8  67.19s user 9.86s system 47% cpu 2:43.02 total
npx ember test  68.90s user 10.36s system 48% cpu 2:42.59 total
ember exam --split=8 --parallel=1  72.78s user 11.34s system 33% cpu 4:08.45 total
ember exam --split=8 --parallel  73.39s user 11.88s system 34% cpu 4:09.36 total
ember exam --split=4 --parallel=1 --random  69.38s user 10.81s system 39% cpu 3:23.96 total
2

Typically the way you’d parallelize in CI/CD is to break each partition into separate “jobs” and run one “partition” from ember-exam in each job. We’re using GitLab now so i’m not up on my GitHub Actions syntax and there’s probably a more elegant way to do this, but what we did before was something like the following (pardon the old Node versions, etc, it’s out of date):

name: CI

on:
    push:
        branches:
            - canary
    pull_request:

jobs:
    build:
        name: Download and cache dependencies and pre-build app
        runs-on: ${{ matrix.os }}
        strategy:
            matrix:
                os: [ubuntu-latest]
                node-version: [10.x]
        steps:
            - name: Check out a copy of the repo
              uses: actions/checkout@v2
            - uses: webfactory/ssh-agent@v0.4.0
              with:
                  ssh-private-key: ${{ secrets.SSH_NPM_KEY }}
            - name: Use Node.js ${{ matrix.node-version }}
              uses: actions/setup-node@v1
              with:
                node-version: ${{ matrix.node-version }}
            - name: Cache yarn stuff
              uses: actions/cache@v2
              with:
                path: '**/node_modules'
                key: ci-modules-${{ hashFiles('**/yarn.lock') }}
            - name: Install dependencies
              run: yarn install --frozen-lockfile --non-interactive
            - name: Build ember app
              env:
                BROCCOLI_ENV: test
              run: yarn ember build
            - name: Upload built ember app
              uses: actions/upload-artifact@v1
              with:
                name: dist
                path: dist


    test-partition-1:
        name: Run tests - Partition 1
        needs: [build]
        runs-on: ${{ matrix.os }}
        strategy:
            matrix:
                os: [ubuntu-latest]
                node-version: [10.x]
        steps:
            - name: Check out a copy of the repo
              uses: actions/checkout@v2
            - uses: webfactory/ssh-agent@v0.4.0
              with:
                  ssh-private-key: ${{ secrets.SSH_NPM_KEY }}
            - name: Use Node.js ${{ matrix.node-version }}
              uses: actions/setup-node@v1
              with:
                node-version: ${{ matrix.node-version }}
            - name: Cache yarn stuff
              uses: actions/cache@v2
              with:
                path: '**/node_modules'
                key: ci-modules-${{ hashFiles('**/yarn.lock') }}
            - name: Install dependencies
              run: yarn install --frozen-lockfile --non-interactive

            - name: Download built ember app
              uses: actions/download-artifact@v1
              with:
                name: dist
                path: dist
            - name: Test partition 1
              run: ember exam --query=nolint --split=4 --parallel=1 --partition=1 --path=dist


    test-partition-2:
        name: Run tests - Partition 2
        needs: [build]
        runs-on: ${{ matrix.os }}
        strategy:
            matrix:
                os: [ubuntu-latest]
                node-version: [10.x]
        steps:
            - name: Check out a copy of the repo
              uses: actions/checkout@v2
            - uses: webfactory/ssh-agent@v0.4.0
              with:
                  ssh-private-key: ${{ secrets.SSH_NPM_KEY }}
            - name: Use Node.js ${{ matrix.node-version }}
              uses: actions/setup-node@v1
              with:
                node-version: ${{ matrix.node-version }}
            - name: Cache yarn stuff
              uses: actions/cache@v2
              with:
                path: '**/node_modules'
                key: ci-modules-${{ hashFiles('**/yarn.lock') }}
            - name: Install dependencies
              run: yarn install --frozen-lockfile --non-interactive

            - name: Download built ember app
              uses: actions/download-artifact@v1
              with:
                name: dist
                path: dist
            - name: Test partition 2
              run: ember exam --query=nolint --split=4 --parallel=1 --partition=2 --path=dist


    test-partition-3:
        name: Run tests - Partition 3
        needs: [build]
        runs-on: ${{ matrix.os }}
        strategy:
            matrix:
                os: [ubuntu-latest]
                node-version: [10.x]
        steps:
            - name: Check out a copy of the repo
              uses: actions/checkout@v2
            - uses: webfactory/ssh-agent@v0.4.0
              with:
                  ssh-private-key: ${{ secrets.SSH_NPM_KEY }}
            - name: Use Node.js ${{ matrix.node-version }}
              uses: actions/setup-node@v1
              with:
                node-version: ${{ matrix.node-version }}
            - name: Cache yarn stuff
              uses: actions/cache@v2
              with:
                path: '**/node_modules'
                key: ci-modules-${{ hashFiles('**/yarn.lock') }}
            - name: Install dependencies
              run: yarn install --frozen-lockfile --non-interactive

            - name: Download built ember app
              uses: actions/download-artifact@v1
              with:
                name: dist
                path: dist
            - name: Test partition 3
              run: ember exam --query=nolint --split=4 --parallel=1 --partition=3 --path=dist


    test-partition-4:
        name: Run tests - Partition 4
        needs: [build]
        runs-on: ${{ matrix.os }}
        strategy:
            matrix:
                os: [ubuntu-latest]
                node-version: [10.x]
        steps:
            - name: Check out a copy of the repo
              uses: actions/checkout@v2
            - uses: webfactory/ssh-agent@v0.4.0
              with:
                  ssh-private-key: ${{ secrets.SSH_NPM_KEY }}
            - name: Use Node.js ${{ matrix.node-version }}
              uses: actions/setup-node@v1
              with:
                node-version: ${{ matrix.node-version }}
            - name: Cache yarn stuff
              uses: actions/cache@v2
              with:
                path: '**/node_modules'
                key: ci-modules-${{ hashFiles('**/yarn.lock') }}
            - name: Install dependencies
              run: yarn install --frozen-lockfile --non-interactive

            - name: Download built ember app
              uses: actions/download-artifact@v1
              with:
                name: dist
                path: dist
            - name: Test partition 4
              run: ember exam --query=nolint --split=4 --parallel=1 --partition=4 --path=dist

The only important callout here is that because GH Actions is consumption based we were pre-building the app and caching that (and node_modules) before running the test jobs, which saved some computation time inside each test job (otherwise the app is built in each test job which multiplies that considerable time and effort by 4, which you then get billed for).

Perhaps nowadays there’s a more elegant way of making a job matrix for the partitions so you only specify the test jobs once instead of duplicating them four times.

@ijlee2 also had a great post about this a little while back which you can find here.

1 Like
3

@dknutsen Thanks for cc’ing me and providing the link to the blog post.

@fluxsaas As @dknutsen mentioned, the idea is to build a test app once, save the build somewhere, then pass the build to each ember-exam partition. This way, you won’t need to build the app 4 (or 8) times before running the tests.

steps:
  ...

  - name: Download app
    uses: actions/download-artifact@v3
    with:
      name: dist
      path: dist

  - name: Test
    run: yarn test --partition=${{ matrix.partition }} --path=dist

The code in my blog post, which shows v2 of GitHub Actions, is outdated. I recommend using v3 because caching is simpler. I’ve updated the workflow templates just now:

(Haven’t tested them so there may be errors.)

1 Like
4

awesome, thanks @dknutsen and @ijlee2 !

i managed to get everything up and running. i think currently the time savings of that setup does not improve over the “classic” setup due to artifact uploading seperate npm installs. i think i can still improve the setup further but needs a little more research!

thanks!

5

Yeah IIRC from our experimentation by far the longest part of our CI runs is the build with istanbul instrumentation (via ember-cli-code-coverage), not really the test suite, so parallelization doesn’t really speed things up a ton. But theoretically it would matter more and more over time as our test suite grows, and if build time drops a lot once we’re on Embroider. Guess we’ll see!

1 Like