Test passes in --server mode but fails without

I have a weird behaviour when running tests with ember t in the Terminal, - there is one that fails with:

Testem finished with non-zero exit code. Tests failed.

not ok 9 Chrome - [undefined ms] - error
    ---
        message: >
            Error: Browser exited unexpectedly
            Error while executing test: Acceptance | Address: Authenticated users can visit /address
            Non-zero exit code: null
            Stderr: 

DevTools listening on ws://127.0.0.1:51916/devtools/browser/f31a18cc-452c-4de2-a3b2-dbfc50865f72
            [0809/114553.464989:WARNING:system_snapshot_mac.cc(42)] sysctlbyname kern.nx: No such file or directory (2)
            [0809/114553.478836:WARNING:gpu_process_host.cc(1188)] The GPU process has crashed 1 time(s)
...

There is much more output, I just dropped it for easier reading. The most surprising is to see:

'kern.nx: No such file or directory (2)\n'
...

When I run the same tests but with --server option, they pass all.

When I add await pauseTest(); to the failing test, start ember s and try just to navigate to http://localhost:4200/tests as explained in Ember Guides - Testing section, I get the below errors in the browser console:

The test itself has nothing special and passed before:

import { module, test } from 'qunit';
import { visit, currentURL } from '@ember/test-helpers';
import { setupApplicationTest } from 'ember-qunit';
import { authenticateSession } from 'ember-simple-auth/test-support';
import setupMirageTest from 'ember-cli-mirage/test-support/setup-mirage';
import { setupWindowMock } from 'ember-window-mock';

module('Acceptance | Address', function(hooks) {
  setupWindowMock(hooks);
  setupApplicationTest(hooks);
  setupMirageTest(hooks);

  test('Authenticated users can visit /address', async function(assert) {
    let shop = server.create('shop');
    server.create('user', { shop });
    server.create('address', { shop });

    await authenticateSession({
      access_token: 'azerty',
      token_type: 'Bearer'
    });
    
    await pauseTest();

    await visit('/address');
    assert.equal(currentURL(), '/address', 'user is on the Address page');
  });
});

Even if I comment out all the acceptance tests with skip, all the other unit tests fail as well with the same error:

Log: |
            { type: 'error', text: 'Error: Browser exited unexpectedly' }
            {
              type: 'error',
              text: 'Error while executing test: Unit | Model | shop: should have a country'
            }
            { type: 'error', text: 'Non-zero exit code: null' }

What’s wrong with that? I didn’t make any code nor settings changes.

Thank you.

2 Likes

I just started noticing a similar issue on my apps after a recent Chrome update. I am very suspicious that this is a Chrome bug (least on MacOS) I suspect it may be the same in your case? Basically running Chrome in headless mode (what ember t does) crashes and the test complain about a browser timeout/GPU crash.

At the moment I have worked around it by running either

ember test --launch=firefox

OR

ember test --server --launch=false

and opening the reported URL manually in a running instance of Chrome.

1 Like

@sukima Wow, thank you very much for your response. Exactly, - I’m running Chrome Version 76.0.3809.100 (Official Build) (64-bit) on macOS. Running the tests with ember test --launch=firefox worked fine for me (my bad, I looked for a way to pass in another browser option to ember t and skipped it).

Thanks a lot.

That is the same version that started breaking for me as well.

Glad to be not the only person in the boat :slight_smile:

Firefox 68.0.1 version displays this weird message though in the right bottom corner when running the tests: 58

Me and my colleague are also seeing this issue. Do we know anywhere we could report this to the Chrome or Chromium team or anywhere it has been reported?

I also discovered that it failed in Firefox 68.0.1 (64-bit, masOS 10.14.6) if Firefox is already open when running ember test --launch=firefox. It opens a new tab to run tests, runs all of them with success:

but displays a failure in the terminal:

I remember running into this a little while back. I believe it’s a testem issue. IIRC the workaround (for our app at least) had something to do with some of the testem flags (disabling gpu and maybe sandbox or something, I forget now, sorry). There are multiple issues in the testem repo, one such is here. You could try playing around with different settings in your testem.js file and see if any of the suggestions help.

1 Like

@dknutsen Thank you for the link, really useful, there are a lot of solutions (the issue is still open since April 2017 :scream:. )

The solution that worked for me was to modify the testem.js content from:

module.exports = {
  test_page: 'tests/index.html?hidepassed',
  disable_watching: true,
  launch_in_ci: [
    'Chrome'
  ],
  launch_in_dev: [
    'Chrome'
  ],
  browser_args: {
    Chrome: {
      ci: [
        // --no-sandbox is needed when running Chrome inside a container
        process.env.CI ? '--no-sandbox' : null,
        '--headless',
        '--disable-gpu',
        '--disable-dev-shm-usage',
        '--disable-software-rasterizer',
        '--mute-audio',
        '--remote-debugging-port=0',
        '--window-size=1440,900'
      ].filter(Boolean)
    }
  }
};

to:

module.exports = {
  test_page: 'tests/index.html?hidepassed',
  disable_watching: true,
  launch_in_ci: [
    'Chrome'
  ],
  launch_in_dev: [
    'Chrome'
  ],
  browser_args: {
    Chrome: {
      ci: [
        // --no-sandbox is needed when running Chrome inside a container
        process.env.CI ? '--no-sandbox' : null,
      "--headless",
      "--disable-gpu",
      "--remote-debugging-port=9222",
      "--remote-debugging-address=0.0.0.0",
      "--no-sandbox",
      "--user-data-dir=/tmp"
      ].filter(Boolean)
    }
  }
};

and it worked for Chrome Version 76.0.3809.100 (Official Build) (64-bit) on macOS.

Yeah I think there are other duplicate issues with various causes and various states of development but not sure exactly what the root cause is. Anyway glad you got it figured out!

1 Like

specifically its the --disable-software-rasterizer thats causing mine to crash

1 Like

I have this same issue, and removing --disable-software-rasterizer or --disable-gpu seemed to help. Edit- In Ember-CLI, they removed --disable-gpu from the blueprints, which seems to be like the change to make to existing apps too.

This problem wasn’t present on a fresh 3.12 app, but I do see it on some existing projects, even if I upgrade ember-cli to the latest.

A minimal reproduction of this for me, is to run the official super rentals tutorial app. https://github.com/ember-learn/super-rentals

Removing that option fixed it for me as well, and it does seem like the latest blueprints have the option removed.