Hi! I’m Timo Tijhof, the project lead for upstream QUnit.
First off, I’d like to say thank you for many years of useful and kind upstream feedback, bug reports, and pull requests.
I’d like to hear from you in the Ember community about assert.expect() and how you like (or don’t like) its use. For example, does it give you confidence when writing tests? Perhaps you recall a specific test where it helped catch a mistake? I’d love to hear these stories.
The reason I care, is that I’m considering to make a clearer default in the QUnit manual and ecosystem about best practices. My current inclination is to not actively promote assert.expect(). It will forever be part of the QUnit API and work perfectly for existing code, but I suspect it is generally of neutral or (slightly) net-negative value to a new project. If I learn here that there is still much value from mass adoption, I might reconsider this!
History
assert.expect() was introduced in QUnit 1. Back then, testing callback-heavy code was not standardised and difficult to do in a reliable fashion. ES6 async-await would not be standardised for another ten years. The only way JavaScript could run code (i.e. an assertion) after something async, was through a callback function. This meant that even when the test function has returned and all assertions upto there passed, there may be other functions that you hoped would run as well, but perhaps didn’t.
QUnit.test('example', function (assert) {
var x = new Gump();
x.on('change', function (state) {
assert.equal(state, 'running');
});
x.on('end', function () {
assert.true(true);
});
x.run();
assert.equal(x.name, 'Forrest');
});
In the above example, if x.run() has a regression and doesn’t invoke the change or end events, the test would still pass. Likewise, if it was working but was asynchronous, we need a way to “wait” for it.
The affordances for this in QUnit 1 were assert.async() for async code (originally QUnit.stop and QUnit.start), and assert.expect() more generally for nested assertions (async or not).
QUnit.test('example', function (assert) {
assert.expect(3); // <
const done = assert.async(); // <
var x = new Gump();
x.on('change', function (state) {
assert.equal(state, 'running');
});
x.on('end', function () {
assert.true(true);
});
x.run(done); // <
assert.equal(x.name, 'Forrest');
});
Today
QUnit.test()accepts async functions (since QUnit 1.16 in 2014). You can rely on QUnit tracking the async test, waiting for it to complete (or timeout), and failing the test suite accordingly. This works both with modern ES6async-await, and in ES5 where you can return the asyn chain as a Thenable from the test function.assert.timeout()lets you enable or change a timeout. Especially in large code bases where perhaps a global timeout can’t be set yet, you can opt-in with an enforced or shorter timeout in invidivudal tests.assert.rejects()is like assert.throws() but for async functions and other promises/thenables.assert.verifySteps()represents the philosophy that nested assertions can and should be avoided.
Hypothesis A: For the majority of tests, both sync and async, failure scenarios are naturally caught. Either as failed assertions, uncaught error, async rejection, or timeout.
The notable exception is nested assertions. Testing data that originates from callback is a frequent need when testing Ember applications. JavaScript is inherently event-driven, and this comes up also when testing vanilla JavaScript code that uses the DOM API, or Node.js apps that use EventEmitter, jQuery plugins, Web Components, etc.
Hypothesis B: Avoiding nested assertions leads to tests that are more insightful, more reliable, and more strict; compared to counting nested assertions.
How (Part 1)
In the cases I’ve come across so far, I found assert.verifySteps() to generally fit best. It has the benefit of being built-in, discoverable, easy to adopt in existing code, with no extra code to make it work.
QUnit.test('example', async function (assert) {
const x = new Gump();
x.on('change', function (state) {
assert.step(state);
});
x.on('end', function (distance) {
assert.step('end');
});
await x.run();
assert.equal(x.name, 'Forrest');
assert.verifySteps(['running', 'end']);
});
Real example from MediaWiki:
QUnit.test('multiple mw.hook consumers with memory', function (assert) {
mw.hook('test.multiple' )
.add(function (data) {
// Listen for future events (no memory yet)
assert.step('early ' + data);
})
.fire('x')
.fire('y')
.fire('z')
.add(function (data) {
// Start with memory from last fire
assert.step('late ' + data);
});
assert.verifySteps([
'early x',
'early y',
'early z',
'late z'
]);
} );
How (Part 2)
Some projects prefer a more idiomatic and JavaScript-native solution. In the below examples I apply local variables and assert.deepEqual, instead of the Step API used above.
Example adapted from Ember.js Guides (original)
test('should trigger external action', async function(assert) {
let value;
this.set('externalAction', (data) => {
value = data;
});
await render(hbs`<CommentForm @submitComment={{this.externalAction}} />`);
await fillIn('textarea', 'You are not a wizard!');
await click('.comment-input');
assert.deepEqual(value, { comment: 'You are not a wizard!' }, 'value');
});
Example from MediaWiki:
QUnit.test('mw.track', function (assert) {
var events = [];
mw.trackSubscribe('foo', function (topic, data) {
events.push([ topic, data ]);
});
mw.track('foo', { key: 1 });
mw.track('foo', { key: 2 });
assert.deepEqual(events, [
[ 'foo', { key: 1 } ],
[ 'foo', { key: 2 } ]
], 'Events');
});
QUnit.test('start hidden and become visible', function (assert) {
mockDocument.hidden = true;
let called = 0;
mw.visibleTimeout.set(function () {
called++;
});
assert.strictEqual(called, 0);
mockDocument.toggleVisibility();
assert.deepEqual(called, 1);
});
I’d also like to draw attention to lack of ordering strictness and (if not using assert.expect) a lack of possibly inintended repeat events.
// Given: "start" is emitted after "change"
// Given: "change" is emitted twice
//> Result: Passing
test('example', function (assert) {
Example.on('start', function () {
assert.true(true);
});
Example.on('change', function (data) {
assert.equal(data, 'running');
});
Example.on('end', function () {
assert.true(true);
});
Example.run();
});
// After 30s:
//> Error: Timeout reached after 30 seconds.
test('example', function (assert) {
assert.expect(1);
let done = assert.async();
Example.on('change', function (data) {
assert.equal(data, 'something');
done();
});
Example.run();
});
Depending on what the application supports, structuring this as awaiting/returning a promise, or passing done to an “end” event. For cases where the only bug is that the “change” event didn’t happen, this results in a faster failure.
// Afer 0.1s:
//> Error: Expected ['something'], Actual: [].
test('example', function (assert) {
Example.on('change', function (data) {
assert.step(data);
});
await Example.run(); // or Example.on('end', …);
assert.verifySteps([ 'something' ]);
});
Benefits
When moving the assertion out of a nested context means that it is always reached, unless the test function fails in a way that is naturally caught.
Performing the assertion in the test function, also means you are in charge of when it runs (you explicitly indicate when you expect the data to be there). In the case of async events, I find this generally encourages developers to engage with and test their own async APIs. By first waiting for when you expect the code to be done, and then performing the assertion, I find projects tend to get either more descriptive or faster feedback.
By asserting data as steps, arrays, or numbers (instead of assertion counts), we naturally perform a stricter assertion, one that also validates the order and frequency of events. In practice, I find this leads to tests that are more self-documenting and more deeply cover APIs, beyond what test coverage metrics can measure, and that would usually be considered too much effort to test explicitly on its own. But, are naturally tested when using steps or an array.
So with that, I ask for feedback on these hypotheses:
Hypothesis A: For the majority of tests, both sync and async, failure scenarios are naturally caught. Either as failed assertions, uncaught error, async rejection, or timeout. (Except nested assertions).
Hypothesis B: Avoiding nested assertions leads to tests that are more insightful, more reliable, and more strict; compared to counting nested assertions.
– Timo (Krinkle)
