I’ll be giving a talk at Ember NYC this upcoming thursday (2018-04-26 meetup details) on the latest and greatest in testing. I plan on including a Q/A type section on “real world examples” and I’d love to have a more rounded representation from the community at large (instead of just the things I’ve personally faced).
Please post some examples (gists, link to GH, inline, etc) of what you have been struggling with lately in your tests, and I’ll try to include some solutions for them in my talk (will also try to update the thread with details).
Awesome! I just want to add that we will be live streaming this meet up as well! So folks that can’t make it to NYC will still be able to enjoy the talk.
Now to ask my testing questions…
One abstract thing I struggle with is having a massive amount of acceptance tests. The past year I’ve been working on an application that is full of forms and I’ve found that I’m writing user centric tests for inputs, validations, successful saves, and server errors. The forms are wizard-like and they depend on the current user, so we’ve been using acceptance tests. However, it takes a noticeable amount of time to run the suite. I’d love to get your thoughts on how you would approach an application that had 1,000 acceptance tests.
Another question - I’ve never been able to successfully test drag and drop behavior when using 3rd party libraries like dragula. I usually just paper over these tests and find some creative way of skipping them. I’m wondering if there is a best practice approach for testing complex browser interactions?
I’ll second drag-and-drop interactions, especially file uploading. Select elements too. You can throw any value you want with fillIn, even invalid options.
We have been struggling with a huge number of acceptance tests too. We split our tests with ember-exam and run on Chrome Headless and over time, the test suite has become pretty slow. At times, due to the sheer number of tests, we also have the browser crashing when the test is going on which results in one or two failed tests.
We have unfortunately had quite a few different problems related to testing our app that uses Ember Engines.
Using the plain setupTest(hooks) it is not possible to test things that are part of the engine. Instead you can pass in a custom resolver using something like setupTest(hooks, { resolver }), with resolver coming from engineResolverFor(). This seemed to work, but once you do that you no longer have access to things outside the engine, which created issues with service injections and looking up things using this.owner.lookup().
What we did instead was use a custom resolver that can resolve from both the host app, and the lazy loaded engine:
import Resolver from 'my-app/resolver';
import engineResolverFor from 'ember-engines/test-support/engine-resolver-for';
let engineResolver = engineResolverFor('the-async-engine');
const resolver = Resolver.extend({
resolve() {
let resolved = this._super(...arguments);
if (resolved) {
return resolved;
}
return engineResolver.resolve(...arguments);
},
}).create();
This seems to work for us now, but is probably not ideal. It also creates a bunch of other problems that got surfaces when trying to use the setupMirage() helper from ember-cli-mirage. ember-cli-mirage sets up an initializer that registers something on the application registry. Later, in the setupMirage() function it is trying to this.owner.lookup() that registered thing, but due to the custom resolver passed to setupTest() the application of the initializer is not related to the this.owner being available in tests.
In summary:
testing code that is using Ember Engines is not straight forward yet
with a few workarounds we can make it work reasonably okay
Testing loading/intermediate states (ie: that stuff that happens while await is awaiting) isn’t the most intuitive. Here is a pattern I’ve settled on, but I’ve got @lukemelia helping me out so I feel like I’m cheating .
let clickPromise = click('[data-test-renew-now-button]');
await waitFor('[data-test-processing-spinner]');
assert.isPresent('[data-test-processing-spinner]');
await clickPromise;
Testing ember-orbit models. I was hoping to drop it in as an easy replacement for Ember Data to get offline support, and changing data reads/writes was easy, but it wasn’t clear to me how to test against an app using ember-orbit. I believe I got some errors, but I’m not sure what they were. The pattern just wasn’t as obvious as it was with Ember Data and Mirage.
Testing that my addon works and plays nicely with fastboot. Often times I’m writing an addon that isn’t made for fastboot and it’s easy to accidentally slip something in that will be problematic for fastboot rendered apps. For example, a reference to window or document
Currently, I’ve been writing node qunit tests that build the app and call Fastboot#visit. This has been helpful with travis, but would love to know what the blessed approach is here.
Yeah, I think we should package this pattern up as its own addon. I would be happy to give pointers if somebody wants to give it a go.
Other patterns also exist like ember-fastboot-addon-tests, but personally I don’t prefer the ember-cli-addon-tests architecture that it’s built on – I think dummy apps plus eslint can exercise everything that addon-tests can, in a faster way that doesn’t involve making your test suite call NPM (which skeeves me out).
One thing I’m yet to come up with a great story on is how to nicely generate some Ember Data models in tests that can be passed into components that expect them as arguments but do not fetch the models themselves.
For example, I may have a blog-post component with a post argument. I want to use Mirage to generate a post using my factories and pass it into the blog-post component in a test, but there’s no great way to do this. I have ended up doing something like this:
test('rendering a blog post', async function(assert) {
const post = this.server.create('post', 'inSomeState');
await Ember.run(async function() {
// Must be inside a Run Loop to use the store, abstraction feels really leaky
this.set('post', this.store.find('post', post.id);
});
await render(hbs`
{{blog-post post=post}}
`);
// Perform some assertion, finally
});
This feels… Not great. I get the benefit of avoid an acceptance test when I really want an integration test, but it feels like this should be easier. Is there a better way?
I’ve wanted to ditch Mirage for ages because I hate spending so much time creating a fake backend on all the Ember apps I work on. The only viable alternative I’ve seen is a DockYard blog post about how they do end-to-end testing using a live Phoenix server - which looks awesome.
I’d love to hear your thoughts on mocking vs having a live server, what you do at LinkedIn for testing Ember apps, and if there are other approaches I haven’t thought about.
Testing components that have debug assertions using the new testing syntax. In the past it would throw, now it doesn’t. There doesn’t seem to be any work around either so we annoyingly have a few files that are struck on the old syntax.
I see that many guys mention asymmetry for the acceptance test. I also found this in our test suite. Would be great to see some great examples of unit and integration test for different types of entities. Like services, models, controllers, routes, initializers etc.
And it would be even greater to see in this examples using of sinon.js
.
