Ember is built for ambitious web apps and ember cli makes really easy to write tests, but I find that with the current setup, when the test suite growth over 500 unit tests, things to get quite slow, with assertions taking around 40ms each.
I know that a good way to speed up acceptance tests is to reduce animations length, and acceptance tests by necessity will take a considerable time anyway, but unit tests should run fast.
Are there any tricks for this? Did anyone investigate the bottlenecks? Could it be the resolver? The module helper? Any particular things to avoid?
I’ve actually had a crazy idea in the back of my head that I’ve been meaning to work on, related to testing performance.
A big part of the FastBoot work we did was improving the Ember internals to be more rigorous about isolating all application state, since your FastBoot server needs to handle multiple requests in parallel. Previously, we got away with being less rigorous because there is only one app running at a time, even in tests, where each test runs sequentially.
However, now that we have the ability to run multiple instances of the same Ember application in one context, I’ve been thinking about if it may be possible to run multiple Ember tests in parallel. For example, imagine running the test harness but instead of one test at a time, it was running four or more, each one totally isolated.
Now, obviously JavaScript is single-threaded so this would only help improve the performance of tests that are async (but that’s most acceptance tests). Additionally, I think moving the testing infrastructure to using the app instance abstraction we built for FastBoot would also improve performance. Right now we setup and teardown the entire app between tests, which is a much higher price than we theoretically need to pay.
Oh, that’s very interesting, thanks for sharing.
Do we also boot the whole app in unit tests? If so, perhaps at least some app modules could be treated as isolated libraries that can be tested without running ember at all?
The application is already pretty lazy in terms of what it instantiates (thanks to the registry/container architecture). One big win of using application instances instead of recreating the application each time is that initializers would only have to be run at the start of the tests rather than between each test.
I still think it would be nice if unit tests could be isolated as much as possible. I don’t care too much about the speed of acceptance tests, although the multithreading option sounds very nice, but I care more about immediate unit test feedback when editing a file.
I also seem to miss some kind of option to just re-run one specific test. Ideally there’d be some file watcher that just runs the corresponding test. Kind of what guard does for Rails. Although I still think the biggest hurdle is having to build/transpile your app. If it were just plain ES5 without imports we could just run individual test files, but obviously that doesn’t work…
Your screen looks somehow different than mine, but yes, I know that I can just rerun particular tests. What would be useful would be if this were somehow triggered upon saving a file.
One perf tip: a lot of behaviors that used to require acceptance tests are now testable with component integration tests, which are significantly lighter-weight because they don’t boot a whole app.
yes, but with livereload all the tests are rerun (including the slow integration tests).
What I want is something like: I save a file → ONLY the corresponding test is run.
As said, with Ruby/Rails there is something like Guard that does exactly that. Additionally, it also integrates with services like inotify.
What I’m doing now is selecting one unit test file (or describe block, I’m using mocha) so at least not all the tests are rerun, but that’s still something I have to do manually.
The web interface is quite nice, but it’s not really toolable, while ember test and ember test --server don’t seem to have an option to just run one specific file.
Maybe there’s a way to do it and I just don’t know it yet.
One perf tip: a lot of behaviors that used to require acceptance tests are now testable with component integration tests, which are significantly lighter-weight because they don’t boot a whole app.
My problem is not with integration tests. I want fast unit tests, but we’re not near 1ms yet. I think it’s probably the need for transpiling the code and setting up the resolver that adds to this, but maybe it’s something else too.
yes, i know that. I only wish that this would happen automatically, because as it is now, I have to go to the main page (which runs all the tests) and select one specific test.
