Ember Testing Improvements


#1

Hey folks,

Tom & Yehuda here. We’ve been reviewing the ember-testing package and we wanted to toss out a few ideas for improvements so that people could discuss them (and hopefully implement and send PRs for! :wink:)

Better Async Hooks for wait

At the moment, ember-testing hardcodes a few specific asynchronous actions in the wait implementation.

function wait(app, value) {
  var promise;

  promise = Test.promise(function(resolve) {
    if (++countAsync === 1) {
      Test.adapter.asyncStart();
    }
    var watcher = setInterval(function() {
      var routerIsLoading = app.__container__.lookup('router:main').router.isLoading;
      if (routerIsLoading) { return; }
      if (Test.pendingAjaxRequests) { return; }
      if (Ember.run.hasScheduledTimers() || Ember.run.currentRunLoop) { return; }

      clearInterval(watcher);

      if (--countAsync === 0) {
        Test.adapter.asyncEnd();
      }

      Ember.run(null, resolve, value);
    }, 10);
  });

  return buildChainObject(app, promise);
}
  1. Wait for the router to finish loading
  2. Wait for pending Ajax requests to complete (based on jQuery.ajaxStart and jQuery.ajaxStop
  3. Wait for any timers to finish
  4. Wait for the current run loop queue to be flushed

Of course, there are many other kinds of async actions in the browser, such as IndexedDB, Object.observe, DOM Mutation Observers, postMessage, promise resolution, etc.

We aren’t going to wrap every single one of these async actions in Ember Core, but it would be useful if there was a way to provide Ember-friendly wrappers for these APIs that also hooked into wait for testing.

The easiest approach we can think of is to instrument RSVP's callbacks, and have Ember-friendly async wrappers use RSVP promises in their abstractions. We would also add a mechanism for registering pending async actions with wait, and hook the instrumented RSVP into that.

Here is an example of using this API with jQuery’s Ajax hooks.

var Test = Ember.Test;

Test.onInjectHelpers(function() {
  Ember.$(document).ajaxStart(function() {
    Test.defer();
  });

  Ember.$(document).ajaxStop(function() {
    Test.advance();
  });
});

Chaining is Unnecessary

The current API relies on promise chaining for asynchronous behavior. Here is an example from the current documentation:

visit('posts/new')
  .click('.add-btn')
  .fillIn('.title', 'Post')
  .click('.submit')
  .then(function() {
    equal('.post-title', 'Post');
  })
  .visit('comments')
  .then(function() {
    equal(find('.comments'),length, 0);
  });

However, it seems like it should be possible for the testing framework to track the current implicit promise, and do this chaining for you. The result is a much more pleasant API that retains the original semantics:

visit('posts/new');
click('.add-btn');
fillIn('.title', 'Post');
click('.submit');

andThen(function() {
  equal('.post-title', 'Post');
});

visit('comments');

andThen(function() {
  equal(find('.comments'),length, 0);
});

The idea is that calls to helpers like visit would produce a global promise (Ember.Testing.currentPromise). When the test called another async helper, it would first check whether such a promise existed, and chain onto it.

QUnit Integration

You may have noticed that the assertions in the examples above (in our case, using QUnit) are required to be inside andThen blocks. It is conceivable that the built-in assertions in QUnit could be patched at runtime to automatically chain onto the Ember.Testing.currentPromise. In that case, we could clean up the above example into the following:

visit('posts/new');
click('.add-btn');
fillIn('.title', 'Post');
click('.submit');

equal('.post-title', 'Post');

visit('comments');

equal(find('.comments'),length, 0);

We think this is quite nice. It looks like synchronous code but is in fact entirely asynchronous. At this point, the only requirement for the andThen block is for arbitrary code running in between testing helpers.

Autorun Flushing

Currently, setting the Ember.testing flag to true causes anything that triggers asynchronous behavior that is not wrapped in a run loop to raise an exception. This feature, known as autorun, makes it possible to interact with an Ember application in, e.g., the developer console and not have to know about the run loop. However, because historically tests have run synchronously, by the time the autorun has had a chance to be scheduled, the tests have already finished running (and presumably failed).

Now that we are moving to an asynchronous model of testing, however, we have realized that it is likely possible that between asynchronous actions we can flush the queue and have the tests run after they are done. In other words, flushing the autorun queue should be treated as an implicit asynchronous event à la waiting for an Ajax response.

test("Setting the `isPlaying` property makes it possible to pause the player", function() {
  view.set('isPlaying', true);

  // An autorun was scheduled above, so `wait` is in a deferred state.

  // Wait until the autorun has flushed before proceeding.
  visit("/posts");  

  // We would use `then` here but it seems bad to make `window` a thenable.
  // Bikeshedding welcome.
  andThen(function() {
    ok(find("#post"), "A post exists");
  });
});

Because any arbitrary code needs to go inside andThen (in order to run temporally after the other async code), this model eliminates any need to understand the run loop or autorun. It “just works”.

From an implementation strategy point of view, it should be relatively straightforward. Anything that causes an autorun to be scheduled would simply set Ember.Testing.currentPromise to a promise that resolves once the autorun has completed. Any calls to click(), visit(), etc. would automatically be chained after this promise and therefore only run after the autorun has finished flushing.

The idea is that code that triggers an autorun can be put at the beginning of tests, but not after another async helper has been called. Once an async helper has been called, subsequent calls to .set etc. should go inside of an andThen().

For example:

test("The /articles route requires a currentUser", function() {
  // Each of these calls schedules async behavior to happen when the
  // current run loop flushes. The first call schedules an autorun. That
  // autorun creates a `Ember.Testing.currentPromise`, which subsequent
  // async calls are chained onto.
  controllerFor('application').set('currentUser', null);
  viewFor('modal').append();

  // This call to `visit()` gets chained onto the current autorun promise.
  visit("/articles");

  // The contents of this function are called once the router has finished
  // moving the application into the `login` route (because there is no
  // current user).
  andThen(function() {
    equal(window.location.path, "/login", "The app is at /login");

    // This will trigger an autorun. As a result, the `wait()` promise returned
    // from the implementation of `andThen` will be in a deferred state.
    controllerFor('application').set('currentUser', App.User.find(1));
  });

  // This call will be chained onto the end of the previous promise, which
  // means it won't run until the async results of setting the `currentUser`
  // have propagated via the autorun flushing the run loop.
  visit("/articles");

  // The code in this function will run once the router has finished moving
  // the application into the `articles` route.
  andThen(function() {
    equal(window.location.path, "/articles", "The app is at /login");
  });
});

Additional Helpers

Currently, most of the helpers in ember-testing seem focused on integration testing. While of course important, we should also assist users in unit testing individual classes in their app.

Because most objects in the system need to be instantiated via the container API, it is not sufficient to ask users to just create a new instance of their class. There is a small amount of additional setup that can be easily abstracted into the testing framework.

We have been using a small testing library that we wrote for our training classes that we think could serve as an inspiration, although it does require some cleanup. Let’s look, for example, at the testComponent helper. Here’s how you use it:

testComponent('audioPlayer', "once the <audio> tag has loaded, the component's duration and isLoaded properties are set", function(component) {
  component.set('src', "audio/Southern_Nights_-_07_-_All_My_Sorrows.mp3");

  propertyShouldBecome(component, 'duration', 219);
  propertyShouldBecome(component, 'isLoaded', true);
});

This testComponent helper is analogous to QUnit’s test(), but instantiates an Ember.Component class, puts the instantiated component into the DOM, and then passes it as the first argument to the test function. Inside, we change some properties on the component and assert that the expected side-effects occur. The propertyShouldBecome helper is simply a way to assert that an object’s property asynchronously becomes a value.

In this case, we use propertyShouldBecome since setting the src will trigger a download in the browser, which happens outside of code that we can easily instrument.

The implementation of this helper is small but should make it much easier for developers to get started with unit testing:

function testComponent(componentName, message, callback) {
  test(message, function() {
    var component = App.__container__.lookup('component:'+componentName);
    component.appendTo('#qunit-fixture');
    callback(view);
  });
}

Obviously, the usage of App.__container__ here is a problem. Instead, Ember.Test.onInjectHelpers should pass the container as a parameter to the code that builds helpers. This should also improve some of the existing helpers, which currently use an injected app.__container__.

This also means that App.reset() would need to rebuild the helpers, unless we’re missing something.

Conclusion

We are very excited about the future of testing in Ember.js. It has been encouraging to see so many people collaborating on improving the out-of-the-box testing story for Ember apps, and we are excited to continue improving it over the next few releases.


HTTP Mocking for ember-testing
#2

I am not quite sold on the implicit magic that is going on in this section, I feel it needlessly muddies the waters.

Is it assumed that QUnit is the ‘default’ testing framework for ember, or did you just pick it for the examples out of preference?

Digging the work being done in this area :smiley:


#3

I disagree; I think the dechaining is extremely awesome and demos really well, but I’ll reserve my final opinion for when I can actually try it out. In the meantime, I’m happy whenever Ember can find a way to turn potential nitty gritty stuff into solid DSLs, so long as the magic is sturdy and reliable.


#4

Look at the post-facelift router.js code for instance; them’s some of the ugliest test cases I’ve ever seen. andThen would do wonders to it.

Speaking of which… can some of this stuff be microlib’d in a way that’s accessible to router.js testing (or any library that depends on RSVP) without bringing in Ember?


#5

I love where this is going!

Thank you guys (and Erik, Stefan and Teddy) for pushing the ember-testing thing forward.

Random nitpick: Erik has pointed out that sometimes promises might (intentionally) never be resolved, or might be long-running for something unrelated. So perhaps “everything that uses RSVP will automagically work” is too optimistic.

I agree it should be extracted eventually.

I could be wrong, but maybe what’s stopping us from extracting it right now is the lack of good enough package management / build tools – which is why it’s all kind of monolithic right now. (I’m still working on this btw, and will continue for a while. Not as much and fast as I’d like – it’s kind of stop and go – but it continues to irk me that we don’t have this stuff working, so I’m quite motivated to put time into it.)


#6

I like this direction a lot and think this is worth experimenting with.

The current approach is extremely functional, but I can confirm from my experience with it the two pain points you identified: 1) needs easy way to hook in other async, and 2) test code could be more readable than it is today.


#7

QUnit is the officially supported framework in ember-testing, but you can use other testing frameworks if you’d like, provided you’re willing to use an “unofficial” adapter. If you prefer Mocha, you can use my adapter: https://github.com/teddyzeenny/ember-mocha-adapter


#8

I like the first part of this proposal a lot. There are many sources of async and we should be able to integrate them into ember-testing easily. As for chaining being unnecessary, I am concerned that the burden on the reader becomes too great. In a way, it reminds me of RJS from Rails of old. Very cool, sort of, and makes a cool demo, but turns out to be a bad idea because of the huge difference in what you write and what actually happens. We’re encountering this problem now with trying to use conditionals or passing arguments to async helpers that happen “too early”. Thoughts?


#9

To be concise.

  • I like the idea of having more async hooks

  • I like the idea of getting rid of Ember.run in my tests (I was talking about that this afternoon with a friend)

  • I like the idea of having test helpers for Unit tests too. For now I’m myself instanciating view through the App.container variable

  • Concerning the chaining, I have to say it does not matter for me. For example, here is an example of what I have written:

    `visit(’/’)

     .clickOnProject('Polop')
     .clickOnTab('Scenarios')
     .clickOnScenario('Wayne')
     .clickOnTab('Action words')
     .clickOnActionWord('Gart')
     .clickOnTab('Scenarios')
     .expectScenarioSelected('Wayne')
     .clickOnTab('Action words')
     .expectActionWordSelected('Gart');
    

`

For me it looks good, and does see any difference without the chaining.


#10

The main reason that chaining is critical in our proposal is that it allows an implicit promise to be created for autorun.


#11

Love where this is going. Thanks especially for posting the helpers.js link, been looking for a better approach to unit-testing.


#12

Looks like this just landed in master. Looks like the new features are backwards compatible also.

@teddyzeenny laid out the various ways to use ember-testing here: