HTTP Mocking for ember-testing


#1

Typically, I write integration tests in the same manner I write Ember.js apps: totally separated from any HTTP services that the service(s) it might communicate with (e.g. not from within ember-rails or another server framework integration library).

This creates a somewhat interesting problem when writing integration tests: how to provide mocked data to the application in way that mirror connection to an actual service. I imagine for people who use ember-rails, they drive integration tests with rspec and fill their database with the correct data before running each test.

For those not integrating with a service, I’d like to see HTTP mocking added to ember-testing to better control what happens when specific responses are returned.

There are three main ways I’ve seen people mock data that comes from the server: fixtures, fake servers, and response emitters.

You’ll be familiar with fixtures if you use a library like ember-data or ember-model: you supply an adapter that has some canned data available so when you make a request for, say find('post', 1) it never goes to the server and just loads your fixture data.

I love fixtures for prototyping but I think they’re not ideal for integration testing for the simple reason that using them will leave your custom adapter code totally unexercised: it’s easy to have a bug with url, HTTP verb, or success handler that goes unnoticed in testing.

Faker servers are probably the most common type of HTTP mocking around. If you’ve used sinon in the browser or something like webmock on the server you’re familiar with this style. You set up the requests you’d like intercepted and the responses they should load ahead of time and then run you tests.

In Ember, it might look something like this:

var server = new FakeServer({
  "/rest/ember/show": {
 
  },
 
  "/rest/product": {
    products: [
      {id: 1, name: "foo"}
    ]
  },
 
  "/rest/ember/create": {
    fields: [
      {id: 1, name: "summary"}
    ],
 
    product: {
      id: 1,
      name: "foo",
      components: [
        {id: 1, name: "foo component"}
      ]
    }
  }
});

I think this approach works well in synchronous environments, where the pattern emerged, but don’t offer the fine-grained control you’d want in asynchronous environments. For example, if you wanted to check of the presence of loading views or test what happens when the response order of two simultaneous requests varies.

Response emitters is the pattern I’ve been using myself. It involves emitting a response at a specific point in the flow of execution by chaining a promise or integrating with the promise-free helpers, should we ever add them

visit("/invoices")
.httpRespond("get", "/api/v1/invoices", {... response object ....})
.click(".remove-invoice")
.httpRespond("delete", "/api/v1/invoices/13917998", {"message":"Invoice was successfully deleted!"})
.then(function(){
  equal(find("#flash-message").text(), "Invoice was successfully deleted");
})

Because the {... response object ....} can get pretty big, we’re also using a factory library, but this is roughly how it works.

So, I need some feedback:

  1. Should we add HTTP mocking into ember-testing
  2. Would you prefer a strategy of fixtures, fake servers, response emitting, or something else?
  3. What extra features would you find handy (e.g. failing a test if there are requests that weren’t responded to or a requests is triggered that isn’t intercepted)?

#2

Discourse uses mocking with fixtures right now and I’ve found it quite effective. We don’t use ember-data, so all of our Ajax goes through one Discourse.Ajax endpoint.

In testing mode, Discourse.Ajax will look for a fixture named the same thing as the URL being contacted. If it exists, it returns a promise with the contents from that file. If a path is contacted without being represented by a fixture, it logs an error to the console so it can be caught.

We also have a rake task that populates the fixtures, given a running instance of a dev server.

As for the fake servers / response emitting, I can’t speak to that. But I can say the fixtures have suited us well and caught a lot of bugs.


#3

One thing that would be handy is “timed out” or “failed” requests. That should cover testing scenarios for slow or mobile networks.


#4

(I do not use Ember day-to-day so take anything I say with a grain of salt.)

  1. HTTP mocking is a great feature for any client side tests. I think it would be ideal if this were reusable and agnostic to Ember, i.e. if you set out to make the best HTTP mocking library in the world, which didn’t need to run within ember-testing or QUnit. But I understand that’s probably adding unnecessary constraints that don’t help your users.

  2. The response-emitter version looks very nice, and as you say, is the most flexible. If I had to choose one, it would be that. Beyond that, fake servers are a simplicity and separation-of-concerns win for the 80% case, so if there’s bandwidth to implement those as well, that would be great.

  3. It’s important to make the failure cases almost as easy to test as the success cases, as error handling flow is often under-tested. Thus APIs should be able to send back arbitrary status codes, not just 200, or they should be able to timeout, or perhaps even simulate an XMLHttpRequest abort.


Resetting the store (fixtures) between tests
#5

The libraries are already totally framework agnostic and consists of two parts: One provides a FakeXMLHttpRequest constructor that mirrors the XMLHttpRequest spec but adds methods for forcing a response and a one add swaps native xhr for fake xhr and does the recording process.

Some of this code is cribbed from sinon and a few other sources that packaged these things too tightly. The goal wasfor either library to be useful on its own.


#6

For what it’s worth, I’m using jquery.mockjax with great success for both jQuery $.ajax projects (like discourse) and ember-data projects as it’s just xhr mocking at the core

Here is a simple example of what my helper looks like (used with ember-testing)

function stubEndpointForHttpRequest(url, json) {
    $.mockjax({
        url: url,
        dataType: 'json',
        responseText: json
    });
}

$.mockjaxSettings.logging = false;
$.mockjaxSettings.responseTime = 0;

Then from my tests I simply setup a given xhr like so

test('ajax response with no embedded records yields empty table', function() {
    stubEndpointForHttpRequest('/api/others/', []);
    visit("/others").then(function() {
        var rows = find("table tr").length;
        equal(rows, 0, "table had " + rows + " rows");
    });
});

I’m using this for a full suite of integration tests around my django adapter and it’s rock solid so far. For anyone looking to see an example check out the branch for ember1.0

https://github.com/toranb/ember-data-django-rest-adapter/blob/ember1.0/tests/adapter_embedded_tests.js


#7

I use Sinon’s Fakeserver throughout my Ember integration test suite which works really well so if you can do this instead using Ember it would be great.

I have not looked at the response emitters pattern. Does this actually fake the http request the same as fakeserver does but just enables you to request and respond in a more controlled way / time ?


#8

Sinon comes with a lot of stuff that Ember folks won’t need. The approaches are similar to each other, but event emitting allows for exact control of where in the application flow a response is triggered.


#9

great +1 for that then cheers


#10

I think this is a tricky nut to crack, and I hope more people chime in with their ideas. Some thoughts:

Regardless of how XHR is stubbed or supplanted (with fixtures for example), having a robust factory language would be a huge boost. I have yet to use an API-dump for tests in practice, so I’m often stuck writing JSON responses by hand. When you have a lot of relations, this quickly becomes hard to manage. That said there are so many possible JSON variations right now (JSON-API, ActiveModelAdapter, RESTAdapter) that I’m not clear on how we could build one DSL. Perhaps the Ember-Data serializer API itself could be used for converting fixtures into response JSON?

Needing to type out all the URLs for an HTTP response also feels messy- The adapter knows how to generate the URL for a given request. Can we use adapter.buildURL to generate API routes?

My biggest issue with Sinon is how it fails. In autorespond mode, if you request a URL the fake server has no respondWith declaration for the server will just hang. This is way more frustrating than debugging an unexpected 404 or timeout. Usually a GET parameter has changed and you didn’t notice, then half the tests start hanging. Very frustrating to debug, since one hanging test obscures all other failures.

The proposal here looks more like Sinon’s respond flow. I’m a bit confused as to how it would work- Today the visit step is unfulfilled while XHR is processing (since the XHR is often in a promise for the model hook). How would specifying the http response after visit has resolved work? Does this imply that visit will no longer fulfill after the route has been entered, but at some earlier point? Does this have impact on other integration test flow?

One advantage of the FixtureAdapter is that you can actually save and update data on it. I would love to see that functionality baked into an XHR mock.

Maybe I’m really hoping for something halfway between fixtures and XHR stubbing. It would have factories generating data based on a DSL involving the model, be able to provide API urls, and do basic persistence (or maybe manually stub out a specific response’s behavior).


#11

Totally agree, but that’s a separate concern we’ll need to solve.

:thumbsdown: if there is an error in your buildURL that you rely on in your tests, your tests will pass even though there is an error. Plus not ever app uses ember-data, so we can’t rely on its presence.

That already exists, since it’s just a response:

  .click('.save-button')
  .httpRespond('post', '/images', { ... })

#12

Sorry for the self-promotion, but I created a little mock server for this sort of integration testing: https://npmjs.org/package/apimocker. It will return XML or JSON from a static file. You can also change responses at runtime, and set latency to test your asynchronous handling. Hope you find it helpful.


#13

Much like @toranb I’ve had great success so far using jQuery.mockjax as a way of XHR mocking for my integration tests at a really low level.

Using a slightly more complex test helper allows to not only set the status code of the response but also load it from an external JSON file:

// called with `mock_http('posts', 'posts/index');` to mock `GET /posts`
// or `mock_http('posts/1', 'posts/deleted', 204);` to mock `DELETE /posts/1`

function mock_http(url, fixture, status) {
    Ember.$.mockjax({
      url: url,
      dataType: 'json',
      proxy: FIXTURES_DIR + '/' + fixture + '.json',
      status: status || 200
    });
}

This way you can not only mock standard GET requests but also test successful PUTs, DELETEs, and, most importantly, error handling of all sorts. Having the ability to define those mocks on a per-test basis has been working well and using external files even allows for re-use on the server which will give you even more confidence in the tests.

I definitely agree that having a way of mocking HTTP requests built right into ember-testing is necessary to make integration testing more seamless and something that’s missing right now.


#14

I think the response emitters pattern looks nice. Right now I’m using Sinon and do find that setting up the endpoints outside the context of the interactions makes them feel a bit disconnected. It also seems like it would be a lot clearer if you’re expecting the same endpoint to return different responses over time, whereas trying to emulate that server state with Sinon can be a bit clumsy.

Having the option to turn on automatic failures for un-mocked requests would be really handy, as @mixonic mentioned those can be the source of some pretty annoying test bugs.


#15

When testing my Ember applications I generally use the fixture adapter. I think the fixture adapter is great for read based applications (lots of querying, little saving) since it is great at materializing models/relationships from data.

The biggest problem I have with the fixture adapter is that saving large graphs becomes difficult. If I want to save an model and I expect my API to sideload a whole bunch of changed data in the response this is really difficult with the fixture adapter. Currently, I’m extending the fixture adapter on a per model adapter basis and having create/updateRecord emulate these large changes while saving the record in test mode. This is hard to maintain though.

I think the HTTP mocking inline with the tests is a really cool solution. It’s great because if your API is really shitty (and lets be honest, most API responses suck) it allows you to test your adapter and serializer rules as part of your integration tests.

Something I am worried about is that the HTTP mocking ties our tests too close to the adapter. Switching from RESTAdapter to AMSAdapter means I could have to change a number of tests.

I’m using a factory in test mode to quickly build objects, but I really like the suggestion of having a factory that builds json/http responses. I think that alleviates my concern above, not having any actual “json” in your suite.


#16

OK friends, I’ve put something up for people to play around with. There are basically three parts:

Play around and let me know how it works.


#17

which library are you using?


#18

Just something we put together ourselves


#19

@trek thanks for putting together the mocking tools, I was looking into a bug and created a couple jsfiddles to compare, one using Sinon and second using your fakehr/FakeXMLHttpRequest:

Using Sinon: http://jsfiddle.net/pixelhandler/ePQ5x/

Using fakehr: http://jsfiddle.net/pixelhandler/7ZWZg/1/

I liked the fakehr.requests method it was easy to inspect what ember data was sending via ajax.

Would you think the fakehr fiddle example would be something helpful to add to the contributing guide here: https://github.com/emberjs/data/blob/master/CONTRIBUTING.md#reporting-a-bug ?


#20

You may want to checkout: teddyzeeny/ember-data-factory. I’ve just started using it, and it has been fitting my needs perfectly.