Verify that Ember can talk to the backend

1

I’m working on a large Ember project and am at the point where it’s hard to track whether the backend and frontend work together correctly.

Both the frontend and backend have unit tests with high coverage; the frontend also has acceptance tests. During development, the backend is also mocked. This means that currently, the frontend and backend have never spoken to one another, and that makes me nervous.

How can my development process be improved so there is more confidence that the backend and frontend can talk to one another? Should I stop mocking the backend during development? Should I implement contract tests, and if so, how? Are there any tools that would help with this?

2

Have a look at [https://github.com/realestate-com-au/pact](Pact testing). This is just one implementation, I’m sure there’s other ones as well.

The idea is that you set up a contract between consumer and producer and verify that it is satisifed.

The alternative is end to end tests, but - been there, done that, not a good idea, except for maybe 1-2 for your own sanity. But they are slow, a pain to set up and flaky.

3

Thanks for your reply. I had a feeling contract tests were a way to go. Am I wrong to feel nervous about never running the application itself in it’s entirety during development? Perhaps by deploying locally using something like vagrant or docker (a la continuous deployment)?

4

FWIW I always manually test my changes against a local version of the API, but I think that trying to do so automatically causes too much pain. Actually, I don’t only think that, I’ve experienced exactly that pain. :smile:

It’s actually fairly easy, you just run ember with

ember server --proxy http://localhost:9292

and it will use your local dev server running on port 9292 (or whatever you want).

5

after test it passed, then it’s your theme to see it in action. No need for test contract. So,you have confidence when see it really works as you expected

6

You’re saying I don’t need contract tests because I can run the application manually to verify that it works as expected? Have I understood you correctly?

If that’s the case, I’m afraid I disagree. Not only is manually testing time consuming it’s also prone to errors. It does have it’s place in the development life-cycle, but automated testing is preferred.

In answer to my own question, I’ve been reading Continuous Delivery by Jez Humble and it covers all the issues I’m experiencing. What my configuration is lacking is contract tests which run automatically and verify that the different components and services can talk to one another, and one-click deployment to a test environment that can be used for acceptance and smoke testing. I could also use this as a developer for a sanity check, but if the other aspects are well configured, I probably won’t need to do that so often.

7

We have a rails+ember stack and serving the ember app via rails (via ember-cli-rails – we have multiple ember applications to serve).

However, we’re doing feature tests via rspec and capybara (beside unit tests). It has some downsides, because the tests are slow (65 tests about ~1:35min) , but it’s worth to have the complete ensurance that frontend and backend plays well together if you really need it. Of course these are only high level tests, like “create a user” → should display one more user on the page + sends email + creates new user in database. Our first attempt was to test in ember and mock the server, but mocked server was always a bit behind the real API.

I also like the fact that when developing a feature where you have to switch between ruby & javascript all the time, that you can at least have the same language if you write the tests.

8

Great answer @nicolasd, thanks. Capybara sounds like the kind of thing I’m looking for. As I understand, this means you have your entire acceptance test suite written in Ruby and it’s separate from your (also Ruby) backend and (EmberJS) frontend?

Another question: how do you setup your environment for acceptance testing? According to the continuous delivery book, this needs to be a production-like environment. I would love to hear your experience on this.

9

Great, I hope it helps @hmblcodr . Rails + ember apps are in one repository, and we just have one spec folder where we have our unit and feature tests. So we don’t seperate anything. Btw: yesterday I tried to improve the performance of the feature-test-suite and was able to get it to ~47seconds for 65 tests (without parallelizing yet). there is probably more room for improvement.

Our local development environment isn’t in any virtual machine (yet) to simulate a production environment. But the test suite runs on travis, so it’s a bit more capsulated than on our local machines. We have our own sysadmin for setting up the chef scripts and server and as far as I know he also runs the tests on the production or staging environment before. - but I have no further info on that sry.

10

What kind of magic do you use for that? For our end to end tests even booting API + Ember takes at least half of that time.

We went down the road of end to end tests and it became just too painful. If I were to start a new app I wouldn’t ever consider this again. The worst offender are flaky tests. You can try all sort of hacks, like throwing in sleep statements or wait for all AJAX requests to complete but we still got a lot of random errors on the CI server.

11

For this particular performance boost I tried https://github.com/raggi/capybara-puma.

I know it can be extremely painful and it’s also not really possible to do TDD with feature tests and your point is totally legit. There are some use cases where these tests make sense - things that are extremely valuable for business (like checkout process - although mocking the stripe API + feature tests was also pain).

However, If the client wants it (mostly enterprise) you just do it. In the beginning we had some flaky tests, but now it doesn’t appear that often. We also have no problems with seleeps when using capybara’s have_content.

12

Yes, ok, I see that sometimes clients/management/whoever wants something badly and then you just have to do it. I just wanted to advise against it as a general solution.

Martin Fowler proposes the following solution: Timebox the time spent developing, maintaining and running your integration tests, if that takes to long, think about removing at least some of the tests and replacing them with other kinds.

I expect that you’re somewhat more experienced with test development (probably app development too), so that’s why you didn’t run into some of the issues I did, which is fair enough. It would at least make end to end tests much more bearable if there was some kind of framework etc. for developing them (I think Angular has Protractor, or so?). In my case, it’s just a handwritten overly complex Ruby script that gets all the necessary projects, allows to specify the various branches (sometimes you want to test against a specific version of the API, for example, or of Internal Library X that the API uses), does all the setup (DB migrations, bundle install, etc.), starts up both apps (and checks if the ports are actually available), then at the end shuts them down etc. That was quite nightmarish to write down in the first place and it’s still not perfect.

As mentioned, the bigger problem was with all the timing issues and even with have_content, waiting for AJAX, sleeping etc. it would not work in all cases. Then add the fact that writing tests for phantomjs is quite a pain in any case, it often behaves subtly differently from “normal” browsers (and that’s ignoring the issue of there being two versions of phantomjs, and that version 2, last time I checked, can only be installed from source on Linux which takes hours to do), and when you try to debug your failing tests with e.g. Selenium you often find that it actually does pass there. :confused:

Yes, Ember acceptance tests also use phantomjs and sometimes that also gives rise to problems, but in my experience much less. It seems the default waiting behaviour etc. is much better handled in that case. I get the feeling that capybara, while great for your usual multiple page app, is wholly inadequate for handling AJAX-heavy pages, although I cannot exactly pin down the reason.

Thanks for the pointer to capybara-puma though.