Ember-cli, testem + qunit, and saucelabs integration tests with Saucie

bennysghost · October 28, 2014, 8:29pm

I would like to get

ember test ci

to work in a saucelabs continuous integration environment so I can test multiple windows/mac browsers with my development website.

I found Saucie (https://github.com/igorlima/sauce-js-tests-integration) which supposedly runs testem qunit tests in Saucelabs via saucelab-connect. However, there are certain challenges I’m facing due to the integration of testem with ember-cli. For example:

The instructions at Saucie say to do some setup to launchers (in testem.json) and add saucelabs credentials to my environment, then run: testem ci --port 8080. This doesn’t work with an app built with ember-cli (ember path issues). The equivalent ember test ci --port 8080 also does not seem to run Saucie
I tried running Saucie manually after setting up a ember server but I still run into 404 errors due to pathing problems, so no tests are actually run. Here’s what I’m doing. Note all the 404s in the log.

ember test --server --port 8080 then

saucie --browserNameSL=“internet explorer” --versionSL=“7” --playformSL=“Windows XP”

produces a log like:

Opening local tunnel using Sauce Connect
Sauce Connect 3.1-r32, build 50
* Debug messages will be sent to sauce-example.log
.---------------------------------------------------.
|  Have questions or need help with Sauce Connect?  |
|  Contact us: http://support.saucelabs.com/forums  |
|  Terms of Service: http://saucelabs.com/tos       |
-----------------------------------------------------
2014-10-28 11:10:49,552 - / Starting \
2014-10-28 11:10:49,555 - Please wait for "You may start your tests" to start your tests.
2014-10-28 11:10:49,569 - Forwarding: None:['80'] -> 127.0.0.1:['63383']
2014-10-28 11:10:49,585 - Succesfully connected to local server 127.0.0.1:63383 in 6ms
2014-10-28 11:10:50,038 - {"squid_config":null,"no_ssl_bump_domains":null,"use_caching_proxy":true,"metadata":{"PythonVersion":"2.5.1","OwnerHost":"127.0.0.1","Release":"3.1-r32","OwnerPorts":["63383"],"Ports":["80"],"Platform":"Java-1.6.0_65-Java_HotSpot-TM-_64-Bit_Server_VM,_20.65-b04-462,_Apple_Inc.-on-Mac_OS_X-10.9.4-x86_64","Build":"50","ScriptRelease":50,"ScriptName":"sauce_connect"},"use_kgp":true,"tunnel_identifier":"Saucie","shared_tunnel":false,"fast_fail_regexps":null,"ssh_port":443,"direct_domains":null,"vm_version":"","domain_names":[]}
2014-10-28 11:10:53,368 - Tunnel remote VM is provisioned ([OMITTED])
2014-10-28 11:10:53,543 - Tunnel remote VM is booting ..
2014-10-28 11:11:18,645 - Tunnel remote VM is running at maki77091.miso.saucelabs.com
2014-10-28 11:11:18,657 - Succesfully connected to local server 127.0.0.1:63383 in 1ms
2014-10-28 11:11:18,661 - Starting connection to tunnel host...
2014-10-28 11:11:18,661 - Connecting to tunnel host maki77091.miso.saucelabs.com as [HOSTNAME]
2014-10-28 11:11:18,760 - Forwarding Selenium with ephemeral port 63390
2014-10-28 11:11:18,766 - Selenium HTTP proxy listening on port 4445
2014-10-28 11:11:19,020 - Successful handshake with Sauce Connect server
2014-10-28 11:11:19,025 - Tunnel host version: 0.1.0, remote endpoint ID: [ENDPOINT]
2014-10-28 11:11:19,028 - Connected! You may start your tests.
Testing tunnel ready
Started Sauce Connect Process
Starting run all tests against Sauce Labs
2014-10-28 11:11:28,381 - Request started: GET http://localhost:23646/
2014-10-28 11:11:28,391 - Could not proxy http://localhost:23646/, exception: java.net.ConnectException: Connection refused
Sauce Connect Error
2014-10-28 11:11:29,400 - Request started: GET http://localhost:8080/
2014-10-28 11:11:29,404 - GET http://localhost:8080/ -> 302 (6ms, 90 bytes)
2014-10-28 11:11:29,443 - Request started: GET http://localhost:8080/tests/index.html
2014-10-28 11:11:29,447 - GET http://localhost:8080/tests/index.html -> 200 (5ms, 1566 bytes)
2014-10-28 11:11:29,497 - Request started: GET http://localhost:8080/tests/assets/vendor.css
2014-10-28 11:11:29,503 - GET http://localhost:8080/tests/assets/vendor.css -> 404 (5ms, 139 bytes)
2014-10-28 11:11:29,545 - Request started: GET http://localhost:8080/tests/assets/[OMITTED].css
2014-10-28 11:11:29,549 - GET http://localhost:8080/tests/assets/[OMITTED].css -> 404 (5ms, 146 bytes)
2014-10-28 11:11:29,588 - Request started: GET http://localhost:8080/tests/assets/test-support.css
2014-10-28 11:11:29,594 - GET http://localhost:8080/tests/assets/test-support.css -> 404 (5ms, 145 bytes)
2014-10-28 11:11:29,637 - Request started: GET http://localhost:8080/tests/assets/vendor.js
2014-10-28 11:11:29,641 - GET http://localhost:8080/tests/assets/vendor.js -> 404 (4ms, 138 bytes)
2014-10-28 11:11:29,683 - Request started: GET http://localhost:8080/tests/assets/test-support.js
2014-10-28 11:11:29,687 - GET http://localhost:8080/tests/assets/test-support.js -> 404 (6ms, 144 bytes)
2014-10-28 11:11:29,727 - Request started: GET http://localhost:8080/tests/assets/[OMITTED].js
2014-10-28 11:11:29,732 - GET http://localhost:8080/tests/assets/[OMITTED].js -> 404 (5ms, 145 bytes)
2014-10-28 11:11:29,770 - Request started: GET http://localhost:8080/tests/testem.js
2014-10-28 11:11:29,775 - GET http://localhost:8080/tests/testem.js -> 404 (4ms, 131 bytes)
2014-10-28 11:11:29,825 - Request started: GET http://localhost:8080/tests/assets/test-loader.js
2014-10-28 11:11:29,829 - GET http://localhost:8080/tests/assets/test-loader.js -> 404 (4ms, 143 bytes)
2014-10-28 11:11:29,927 - Request started: GET http://localhost:8080/favicon.ico
2014-10-28 11:11:29,933 - GET http://localhost:8080/favicon.ico -> 404 (5ms, 127 bytes)
not ok
Check out test results at http://saucelabs.com/jobs/d5353591620a46699756f9f2d34350e5

2014-10-28 11:11:32,174 - received SIGTERM
2014-10-28 11:11:32,180 - Shutting down tunnel remote VM (please wait)
2014-10-28 11:11:35,335 - Finished running tunnel
2014-10-28 11:11:35,871 - Finished shutting down tunnel remote VM
2014-10-28 11:11:35,872 - \ Finished /
Closed Sauce Connect process
Shutting down
Closing Sauce Connect Tunnel

Can anyone nudge me in the right direction? I’m open to using other testing solutions that can incorporate saucelabs. From what I’ve read though, the cli testem qunit solution is preferred due to the support for the async-ness of ember.

johannesw · January 9, 2015, 8:40pm

A bit late, but I created an ember-cli addon for this: GitHub - johanneswuerbach/ember-cli-sauce: Cross browser testing for your ember-cli app using SauceLabs

Hope this helps.

amyrlam · December 29, 2015, 11:47pm

Hi! Also trying to get Sauce Labs working with an Ember app.

(1) I have a Sauce Tunnel set up in Travis by adding this to my .travis.yml:

addons:
 sauce_connect: true

The Sauce tunnel opens when I push to GitHub, which is already linked to Travis.

However…the tests that run in Travis (Ember QUnit / Mirage tests) aren’t running in Sauce.

(2) I tried adding this add on GitHub - johanneswuerbach/ember-cli-sauce: Cross browser testing for your ember-cli app using SauceLabs and using that instead.

It opens up a Sauce Tunnel successfully.

I added this to my testem.json to try testing one browser in Sauce:

  "launchers": {
    "SL_internet_explorer_public_11_0_Windows_10": {
      "exe": "ember",
      "args": [
        "sauce:launch",
        "-b",
        "internet explorer",
        "-v",
        "11.0",
        "--vi",
        "public",
        "-p",
        "Windows 10",
        "--at",
        "--no-ct",
        "--u"
      ],
      "protocol": "browser"
    }

I’m still not getting anything automated running in Sauce. I tried running ember start-sauce-connect and then ember test in my terminal but no luck.

Stuck, any guidance on how to move forward?

cspanring · January 6, 2016, 4:13pm

Does your testem config also have the sauce labs browser in its launch_in_ci property?

...
"launch_in_ci": [
  "SL_internet_explorer_public_11_0_Windows_10"
],
...

amyrlam · January 7, 2016, 2:26am

I got this working with Travis. With the ember-cli-sauce plugin, I need to run ember start-sauce-connect and then ember test --launch='SL_internet_explorer_public_11_0_Windows_10'.

I think I can get that working in Travis, but am woking on some new issues now… our test suite is ~1000 tests (Acceptance, Unit, JSHint, JSCS) and it times out on Sauce Labs (and sometimes Travis too).

Appreciate you chiming in, @cspanring!

Topic		Replies	Views
How are you testing your Ember app? Testing	2	1775	September 13, 2014
Request for help with testing setup Testing	5	3739	December 18, 2014
How can I run Ember integration tests using Testem?	3	2974	May 28, 2014
Ember Automated Testing Tools Testing	1	942	November 8, 2018
Using chai-qunit on top of ember-cli's ember-qunit? Testing	3	1952	May 15, 2015

Ember-cli, testem + qunit, and saucelabs integration tests with Saucie

Related Topics