Synthetic benchmark: comparing Ember 1.12.0, Ember Canary, and React

Performance
1

Hi all,

Ever since I saw this React demo, I’ve been looking to reimplement it in Ember to see how the different approach to re-rendering DOM elements would affect performance.

I finally got to do just that. There are two versions: a demo that uses Ember 1.12.0, and the same demo using Ember Canary. I have embedded the original React demo there to allow for quick eye-balling of performance when doing the same operations using different views. The complete source code for the demo is on GitHub.

Points of interest:

  • If you click on “Add Many Items”, you’ll get 1000 elements added to the page. You can see the difference between Ember 1.12.0 and React when dragging the width slider of any item while there are a 1000 other elements on the page. Ember works much smoother here, which is not at all surprising given its 2-way bindings vs React’s diffing.

  • React shows better performance when adding and removing elements.

  • To my surprise, Ember Canary shows worse performance in all possible respects in this demo. For each view, I first add many items, then drag the width slider of any of them to eye-ball the lag. After that I remove a few elements individually, then click on “Remove All Items”. After having done that three times, I get all of the timings filled into the table on the page.

    Here’s the table for Ember 1.12.0. (Note: the timings are not 100% accurate, most of them are more optimistic than the perceived performance).

    And here’s the one for Ember Canary:

    Moreover, dragging the width slider is noticeably more sluggish with Canary.

With that, I have a few questions:

  • Does Ember Canary show worse performance due to it being an as of yet unstable branch? Or is it something I’m doing wrong in my code that could cause additional overhead, esp. in how Canary compares to 1.12.0?

  • Where can I find benchmarks demonstrating the use cases where Glimmer shows superior performance compared to Ember 1.12.0?

  • I would greatly appreciate it if someone knowledgeable in Ember’s internals could point out the biggest contributors to Ember’s bigger overhead in this demo.

Disclaimer: I’m an Ember newbie, been learning it for the last few days. I don’t know of any feature flags or optimizations that could be turned on to speed things up a bit for Ember. I tried to describe everything there is to know about this demo in the project’s Readme.

Best,

Alex

3 Likes
2

One thing I’ve learned is that when benchmarking, run your tests in a production mode, ie ember --environment production. Development mode has a lot of expensive assertions and operations that do not occur in production. It’s generally a few magnitudes slower.

Also I see your tests using DateTime for timing your benchmarks. You should use the Performance API instead: performance.now() - Web APIs | MDN

I also don’t know if your use of Ember.run.schedule will result in accurate numbers.

/cc @stefan who has a ton of experience around benchmarking Ember

3

Hey Jason, thanks for taking a look at the code.

I only added automatic time measurements to be able to show a quick screenshot that makes it obvious that there’s a difference in performance. They were never meant to be precise measurements. In fact, I have created another branch called no-timings so as not to steer the discussion in that direction (consequently, there’s now less code).

When you’re interacting with the demo, it is easily noticeable how adding and removing items and dragging the width slider of any item feels different between Ember and React, and there’s an especially noticeable difference between Ember 1.12.0 and Ember Canary (with the latter showing worse performance).

I was’t able to build the project with the production environment, unfortunately. I get some cryptic error related to emeber-cli-uglify.

Has anyone got a chance to clone the repo and try comparing the Ember 1.12.0 and Canary built with production env?

Cheers,

Alex

4

I did some more tests, and got these numbers - DBMonster-Performance - Google Sheets

More analysis here - Parashuram's blog: Performance Comparison of Javascript Frameworks - DBMonster App

Uses the same dbmonster app for testing.

1 Like