Why was Ember 3x - 5x slower on Android?

Performance
1

One thing we have struggled with for a year at Discourse is how slow Ember is on Android devices. How slow? See for yourself:

Ember 1.8, Complex List test: http://samsaffron.github.io/ember-performance/?ember=1.8.0

iPad Air 2 – 224 ms
iPhone 6+ – 252 ms
iPhone 5s – 300 ms
iPhone 5 – 600 ms
iPhone 4 – 2031 ms
Galaxy S5 – 1297 ms
Nexus 5 – 1426 ms
Nexus 7 – 1930 ms

Surface Pro 3 Core i5
IE 11 (x64, touch) – 613 ms
Firefox latest – 312 ms
Chrome latest – 260 ms

Desktop Core i7-4470k
IE 11 x32 – 390 ms
Firefox – 245 ms
Chrome – 181 ms

Intel Bay Trail Atom Asus T100 1.33 Ghz Z3740 IE 11 (x64, touch) – 2.2 sec
Firefox latest – 1.6 sec
Chrome latest – 1.2 sec

There is a year old bug open in Android Chromium on this…
https://code.google.com/p/v8/issues/detail?id=2935

I had some hope that Android 5 / L would fix this somehow as the old creaky Dalvik was replaced with native code. But people have replied with preliminary Lollipop results and they look no better, so I am no longer optimistic. I do have a Nexus 9 ordered, which is as good as it gets for the forseeable future, and I will report in when that arrives.

It’s been a year now since we originally observed the problem. Ember performance is great on iOS and laptop and desktop, and improves every year as hardware gets faster… but horrendously bad 3x - 5x slower Ember Android performance, for better or worse, seems to be the long term status quo. Is there anything Ember can do to help address it? Either by nudging people at Google or identifying exactly why the very same Javascript is so incredibly slow on every Android device?

12 Likes
2

Do you have numbers for Firefox on Android? I would be interested to see if it’s something about the hardware platform, or specific to Chrome.

3

I only have my Nexus 7 at hand:

Google Chrome beta – 1585 ms
Firefox stable – 2443 ms

The standard deviation on these 40 runs is high, around 260ms for Chrome and 340ms for Firefox.

4

Maybe it has something to do with ARM, and iOS Safari is just properly optimized?

Have you tested an ARM Surface?

5

Someone on a different Discourse instance produced this number:

Surface 2 RT, IE11 – 1747 ms

So between the Nexus 5 and the Nexus 7 performance wise.

6

So in other words there is a good chance that these issues are just related to the JITs not being well-tuned for ARM in general, with the exception of iOS Safari.

7

Did you look at the bug report? I suggest you look at the bug closely.

It seems very unlikely to me that every other ARM team except for Apple’s is … uh, incompetent? I guess, maybe?

8

Ive been noticing atleast a factor 3, often more difference in JS performance between the same generation android / iOS devices (samsung exynos octacore vs apple A7). The gap between apple devices and same generation androids can be huge if you pressure certain areas (shader compiler speed, gpu speed, JS JIT speed) The general public might not notice with simple webpages or apps, but if you push the limits its extremely clear.

9

I suspect not incompetent, but the wide array of different hardware configurations likely makes this significantly harder.

1 Like
10

Yes we can, and are working towards overall improved performance. The limiting factor is just time.

A thorough Android performance investigation would be great. Although I have done similar work for Mobile Safari (likely because I own various iDevices but no android ones) I have not had the time or opportunity to do the same on Android. If someone can get me the time, I will gladly dig in.

Facts I am aware of:

  1. other then some annoying bugs, JSC + Safari are really killing it in terms of real world performance (both desktop and mobile)
  2. V8 for androids currently gets steam rolled by JSC pretty much regardless of framework
  3. There exist a huge number of potential ember performance improvements that will yield cross platform improvements.
  4. 1.7 + 1.8’s use of an AMD style loader really hurts mobile (and especially androids) performance. The solution is to continue the effort to upgrade to @eventualbuddha’s latest es6 transpiler, so we can take advantage of bundled builds.
  5. 1.8, 1.9 and 1.10 are meant to incrementally (as we complete it) role out our new view rendering system. The first step 1.8 takes a known performance hit, to normalize handlebars into something htmlbars would emit. This is very likely (but obviously not proven on android) the majority of any performance regression we are seeing.

Historically, ember has only done performance work in an ad-hoc way. This increases the barrier of entry for performance work for both core contributors and the community at large. This also results in wildly inaccurate benchmarks #'s because they have not had sufficient community scrutiny and this inherently does a poor (nearly non-existent job) of related knowledge sharing. Finally, ad-hoc prevents running performance regression tests.

This is clearly poor but a problem I have been noodling on for quite some time.

What we need is.

Reliable benchmarking that is representative of real world use-cases and app level performance concerns. Simply running X, N times may be sufficient for micro benchmarks but not the macro benchmarks I believe we are desperately lacking. Basically, given user action Y, what N complexity system still renders at 60FPS or 30FPS or … This should allow us to better understand what is actually slow, what is sufficiently fast and will likely illuminate a triage priority.

Another note about micro benchmarks against app level concerns, running a user action N times, will often skew results as actions that normally wouldn’t become JIT’d or actions that normally deopt infrequently now often skew results as N increases.

Ideally ^^ style of tests could just be part of discourses normal development process.

In addition to this, micro-benchmarks that cover targeted parts of the framework. Ember.Object.create + actionsFor etc.etc. will continue to be valuable.

And finally these tests need to be actionable.

  1. able to run on multiple platforms, mobile/desktop/ and likely cross evergreen browsers
  2. easily profitable
  3. point to real problems, not benchmark induced GC pressure.
  4. stable re-runs over sufficient N should result in consistent numbers.

An early quick, helpful + actionable effort has been: GitHub - stefanpenner/perf-stress which has allowed me and others to dramatically improve many ember + ember-data performance issues. (I suspect 1.8 would be in a much worse situation without that effort). This obviously suffers from the lack of automation.

This weekend, I decided to give angular benchpress project a try: spike with benchpress · stefanpenner/ember.js@820a8d6 · GitHub it is interesting, but really is lacking especially when it comes to delivering the metrics i described above. Also, it doesn’t have a concept of asynchronous tests, so many real-world scenarios become impossible. It’s also suffers from the run X N times sequentially problem, which really isn’t a natural way to interact with an application and doesn’t give me the #'s I want.

I will likely use this as some form of inspiration and transform it into something I believe will be helpful, I have made a short todo-list spike with benchpress · stefanpenner/ember.js@820a8d6 · GitHub

Since then I have also begun spiking on a enhanced test runner that actually yields the numbers and metrics we care about. Hopefully I will have something working an on GH in the next few days.

/end stefan.brain_dump

TL;DR performance needs to get better.

5 Likes
11

Also proper automation will likely help reduce the likelihood of mobile regressions as it will lower the cost of running against a suite of browsers and devices

12

We’re trying to land HTMLBars to master in the next five weeks, but timing is tight and all nights & weekends driven.

13

Great! It is entirely possible this has nothing to do with Ember; I just wanted to make sure everyone was aware of the problem and how long it has gone on. We’ve known about the severe Android performance disparity for a year now, and over time it has gotten worse and worse as iOS performance naturally improved on newer, faster hardware and Android performance… didn’t.

Unfortunately, it is becoming a very severe problem over time, enough that it should give people adopting Ember serious pause – if you have a lot of Android users on your Ember app they will suffer from 3x to 10x slower client side performance. That is not an exaggeration, it is a benchmark able fact: an iPhone 6 is literally ten times faster than the 2013 nexus 7 (or any 2013 Android hardware) in Ember. Even a brand new Samsung Galaxy 5 flagship phone is 4 times slower!

The last time @eviltrout looked at this, there was no smoking gun on the Ember side, no one thing he could point to that was particularly slow in Android. We hoped removing recursion and metamorphs e.g. metal views might help, but as we are testing on 1.8 that clearly is not the case.

I think the best outcome is to publicize this performance disparity more and really drive Google to fix the bug they filed on this a year ago: https://code.google.com/p/v8/issues/detail?id=2935

2 Likes
14

I wouldn’t go that far. We’ve shown that Ember is much slower on Android, but I use a Nexus 4 every day and Discourse doesn’t feel significantly slower than other web apps.

If we could show that Ember in particular is slower than all other web applications on Android that would be one thing, but it really seems to me that JS is just piss poor in general on that platform.

3 Likes
15

@jatwood it is worth noting, that in your last post, each use of ember can legitimately be replaced with the web. Android users legitimately have a poorer web experience.

This doesn’t mean we don’t want to improve the performance of ember, and doesn’t mean won’t. But it does mean that until android steps it into high gear, iOS users will continue to have a better browsing experience.

I believe we can all agree the resolutions are:

  1. investigate android performance characteristics further
  2. given 1. we may be able to prioritize various planned ember performance work
  3. work with the v8/chrome and google dev-rel teams to improve this.
  4. regardless of platform continue to make ember faster (likely via what I proposed in my last post)

It is pretty funny although the v8 team despises micro-benchmarks, they are almost always the winners in such benchmarks. But when it comes to macro benchmarks (real world usage) JSC/Safari is really doing much better lately.

4 Likes
16

You are crazy. For 9 months I was using a marginally faster (for Ember) Nexus 5 which renders in 1.4 vs 1.7 on the above benchmark and I wanted to claw my face off when reading Discourse on it. Regular 5 second topic load times, versus nearly instant (1 second) on an iPhone 5s and practically desktop speeds on the iPhone 6.

There is a ton of experimental data supporting the idea that speed directly equates to more usage…

[Google found that] the page with 10 results took 0.4 seconds to generate. The page with 30 results took 0.9 seconds. Half a second delay caused a 20% drop in traffic. Half a second delay killed user satisfaction.

In A/B tests, [Amazon] tried delaying the page in increments of 100 milliseconds and found that even very small delays would result in substantial and costly drops in revenue.

And a performance difference of 5x ain’t exactly subtle. If it was 20% or even 50% slower, who cares. But five times slower?

I guess if Android users don’t know any better, but five second page load times are extremely poor even relative to most other common websites. So when they get a visible Discourse site, an Android user would think, oh, this is going to be super slow again, why don’t I try somewhere else?

Can’t say I blame them.

Great experience on iOS though, no question.

17

By far the best outcome, definitely what I recommend. Feed them traces and other low level artifacts of the problem. I agree that the ball is definitely in Google’s court here, but the overall chilling effect on the ecosystem (Ember is a terrible experience for Android users) has negative, and growing more negative every day, consequences for this project.

18

Yea this is a bit of concern of mine because I have worry about markets like India where they have poor infrastructure (not my problem). To add the infrastructure problem, the fact is that they are large users of lower powered Android phones. This is what a Moto G (rather common) on a 3G network in India looks like. This isn’t an extremely complicated application either. These benchmarks aren’t scientific by any means but what we have observed.

1 Like
19

I meant other web apps on Android. What evidence do we have that Ember is slower on Android than other web apps on the same platform?

It is dangerous to call out Ember as the culprit here unless we can show that there is something about Ember that is much slower on Android than other apps on Android.

20

I can confirm that other web apps on Android are pretty crappy as well. Here is a production quality mobile Backbone application on the same device.

2 Likes