In my current app, Ember Data is a lifesaver…
BUT…
If I try to load say, 4000 or something… it will literally hang the browser for multiple seconds processing the server result.
Any thoughts on how to handle this any better?
@stefanpenner has been making GREAT progress lately on performance using Ember Data with large payloads. You might want to try using the canary builds and see how that helps.
3 Likes
Hi Roger,
In regards to your backend do you have the ability to paginate the collection of resources you are trying to load? This in the long run would be the most scalable solution if your resource collection continues to grow (what happens if you have 500,000 records?). You could simply use page/limit parameters to define how many you want to load. Then in the result include the total # of records so you can setup a pagination component and query the endpoint as you see fit.
If this is not an option I would suggest you follow Ember best practices regarding mass object/model creating using Ember.beginPropertyChanges() and Ember.endPropertyChanges() when dealing with large collections of objects. This will help prevent observers/properties that watch the record array from firing upon each individual object creation.
I would implement loading using server paging in my route or controller and then use a lazy loading technique to load the data when needed
Thanks for all the replies.
I do actually have pagination, and I can load it in parts etc if I have to.
This is a case where I really do want like… 4000, 10,000 etc models of a particular type all pre-loaded (imagine trying to visualize a giant multi-region AWS infrastructure).
Memory, once loaded, isn’t an issue at all – it was literally the dead-locked browser as I broke a thousand-ish objects in one big chunk 
1 Like
Awesome that this is a known concern and actively being optimized. We ran into this the past week loading 6000 records with circular references (hasMany with a belongsTo back to the owner). It took about 15 seconds to unblock the UI after the data was received.
I have found ember data to be too slow for this in the past, on the order of 1ms for each loaded record. It would be good if this has improved now in recent builds, I have not benchmarked recently.
Other options are:
- Loading the JSON manually and then pushing in to the store a few records at a time - I think you could probably abuse requestAnimationFrame for this to avoid locking up the UI
- I found ember-model to be an order of magnitiude faster than ember data for loading records, with a similar API
- Not using a model library at all and using POJOs is the fastest solution and may be required for your use case
Interesting - if the browser is locking up have you tried profiling the javascript and looking at the flame graph? This may be your best bet as you can see exactly what is taking the longest. Rather than randomly guessing and doing premature optimizations.
I wonder if Ember Data supports streaming JSON?
1 Like
I wonder if this is something were you can use web workers since your problem isn’t really loading all those records, but the browser blocking the UI for the user.
Wouldn’t know if it’s possible but the idea would be to load those records via a web worker and show a simple loading animation while it loads all those records.
Yeah, I think it does right? I feel like I’ve done it before but I can’t remember. Using store.push or something like that.
You don’t have access to the store inside a webworker right?
Properties do not “watch” anything. They are computed on demand.
Web workers won’t work here, because they have no access to the rest of the world, notably the Ember store into which you want them to load the records.
Web Workers will work here, you just have to wire up a little registry within them.
Could you clarify what kind of registry you have in mind, and how you would wire it up?
Do you have an example of creating such a registry and how to get the data back to the main thread?