While developing ambitious web applications with ember, I’ve found that they often naturally result in databases with large collections of records (> 10,000). Obviously, it’s impractical to transmit and load all of these records to the client, which means ambitious web apps almost always need a solution for handling collections that are impossible to fully resolve into a complete list of records.
Here’s the problem: ember-data doesn’t really address this issue, especially when it comes to hasMany relationships. In fact, ember-data tries to fully resolve a hasMany as soon as it is accessed with get, and then once the promise resolves, ember-data assumes that it has all the related records. This behavior can be clunky and does not work for large collections that cannot be resolved. To avoid this behavior, I’ve had to avoid hasMany completely and hack together my own class for dealing with large relationships/collections and preventing ember-data from trying to resolve them automatically.
I can definitely understand ember-data’s approach here, to defer this problem to the developer (while providing methods like query, getReference, and peekRecord to make ambitious developers’ lives easier). Large record collection management requires pagination, filtering, and sorting to work properly, and all three of these techniques are often implemented slightly differently depending on the back-end API.
Nonetheless, JSON API does have recommendations for how to handle pagination, filtering, and sorting, and it would be great if ember-data catered to this recommended structure, at least.
In short, I think ember-data is missing a good solution for managing large record collections. When I encounter a large record collection in the wild, I avoid ember-data’s built-in collection management completely in order to prevent ember-data from trying to load all the records or getting confused when it can’t fully resolve them. A good solution should support JSON API’s recommendation for pagination, filtering, and sorting; it should never try to fully resolve the collection; and it should maintain record parity with ember-data’s store.
If there’s legitimate interest in this, I can try to sketch out an interface for a class that handles large collections and share some of my old attempts at hacking together a solution.