Ember data partial loading + merging

I spent the last half a day battling with Ember Data over partially-loading models in my app. There’s quite a lot of discussion about partial loading, with some OK solutions, but I think my situation is a little different and raises some more issues. Here’s my scenario:

A “discussion” hasMany “posts”.

I have a list of discussions on a search results page. These are fetched at the /api/discussions?q=whatever endpoint. The “posts” relationship is not loaded at this stage. However, as a feature of the search results, a hasMany “relevant_posts” relationship is included, showing a couple of posts which are relevant to the search terms.

Clicking through to a discussion caches/stores the search results for later. Then, the discussion model is reloaded in order to get its “posts” relationship. At this point, even though the newly-loaded discussion data does not contain any information about “relevant_posts”, (i.e. the key is not even specified in the endpoint JSON), Ember Data overwrites the whole dataset and wipes the relationship.

Returning to the search results page retrieves the results that were cached before. This is essential in my application for interface snappiness (as opposed to having to wait for the results to load again.) However, because the “relevant_posts” relationship was cleared for that one discussion, it has now lost its relevant posts.

So I’ve done quite a bit of reading about partial loading. The general consensus from the ember-data team around partial loading seems to be "this is something that we’ve never claimed to support, but maybe one day.”

One workaround idea that people have suggested is to use two different models: have one for the discussion search results, and one for the actual discussion. A couple of problems with this:

  • Unnecessarily awkward stuff in adapters/serialisers to make a request for “discussionResults” models still hit the “discussions” API endpoint, and then translate the returned “discussions” into “discussionResults", but only for requests made to get results in the first place. (Changing the API is not an option.)
  • Loss of synchronisation between discussions in the context of search results and on the discussion page. For example, if I click on a discussion, update its title, and then go back to the search results, the title change would not be reflected.

What I really would like to be able to do is to specify that I want to merge data with the already-existing data when I call reload on a model — or any kind of find method on the store for that matter.

A workaround for this is to make an AJAX request manually, and then manually call update on the store with the returned data (like in the end of this article.) But this feels pretty flimsy; I don’t want to have random AJAX requests scattered around the place.

I thought this would be a good opportunity to initiate a discussion about how partial loading and merging might be properly implemented in Ember Data’s API.

At least for my use-case, when pushing new data into the store, the new data would need to be merged with the old data instead of replacing it completely. Any key specified in the new data should take precedence; any key not specified in the new data should remain from the old data.

I dug into the Ember Data source and determined the main places this change would need to be effected:

  • The JSONSerializer’s applyTransforms method needs to be amended so that attributes that aren’t present in the data hash aren’t transformed (and therefore set to undefined in the data hash.) i.e. add if (typeof data[key] == 'undefined') return; to the start of the type.eachTransformedAttribute loop.
  • Public APIs (store.find, model.reload, etc.) would need to be amended to allow an additional boolean argument called “merge”, which would be passed through private methods until it reaches the store.push method, which already has a private _partial argument to do this kind of thing.

This way, partial data for a model could be loaded initially, and then more partial data could be merged with that subsequently, as follows:

var discussions = this.store.find(“discussion”, {q: “keyword”}); // get partial search result data containing relevant_posts
var discussion = discussions.objectAt(0);
discussion.reload(); // current behaviour: get completely fresh discussion data, replace existing data
discussion.reload(true); // get additional discussion data, merging it into existing data

I haven’t really thought about any other use-cases though. I’d be very interested to hear some thoughts about this from people with more Ember Data experience.

This is an area that ember-orbit has really solved properly, it uses json patch for updates so the data is merged nicely as it becomes available. I’d be interested to hear what the ember-data team’s thoughts are on it and whether any consideration had been given to a merge of some sort down the road.

Orbit looks really cool. I’ll keep an eye on it.

After a bit more thinking, I realised I was conceptualising my scenario incorrectly and coming to the wrong conclusions about what the solution should be.

I realised that in my app, the relevant_posts relationship is not actually a real property of a discussion entity. Rather, it is application state-specific data, linked to the search results view instead of the discussion itself. Therefore, I shouldn’t be trying to store it as part of the ember-data model, and I shouldn’t blame ember-data when it gets overwritten.

My solution was to create a new object called DiscussionResult which extends Ember.ObjectProxy and has a relevant_posts property (an empty array.) After the search results are loaded, I hydrate new DiscussionResult objects with the discussion model as the content, and pull out posts from the model’s relevant_posts relationship into the DiscussionResult’s relevant_posts array.

This way, the discussion data itself stays in sync, but the relevant_posts data is independent of the model. And yet, because of the magic of ObjectProxy, it behaves just like a normal model.

I love discovering these tidbits of awesome that Ember makes available to you!

@tobscure, could you post the code for your ObjectProxy sol’n – I’m new to the technology and I’m having a hard time w/ the abstractions.