findRecord with streaming data

Ember Data
1

I have an adapter and serializer set up to stream data over a websocket. The general idea is that when store.findAll('model-name') is called, a web socket is connected, and data is streamed back. Updates over the channel are pushed into the store, creating new records without ever hitting the server again.

This all works well, but the problem that I’m running into is that somewhere in the application, we trigger and event on the server, which creates a new object, and sends us the id. Then the application must take that new record and do something with it. Here lies a race. If the data is received quickly enough and pushed into the store, then the store.findAll('model-name', 'id') works as expected and just looks the record up locally (shouldRefetch and shouldBackgroundRefresh have been set to never hit the server again). If the data hasn’t been loaded yet, then the server is hit for that one object, even though all I really want to do is wait for the object to show up on the stream.

I came up with a partial solution:

  1. When a findAll is done, I set a flag that streaming is active, and add an ArrayObserver to the type’s LiveRecordArray.
  2. When a findRecord is done, but the ID is not found, it returns a promise which is stored in a map by type and id.
  3. On add LiveRecordArray events, I take the materialized record, and see if the id has an outstanding promise in a map, and resolve it with the record

The problem now is that all of this is being done in the adapter, and the adapter’s findRecord method is expected to return the payload from the server, ready to be passed into the serializer, which then returns JSONAPI ready to be pushed into the store. I already have the full record though, and really just want that to bubble all the way back to the store.findRecord call.

I see a few methods forward, all of them are problematic:

  • Remove the array observer, and instead hook into the data pipeline from the server. Extract data that is needed to resolve the adapter.findRecord, and resolve the promise with that.
    • This will be very messy trying to manage relationship data, as some relationships are async and others not. The sheer introspection into the model will take a toll on performance which is already a pain point.
  • Reopen store.findRecord to check the type of the promise result and skip the serializer entirely when a record is returned instead of server data.
    • This is very brittle, and is just asking to break on any given update. We would have to maintain a near mirror of Ember-Data code to keep it working.
  • Make serializer.normalizeResponse aware of this, and if it receives a record as a payload, then it generates a dummy return of {data: {id: record.id, type: record.constructor.modelName}}, which should end up getting the store.findRecord resolved with the record as intended.
    • Now the adapter and serializer are coupled and just in general, this seems silly.

Right now I’m leaning toward option 3, but it seems like an interesting enough situation that I’m curious how others would handle it.

Thanks for any feedback, hopefully I didn’t ramble too much.

EDIT: I ended up going with option 3 like I had initially planned. It’s very unlikely that we would ever be able to swap out for a standard serializer with our streaming updates anyway. So coupling the serializer and adapter didn’t seem like too bad of an option.

4 Likes