Ember data in an on- and off- line scenario


#1

I am building my first ember application and trying to decide how best to architect the data access… The application will run on mobile devices and needs to run offline. When it is online it needs to do some simple sync and make sure its local database is synced with the server.

My first plan was to use a LSAdapter when the app is offline and then to write custom sync code to do ajax requests to the server and push them into the store. I’m afraid that I’ll run into the space limitations of LocalStorage with this approach but if that can be worked around then does this sound like a sensible approach? Are there any alternative approaches people can suggest?

Thanks!


#2

I’m exploring a very similar scenario. I’m probably going to use IndexedDB Shim or PouchDB.

I look forward to seeing what others will say on this topic.


#3

Thanks for the suggestions :smile:

I’ve had a look and can’t find any up to date adapters to use IndexedDB with ember-data. There is this one but it doesn’t work with the current versions of ember and ember-data. Another possibility would be via LawnChair and the ember-data-lawnchair adapter (sorry - I’m not allowed more than two links in a post!) but again I think it’s out of date…

Interested to hear if anyone else has any ideas, otherwise when the time comes I’ll probably look into building the relevant adapter myself. Would like to get a bit more familiar with ember first though!


#4

This is not the primary focus of the core team, so anyone interested in these solutions would be going off the beaten track. This doesn’t mean that there is anything wrong with using this technology, it just not the primary focus on the core team.

https://github.com/ebryn/EasyIndexedDB would be a good foundation for an Ember Data adapter.


#5

We are hoping to accomplish the very same scenario. From what I’ve read, localStorage doesn’t look promising. If you have any luck or make any decisions, please let me know. I’m currently toying around with the idea of writing my own adapter to use a HTML5 Web SQL database for local storage. I’m not quite sure how to handle syncing between the two data stores… but I imagine it’ll be pretty straight forward after I get both stores working properly.

If you want to bounce some ideas around you can email me at jbj at me dot com.


#6

I actually did some work on an ember-model-lawnchair adapter which allows you to sync to IndexedDB (or supposedly any other backend that Lawnchair supports). I ran into some issues with relationships in ember-model though and put it on the back burner.

I’m now using ember-data again in my app and running from the REST adapter. Once the app is working with that I am planning to implement either a Lawnchair or an IndexedDB adapter for ember-data (unless someone else has in the meantime). When I started this app ember was completely new to me and ember-data was pretty overwhelming but everything is becoming clearer over time…

In terms of syncing I’m planning to do something pretty naive. Basically I plan to switch over to the new adapter but create a sync method which makes an AJAX request to /sync?since=x method on my server. Any relevant data will be returned in one JSON document and then be side-loaded into my IndexedDB store. Like I said, it’s a fairly naive approach but will be “good enough” for my situation given how the data is generated and used.

Interested to know what people think of this approach and if anyone has tried anything similar…


#7

Hi @vitch any update on this? I’m thinking of doing something similar, except that I was considering to do it with the simple LocalStorateAdapter (at least for now), but my logic was slightly similar. The models (only some) will be stored locally and then I’m planning to sync over via an AJAX Request that will just get deltas.

I guess the difference in my use case is that the models I store locally are read-only, so the sync is easier because it goes in only one direction.

The other different with what I’m doing is that I was thinking of having other models w/o Ember-data or with a different Store (e.g. maybe one that uses the RESTAdapter). What do you think? And how’s your app doing?


#8

Hey everyone, I spent about 2 weeks trying to get EasyIndexedDB. It turned out to be more problematic than I thought because its designed to be framework agnostic, which introduced some complexities that I didn’t want to spend time resolving. The other issue is poor cross browser support, it would require IndexedDBShim to provide compatibility for Safari.

I ended up using PouchDB and writing a wrapper for Ember. So far, I’m very happy with it. Here are some of the things that PouchDB offers.

  • Cross Browser compatibility: Firefox 12+, Chrome 19+, Opera 12+, Safari 5+, Node.js 0.10+, Apache Cordova, Internet Explorer 10+
  • Automatic and continuous replication
  • Revision tracking
  • Change tracking
  • Seemless integration with PouchDB Server( which provides HTTP API ), Cloudant (hosted CouchDB server) and CouchDB

My library uses uses promises which allows you to use them directly in Route#model hook. I implemented all of the common operations GET, POST, PUT & DELETE. I’m working on creating a distributable package and going to be writing documentation this weekend.


Switch between online and offline data storage - local storage and ember-data
#9

Thanks for sharing. Please post the link to the package once you publish it. Sounds interesting even though we have a different backend, so the integration part with PouchDB compatible server’s might cross this out for us.


Switch between online and offline data storage - local storage and ember-data
#10

Hi @MiguelMadero! I can’t give you any update on this yet I’m afraid…

I’ve been working on other aspects of my app first and waiting to implement the sync and offline capabilities until the end. This way I’ll have a functioning app sooner and if there are further changes to ember-data in the meantime then I won’t have to update my adapter multiple times… I’ll let you know here once I’ve had time to tackle the problem though…


#11

Thanks for this @tarasm - it looks like it could be all types of awesome!

When I come to implement offline support I’ll definitely weigh up using this solution and writing an adapter at the backend level for my existing backend vs. writing an IndexedDB adapter and sync code on the frontend…


#12

All good, no pressure. I might work on something similar soon. I might be able to extract at least some reference code that might be helpful.


#13

Hey Guys,

I released PouchDB Wrapper for Ember.js https://github.com/taras/ember-pouchdb. Next, I’m going to incorporate it into CRUD without Ember Data.

Taras


#14

I’ve finally got around to tackling the offline portion of the project I’m working on and I’ve come up with a hacky/ingenious workaround (delete as appropriate) that simplifies things quite a bit. I’ll try to write the approach up in more detail and with examples when I have some time but here is a quick(ish!) description because I’m wondering if there is a better way to do some of it…

I’m using the appcache to make my files available offline. All my data is loaded from a /api/syncs URL - the syncs model has a bit of meta data in it but all of the other records that I need in my app are included in the payload and so get side-loaded into the store. And I’ve added a fallback for my sync URL which returns:

{
  "syncs": {
    "isUnavailable": true
  }
}

Then I made a custom app adapter (extending DS.RESTAdapter) and overrode the (private) ajax method:

ajax: function(url, type, hash) {
  var adapter = this;
  return new Ember.RSVP.Promise(function(resolve, reject) {
    adapter._super.apply(adapter, [url, type, hash]).then(function(data) {
      switch(url) {
        case '/api/syncs':
          if (data.syncs.isUnavailable) {
            resolve(JSON.parse(window.localStorage.getItem('syncCache')));
          } else {
            window.localStorage.setItem('syncCache', JSON.stringify(data));
            resolve(data);
          }
          break;
        default:
          resolve(data);
      }
    }, function(jqXHR) {
      // Handle server failures
    });
  });
}

As you can see, when the /api/syncs URL loads correctly we store the raw JSON returned in the browser’s localStorage. When we are offline and the appcache returns the fallback then we load the “last known good” data from localStorage and the rest of the app works fine and doesn’t need to be aware that the app was offline.

This works great so far. The problems start when my app needs to create or update models. Doing so causes POST or PUT queries to the server which don’t get caught by the appcache fallback. They do trigger the error handler of the ajax promise (where the “// Handle server failures” comment is in the above code) and I am catching the errors here and saving the URL, type and hash to localStorage (as a QueuedRequestModel).

I then need to resolve the outer promise. In doing so I need to make sure I create a representation of the object that matches what the server would have returned. This means converting dates to the expected format (they are sent to the server as a string e.g. “Fri, 17 Jan 2014 07:42:22 GMT” but are expected back as an integer e.g. 1389944542000). And making sure information about all relationships is present. And adding IDs to newly generated records.

This also works and when the user is back online they can trigger a sync where all of the pending requests from localStorage get sent to the server.

The problem that I still have is if a user reloads the page while still offline. Then the cached data for the /url/syncs endpoint doesn’t include the changes and additions that are recorded in the QueuedRequestModels. I need to find a way to replay these changes so that the app is in a consistent and up-to-date state.

So my questions are:

  • What do you think of this approach? Can you think of a simpler alternative or do you think it’s worth pursuing?
  • Is there any way around manually transforming the models in the POST / PUT callbacks to the structure that ember seems to expect? Is it correct behaviour of ember-data that the data sent to the server is formatted differently to what it expects to be returned? Is there a way to tell ember-data not to update the related model?
  • Can you think of a clean way to replay the queued changes when the app re-starts with QueuedRequestModels in the localStorage?

Thanks!


#15

Any updates to the subject of on and offline scenario???

VJ


#16

@tarasm Can ember-pouchdb be used with Ember-Data RESTAdapter ? If so I would love to use it for our current project. I have a php backend which sends json to ember-app-kit . Right now the database backend is MySQL. Can it be integrated ? I am fairly new to this offline caching and sync, apologies if it sounds dumb or clueless.


#17

@shadow_fox I didn’t write an ember-pouchdb adapter mostly because I haven’t worked with this library in a while. pouchdb in general requires a PouchDB or CouchDB backend that you can sync with.

Do you have an API server that provides data from MySQL?


#18

@tarasm , Yes , the backend is PHP ZF2 Framework which works as api layer sending json as per defined in ember-data.


#19

@shadow_fox how were you thinking about using in browser storage? what’s your use case?


#20

@tarasm I was thinking something like this:

RESTAdapter will work when the internet is present or else user can make transactions in offline but that will stored in local storage. And the user is back online , it will hard refresh the local storage to sync with the database and then one call to RESTAdapter that it will sync with the database.

Our product is somehow gonna be used in online mostly but some fraction of users will use it offline also. Data transaction has to flow even if there is no internet connection .

If this makes any sense or I am talking way out of my league.