Dirt tracking with Ember-Data - only save dirty properties

[gerrit]

Hi,

First things first. I am no native english speaker, so if I have some strange saying, I excuse myself in advance.

To the point:)

I am making the transition from Ember Data 0.13 to Ember Data 1.0.beta.6. At some points its quite easy at some, but at other points its hard to find the right practice.

I am using the ActiveModelSerializer and -adapter with a rails backend and all is quite nice yet. The Problem I get is when saving records. I need to only save dirty attributes. Excample:

user = @store.find('user', 1) 
user.get('name')                        # prints 'sergio leone'
user.get('role')                        # prints 'cowboy'
user.set('role',  'chicken').save()

this should make a post request with only the changed properties:

{ user: { id: 1, role: 'chicken' }

also I dont wanna send attributes with no value. When I create a new record I send something like this, when no role is set:

{ user: { name: 'Pling', role: null }

Am I seeing it right that there is no dirt tracking included in Ember Data? I searched the source code abit and I think that the right point to handle this would be the serialize method of the ActiveModelSerializer like this:

App.ApplicationSerializer = ActiveModelSerializer.extend
serialize: (record, options) ->
  json =  this._super(record, options)
  //check if some value is null and what attributes are dirty
  return json

Am I on the right way? I am really getting headaches by searching the right point to interfere. And how can I check which attributes changed. Aint the adapter managing it?

I hope you can bring some light into my damaged head.

cheers klopfer

[radq]

You are right that the correct place for this is the application serializer. I wrote one a while ago but I haven’t tested it much, perhaps it would be helpful as a starting point at least?

Saving models with only changed attributes/relationships

You can do this by overriding the serializer. Hummingbird.ApplicationSerializer = DS.ActiveModelSerializer.extend serialize: (record, options) -> json = {} if options and options.includeId if record.get('id') json[this.get('primaryKey')] = record.get('id') changedAttributes = Object.keys record.get('_inFlightAttributes') record.eachAttribute (key, attribute) -> if changedAttributes.indexOf(key) != -1 @serializeAttribute(record, json, key, attribu…

[gerrit]

Ok, I will try my best and report back.

Thank

[gerrit]

ok, it worked out. Thanks mate

I also added an exception for embedded objects and a readOnly flag for attributes not meant to be saved.

App.ApplicationSerializer.reopen

  serialize: (record, options) ->
    json = {}

    if options and options.includeId
      if record.get('id')
        json[this.get('primaryKey')] = record.get('id')

    changedAttributes = Object.keys record.get('_inFlightAttributes')
    record.eachAttribute (key, attribute) ->
      # only serialize attributes which have been changed and
      # arent readonly
       if changedAttributes.indexOf(key) != -1 
          if !attribute?.options?.readOnly?
            @serializeAttribute(record, json, key, attribute)
    , this

    record.eachRelationship (key, relationship) ->
      # if a relationship is embedded, remove that option,
      # as we only wanna deserialize embedded objects
      isEmbedded = @get("attrs.#{key}.embedded")
      @set("attrs.#{key}.embedded", null) if isEmbedded?

      # only serialize relationships that arent read only
      unless relationship?.options?.readOnly?
        if relationship.kind == 'belongsTo'
          @serializeBelongsTo(record, json, relationship)
        else if relationship.kind == 'hasMany'
          @serializeHasMany(record, json, relationship)  
    , this
    # set back the embedded flag for deserializing
    @set("attrs.#{key}.embedded", isEmbedded) if isEmbedded?
    json

It was a beast. Im just getting into this stuff, but this helped me alot

[cgonzalezvazque]

This is exactly what I was looking for. I’ve used exactly this serializer and it’s working fine for me. Thanks

[adamsrog]

Working perfectly on ember-data 1.0.0-beta.8, thanks @gerrit!

[gerrit]

Ok, I found a problem. Once you save your model and the server returns its data, you cant save it again. The inflightAttributes are empty and only the relations are getting saved.

I will tinker about it next week.

[adamsrog]

I’ve ran into an issue as well.

I’ll try and explain how I found this: I’ve got a drop-down box that allows the user to specify a ‘category’ attribute. Assume there are 4 categories in the drop-down and it starts as category 1. I can change to categories 2, 3 and 4, but if I try changing back to category 1, the dirty tracking doesn’t recognize it as being a change.

In other words, I can’t change back to the original value after changing it (without a refresh). Will post when I figure out the fix.

[gerrit]

exactlcy that. Afaik Ember data stores all the properties and values in a property called “_data”, whenever you change an attribute ember adds it to the “_data” property of the model. The first time it works, but once you saved it, _data never gets refilled again.

[gerrit]

Ok, Im getting closer… Em.Data holds the Models properties in “model._data”. if you change an attribute it gets copied to “model._attribute”. If you run the ember save function, “_attributes” will be moved to “_inFlightAttributes” and those are the attributes we use for the dirt tracking. Its like this:

user = User.find(1)  

user.get('name')                   // 'sir lancelot'
user._data                         // Object { id: 1, name: 'sir lancelot' }
user._attributes                   // Object {}
user.set('name', 'rocky')
user._data                         // Object { id: 1, name: 'rock' }
user._attributes                   // Object { name: 'rocky' }

It looks like the first time you save, everything is dont properly. If you edit the record and save it again, the save() function gets called twice. So the first time the function is called, this happens:

this._inFlightAttributes  // Object {}
this._attributes          // Object { name: rocky }

this._inFlightAttributes = this._attributes
this._attributes = {}

this._inFlightAttributes  // Object { name: rocky }
this._attributes          // Object {}

When save gets called the 2nd time _inFlightAttributes gets overwritten with the by now empty “_attributes”.

Now its time to find out why it gets called twice.

[christopher]

I have run into similar situations with Ember Data, but for me the need has varied. It is not simply “always save all attributes” vs. “only save modified attributes”. I have noticed three distinct use cases:

• Saving all attributes, which is what Ember Data currently does by default.
• Saving only the modified attributes.
• Saving specific attributes. For example, if the user can adjust a start and end date, both should be saved even if the user only changes one.

To accomplish this, I added two additional methods to DS.Model: saveChanges() and saveAttributes(). My implementation is sub-optimal (it requires two server requests) and not atomic, but it has served its purpose quite well for me. If you are content with the default “save all” and just need some added flexibility when the need arises, this gets the job done.

DS.Model.reopen

  # only save the specified attributes to the server
  saveAttributes: ->
    return Ember.RSVP.resolve() if arguments.length == 0
    attributes = @getProperties(arguments...)
    @rollback()
    @reload().then =>
      @setProperties(attributes)
      @save()

  # only save the modified attributes to the server
  saveChanges: ->
    attributes = Ember.keys(@changedAttributes())
    @saveAttributes(attributes)

Love to see an interface like this make its way into Ember Data, obviously with an implementation that only saves the specified attributes. If the core team is open to it, I’d be happy to give it a go.

[pixelhandler]

@christopher Curious have you looked at JSON Patch? I think that it would be great to have an option to persist only small changes. Here is an idea… JSON Patch support for Ember Data - #2 by machty

[christopher]

@pixelhandler Thanks for the response, and thanks for your blog post on the topic! I read it a few days ago and am still absorbing it. I think JSON Patch is ultimately a great approach and certainly better than what I am doing today, but it seems like a lot of effort to attempt to support it at this point, both on the client and server. My APIs (in Rails) can take a subset of attributes out of the box, so my implementation just works without the need for separate PATCH requests. I will definitely take a closer look at Ember Orbit when starting my next project and will be keeping an eye on your RFC though. For me for now, staying close to the current happy path with Rails API and Ember Data is ideal.

[coladarci]

Hey @gerrit, I’m curious, did you ever make it past this problem? Saving only dirty fields is pretty important for our use-case and are hitting the same wall as you.

[gerrit]

Hi @coladarci , generally yes, but there is a main problem. You have to mimic everything that Ember does. Especially Embedding for belongsTo and hasMany. I can show it up, but whenever there is a change in Ember Data you have to adapt your serializer. At the moment Im in going for final examens, but I can show up my solution and the one we use at the company. I think the goal should be to go in depth and make a PR, but for me personally thats not possible before jan/feb. I will clean up my solution and post in the next days and maybe it will help you out. Might be good to know what Version of ED you are using.

[coladarci]

Hey @gerrit, that would be amazing. Currently, we are on Ember 1.8.0-beta.1 and Ember Data 1.0.0-beta.9.

As an FYI, the use-case that is failing for me is when you get a validation error back from the server; ember doesn’t treat the un-saved properties as dirty so they never get sent to the server.

We’d love to help with the PR, part of me thinks this is flat-out-broken in that if I have a model, I make a change to an attribute, save but don’t get a 200 back, dirty attributes is empty.

[forkbomb]

Hi everyone. Here is my solution for the problem for ActiveModelSerializer. It was not tested much yet, but i hope it helps someone. Ths solution tracks relationships as well as regular attributes, and it does not uses much internal api nor re-implement anything. But the downside is that it uses one observer per relationship on every model instance.

DS.Model.reopen
  setupChangeObservers: Em.on 'init', ->
    @_changedRelations = []

    @eachRelationship (name, relationship) =>
      observer = change: => @_changedRelations.push(name)
      @addObserver name, observer, 'change'

  resetChangedRelations: Em.on 'didLoad', 'didUpdate', 'rollback', ->
    @_changedRelations = []

  allChangedKeys: ->
    relations  = @_changedRelations
    attributes = Em.keys @changedAttributes()
    inFlight   = Em.keys @_inFlightAttributes

    [].pushObjects(relations)
      .pushObjects(inFlight)
      .pushObjects(attributes)
      .uniq()

DS.ActiveModelSerializer.reopen
  serializeAttribute: (snapshot, json, key, attribute) ->
    if snapshot.allChangedKeys().contains(key)
      @_super snapshot, json, key, attribute

  serializeHasMany: (snapshot, json, relationship) ->
    key = relationship.key
    if snapshot.allChangedKeys().contains(key)
      @_super snapshot, json, relationship

  serializeBelongsTo: (snapshot, json, relationship) ->
    key = relationship.key
    if snapshot.allChangedKeys().contains(key)
      @_super snapshot, json, relationship

[gerrit]

@forkbomb

i everyone. Here is my solution for the problem for ActiveModelSerializer. It was not tested much yet, but i hope it helps someone. Ths solution tracks relationships as well as regular attributes, and it does not uses much internal api nor re-implement anything. But the downside is that it uses one observer per relationship on every model instance.

I was thinking, since there is now the new snapshot api, it could be done with a one-liner. Think of it. A snapshot gets newly generated each time you serialize a record. So the simplest thing would be.

serialize: (snapshot, options) ->
  snapshot._attributes = snapshot.record._inFlightAttributes
  @_super(snapshot, options)

And the next the the model gets serialized, theres a new snapshot and the original record isnt touched.

[forkbomb]

@gerrit, this approach was the first thing i tried, but in my case snapshot.record was null, and relationships were not appearing in _inFlightAttributes

I am using Ember 1.9.1; Ember Data 1.0.0-beta.14.1

UPD: Ok, Ember-data beta 14 is passing records, not snapshots. I have updated to beta 15, but i still need realtionships tracking. Here is my code updated to work with beta 15, plus i’ve made it a bit more compact:

DS.Model.reopen
  _setupChangeObservers: Em.on 'init', ->
    @eachRelationship (name, relationship) =>
      observer = change: => @_changedRelations.push(name)
      @addObserver name, observer, 'change'

  _resetChangedRelations: Em.on 'init', 'didLoad', 'didUpdate', 'rollback', ->
    @_changedRelations = []

  hasKeyChanged: (key)->
    [].pushObjects(@_changedRelations)
      .pushObjects(Em.keys @changedAttributes())
      .pushObjects(Em.keys @_inFlightAttributes)
      .uniq()
      .contains(key)

DS.ActiveModelSerializer.reopen
  serializeAttribute: (snapshot, json, key) ->
    @_super.apply this, arguments if snapshot.record.hasKeyChanged(key)

  serializeHasMany: (snapshot, json, relationship) ->
    key = relationship.key
    @_super.apply this, arguments if snapshot.record.hasKeyChanged(key)

  serializeBelongsTo: (snapshot, json, relationship) ->
    key = relationship.key
    @_super.apply this, arguments if snapshot.record.hasKeyChanged(key)

[gerrit]

that really looks nice… I remember something that Em.Data Beta15 could track changes in relationships, but I didnt find it yet.

I am not to sure if observer are the best way, still it works and is better than none.