Ember-Graph - An Ember-Data alternative


#1

A few months ago, I wrote an Ember persistence library to act as an alternative to Ember-Data. It’s been open source for a while, but I haven’t really ‘released’ it yet because I’ve been trying to polish some things up. But as the old saying goes, “if you’re not embarrassed by the first release, you’ve released too late”. So with that in mind, hell, let’s show it off.

I’d like to introduce you to Ember-Graph. Ember-Graph is an Ember-Persistence library that is built with complex object graphs in mind. I wanted something that could intelligently handle all of my relationships, and Ember-Data just wasn’t cutting it. So I set out to design a library that would not only work for our application, but for the community as well. And so far I’d like to think I’ve done a pretty good job (we’ve been using in our app for 4 months).

Some of the features aren’t finished, the documentation is poor and there are no usage guides. What more could you want, amirite?! But seriously, it’s in the early stages, so go easy on me. But there is a lot of functionality there. And, because it’s early in its life, now would be the perfect time to suggest features. :smile:


So what’s the big deal? Well, I want to show you just a few features that Ember-Data doesn’t (yet) provide.

First, let’s define a couple of models to work with:

App.Post = EG.Model.extend({
    title: EG.attr({ type: 'string' }),
    body: EG.attr({ type: 'string', defaultValue: '' }),
    posted_on: EG.attr({ type: 'date' }),

    comments: EG.hasMany({
        relatedType: 'comment',
        inverse: 'post',
        defaultValue: []
    })
});

App.Comment = EG.Model.extend({
    body: EG.attr({ type: 'string' }),

    post: EG.hasOne({
        relatedType: 'post',
        inverse: 'comments',
        defaultValue: null
    })
});

Looks pretty familiar, huh? So let’s say you’ve got some data that you want to load into the store. Screw adapters, screw serializers and screw anything that isn’t your own code.

App.get('store').extractPayload({
    post: [
        { id: '1', title: 'First Post!!1', body: 'Heh?', posted_on: new Date(), comments: ['1', '2', '3'] }
    ],
    comment: [
        { id: '1', body: 'First comment!', post: '1' },
        { id: '2', body: 'Sigh. Why do I even come here any more?', post: '1' },
        { id: '3', body: 'guyz, first post died in like 2006 get over it', post: '1' }
    ]
});

Now let’s do some manipulation:

var store = App.get('store');
// getRecord skips the promise if the data is loaded already
var post = store.getRecord('post', '1');
// in both Ember-Graph and Ember-Data, this returns a promise
post.get('comments');
// but what if we don't want a promise, just the IDs? EG has got you covered
post.get('_comments'); // ['1', '2', '3']

Now for the real magic:

// let's add another comment to our post
post.addToRelationship('comments', '4');
// let's double check
post.get('_comments'); // ['1', '2', '3', '4']
// wait, comment 4 doesn't exist! let's fix that
store.extractPayload({
	comment: [
		{ id: '4', body: 'Fourth post!!!!11one LOL I\'m so funny. NOT.' }
	]
});
// Notice how we didn't associate the comment with a post
var comment = store.getRecord('comment', '4');
comment.get('_post'); // '1'
// EG knew enough to connect that relationship for you

What about deleting or changing relationships? We do that too:

store.extractPayload({
	post: [
		{ id: '2', title: 'First Post: Part 2', comments: ['5'] }
	]
});
// We hate comment 5, let's replace it with comment 6
var post2 = store.getRecord('post', '2');
post2.removeFromRelationship('comments', '5');
post2.addToRelationship('comments', '6');
// But what about the posts you say?
store.extractPayload({
	comment: [
		{ id: '5', body: 'GTFO.', post: '2' },
		{ id: '6', body: 'That\'s what she said!', post: null }
	]
});
var comment5 = store.getRecord('comment', '5');
var comment6 = store.getRecord('comment', '6');
comment5.get('_post'); // null
comment6.get('_post'); // '2'
// Again, Ember-Graph knew to connect the relationships

But what if we make a mistake?

comment5.rollbackRelationships();
comment6.rollbackRelationships();
comment5.get('_post'); // '2'
comment6.get('_post'); // null

What about attributes? We handle those too.

// Let's change our post title
post.set('title', 'Better Title');
// Then, let's accept a Socket.io update from the server
store.extractPayload({
	post: [
        { id: '1', title: '[Closed]First Post!!1', body: 'Heh?', posted_on: new Date(), comments: ['1', '2', '3'] }
    ]
})
// it didn't overwrite the client-side change
post.get('title'); // 'Better Title'
// so let's discard the change
post.rollbackAttributes();
// Now it takes the last know server value
post.get('title'); // '[Closed]First Post!!1'

That’s enough examples, you get the picture. The idea is that Ember-Graph wants to handle relationships as intelligently as possible, and that includes accepting push updates from the server at any time. After that merging the data is configured with just a few options. Ember-Graph knows the importance of real-time updates and concurrent viewing/editing, so it’ll always be ready for your data. :slight_smile:

If you have any questions, just ask. If you have bugs or feature requests, files them on Github. And if you have complaints, PM me. :slight_smile:

Despite the fact that Ember-Data has announced their single source of truth (only a month after I created this, d’oh!), I still believe Ember-Graph serves a purpose. Hopefully you think so too.

EDIT: Sorry, but the markdown parser went haywire. The examples are a bit hard to read.


#2

This looks like it could be very useful for people (like me) that are working on projects with large datasets and complex relationships. How well does EG play with APIs with varying conventions? Does it follow Ember-Data’s philosophy? or is it relatively easy to modify adapters and serializers?


#3

Like Ember-Data, Ember-Graph has the concept of an Adapter. It comes with a few adapters, including a REST adapter based on the JSON API, but you can create an adapter for any API. I tried to make it as easy as possible to push data into the store, so I created a normalized format the the store accepts (described here). If you can convert your data to that JSON format, it can come from anywhere.

My company’s application at work depends on Ember-Graph, so a lot has been done to it since this thread was created. Over the next few days, I plan on creating a getting started guide that will walk through making an application. I’ll also cover topics such as writing a custom adapter or serializer, and modifying the built-in ones. I’ll update this thread when those are up.


#4

looking forward to it for sure


#5

This looks very interesting. I am actually kind of not very happy with ember data right now. How hard would it be to do these two things with ember-graph:

The client receives a group object like:

{
  'name': 'my-group',
  'items': ['item-id-1', 'item-id-2']
}

When I render the associated group template, I want to display that there are 2 items in the group, without actually loading child the items (couldn’t yet figure out how to prevent ember data from loading everything whenever it thinks it might need it somewhen). I just want to load the items whenever that group is selected and I need to actually render the items (unless they are actually present in the store).

In addition to communicating with the server, I would like to store everything in local storage. So if I come back to my site later, I can pick everything up where I left without necessarily making a request to the server / or just continue offline. I haven’t yet had the chance to look at that aspect when using ember-data. But that’s definitely something I want for my app.

Ultimately what I am going for is: When the user logs in the server starts to push out (over websockets) all the data the client needs to know about. This is mostly just metadata, so that should never grow too big. From that point on the client is in sync with the server. Now there are just two things that might happen: Either the client changes something and needs to submit the changes to the server, or someone else changes something and the client receives the updates from the server. Would ember-graph be a good match to do something like that?

And just one more last thing: Is it possible to store embedded object within ember-graph? Something like a user object that looks like:

{
  'username': 'joe',
  'quota': {
    'total': 100,
    'used': 20
  }
}

In case some of these things currently are not supported, do you think they would make a good addition to ember-graph in the future?


#6

ooof, your example makes me want to dump ED so hard. There are numerous hacks I’ve used to get things like IDs or hasMany.length, but the biggest problem has always been sideloading records when the server’s response is non-standard.

For instance, consider the relationship between a conversation and messages. In our setup, we don’t return the IDs as an array on the property conversation.messages because that would often require an absurd number of IDs to be loaded. We do, however, sideload the first 50 when we initially load the conversation. This quickly turns into a mess.

Whenever a new message comes into the conversation, we’ve got to remember to call conversation.messages.pushObject(message).


#7

I’m going to try to answer all of your questions. If I miss one, let me know. :smile:

  1. To count items without loading them, use the relationship name prefixed with an underscore. So for you, object.get('_items') will return something like this:

    [
        { type: 'itemType', id: 'item-id-1' },
        { type: 'itemType', id: 'item-id-2' }
    }
    

    This won’t load any data from the server, so it is a synchronous call. As you’ll see from the example that I’ll post, this is handy when you need to do things like count the number of comments for a thread, without loading all of the comments.

  2. I just finished the first draft of a localStorage adapter on Thursday. It’s not perfect yet, but I plan on using it for integration testing for our Ember app, so it should progress fairly quickly. The goal is for the adapter to handle all relationship data for you, while at the same time, being flexible enough to implement custom ‘server’ logic. And because Ember-Graph doesn’t care where data comes from, it should be fairly easy to use both the localStorage adapter AND a REST adapter.

  3. For live server updates, I believe that Ember-Graph will be perfect. I’m sure it has flaws now, but this is exactly the thing Ember-Graph was designed to do. Ember-Graph is setup to receive updates from the server, even if your records are currently dirty. If you look at this (extremely long) file, you will see how much effort has gone into merging server and client side changes without creating conflicts.

    Also, I hope to have built-in Socket.IO support in the adapters soon.

  4. There is no embedded record support yet. There are a few reasons for that, but the biggest one is that I don’t know enough about embedded records to really implement the feature nicely. For instance, in your example, I wouldn’t call that an embedded record (since it doesn’t have an ID), I would call it an attribute. By extending the AttributeType class, you can create a QuotaType fairly easily. I do something like this for one of my models.

        App.QuotaType = EG.AttributeType.extend({
            serialize: function(quota) {
                return quota;
            },
            deserialize: function(json) {
                if (json && Em.typeOf(json.total) === 'number' && Em.typeOf(json.used) === 'number') {
                    return json;
                } else {
                    return { total: 100, used: 0 };
                }
            },
            // Ember-Graph uses this to only save changed attributes
            isEqual: function(a, b) {
                return (a && b && a.total === b.total && a.used === b.used);
            }
        });
    

    Then, you can change the values just like any other attribute:

    user.set('quota.used', user.get('quota.used') + 10);
    

Sorry, a bit of a long post, but hopefully I got your questions. If you have any feature requests, feel free to post them here or in Github Issues.


#8

Sideloading is something that I focused on in Ember-Graph, since I always had trouble with it in Ember-Data. As of right now, the store expects complete record representations when loading data, but that is going to change soon. I want the store to be able to support partial-loading of relationships so pagination can be done easily. This is a definite use case for us. When one relationship has 150 IDs, and your IDs are all UUIDs, the JSON size grows very quickly.

In fact, I’ve just put an issue in Github for this. I expect it to be fairly easy to accomplish.

Also, something that Ember-Graph does, and does well, is maintain relationships. So I’ve made every effort to ensure that you never have to do something like conversation.messages.pushObject(message). In fact, it’s not even possible in Ember-Graph. It handles all of that for you. :smile:


#9

Thanks a lot. That all sounds great. I will see if I can find time to migrate my project to ember graph within the next days and see how that goes.

I am using SockJS for all my communications exclusively. (Backend runs on vert.x which comes with a full implementation for the SockJS server side). I had a quick look the Socket.IO documentation, but they don’t seem to be that different from each other API wise.


#10

I actually started to try ember-graph out and I have come across one thing I am not really if sure if I am doing something wrong or if this actually might be a bug:

_findQuery: function(typeKey, options) {
  var promise = this.adapterFor(typeKey).findQuery(typeKey, options).then(function(payload) {
    var ids = payload.meta.ids;
    this.extractPayload(payload);

    return ids.map(function(id) {
      return this.getRecord(typeKey, id);
    }, this);
  }.bind(this));

  return EG.PromiseArray.create({ promise: promise });
}

It tries to look for the ids using the meta field of the payload:

 var ids = payload.meta.ids;

However, the serializer transforms the ids into a field called payload.meta.queryIds. If I change that, it seems to work, but probably I am just making a wrong assumption at some other point and this just shows up because of another problem.

But I need to say, it feels kinda good for now. A lot of things seem to make sense.


#11

That is indeed a bug. I meant to refactor that last week, but I was a bit crunched for time. I went ahead and fixed that particular bug.

However, I’m still a bit weary about the querying. Right now, you can only query over one type of object, which I don’t think works very well if you have polymorphic models. In the next few days/weeks, I hope to come up with a decent way to query over many types of objects (or a single super-type). We don’t use querying in our application (yet), so it’s not the strongest part of the library (yet).


#12

Great. Thanks.

I just noticed that createRecords already expects all the required fields of a record to be there. What I was doing with ember data was: In case the user creates a new record that does not have an id assigned, it maps to the url slug ‘new’. Then the user fills out all the needed information and after the user clicks the ‘create’ button, I persist the record. Any chances to change createRecords to allow the creation of records that currently don’t fulfil all requirements of a persisted record and only validate on saveRecord?


#13

I’ve had a similar use case to that before, but I wasn’t sure exactly how to solve it. For now, EG expects all required attributes to be present at the time of creation. The reason being is that required should mean required. If you have a required attribute that can be omitted, it’s not required, is it? :wink:

I think at some point in the future, I want to allow an attribute to be required by the server, but not by the client. But for now, my recommendation would be to use a reasonable default for that value. For instance, I have a list of items that all require positive, unique, sequential integers. But as a default, I use 0 to signify that the object hasn’t been persisted yet. I think this is better than leaving it as undefined, since a rogue undefined value can wreak havoc on your code.

EDIT: Also, I’m not sure it’s obvious, but EG uses temporary IDs for records that aren’t persisted yet. To know if an ID is temporary, you can use the isTemporaryId method. Or you can just get the isNew property of any model.


#14

Thanks for the reply. But, it’s not that always that easy… UI wise I currently treat a new record which still has undefined properties and has not been persisted the same way as every other record. So, the user sees it in the place it ends up when it is persisted, but its actually in a dirty state unless the user decides to actually save it or discards the creation and the record is thrown away. So, technically you can not omit required attributes when persisting them. But you can create a record which does not have all required fields set. You just fulfil the requirements later.

Probably the thing why I am struggling with this is that it is not obvious to me why there is a createRecord and a saveRecord if they all must meet the same requirements. In the end I think it is way more complicated to deal with maintaining requirements on created but not persisted records. For instance, I create a record which fulfils all requirements. This would be fine to persist. Now I change a required attribute to undefined and you enter a state where you created a valid record, which still should be valid to persist but it is not. In the end the only good solution would be to track every created record for its validity until it is persisted. But that sounds really painful to me. So why not just validate the records when they are actually persisted? I don’t think that would hurt and it would be way more flexible.

Or to view it from another standpoint: What if I load a record, which has a requirement for a property having a certain length. For instance the username must be at least 6 chars. Now the user edits that property and while doing so, he erases all the previously entered chars. If I should not be able to create an invalid record at all, than I also should not be able to put it into such an invalid state. Everything else just breaks consistency of an api in my opinion.


#15

But you can create a record which does not have all required fields set.

EG doesn’t force you to give the fields final values, it just forces you to initialize them to something other than undefined. EG properties aren’t allowed to be undefined. There are multiple reasons, but the main reason being that undefined serves a special purpose in the Ember object model. In order to make things easier to work with, and to avoid a whole class of errors, I required all attributes to be defined. If you don’t have a value for them yet, use null or an empty string.

Also, complete validity checking in on the roadmap. (It actually used to be in the code, but I removed it because it wasn’t quite ready yet.) So what you describe will be implemented soon: if a username must be at least 6 characters long, EG won’t let you edit the property to a string less than 6 characters.

So the features to get what you want are coming. :smile: Unfortunately, I don’t think EG will ever support undefined as an attribute value. (I suggest null instead).


#16

thanks, that sounds like a reasonable workaroundt. I am not entirely sure if that is the best way though, but probably I am just too new to ember anyway. On another note, I am struggling with my relationships a little. Right now it seems the serializer expects something like this from the adapter for a group with related items:

{
  'groups': [{
    id: 'group1'
    items: ['item1', 'item2']
  }],
  'items': [
    {
      'id': 'item1'
    }, {
      'id': 'item2'
    }
  ]
}

If the items array is missing, the serializer complains. However, that is actually what I want to do, as I have quite some nested relationships and I don’t want to load the whole tree at once. I can see two ways of solving this:

  • write my own serializer, which just adds objects like `items: [{‘id’: ‘item1’, ‘type’, ‘item’] to the group record
  • add a lazy loading option to the model, which allows the adapter/serializer to load the records later when they are needed.

In favour of reusability I would vote for option two, but it’s up to you to decide if thats the right way for ember graph to go…


#17

I would suggest making the items relationship optional. Required relationships are usually for relationships that must exist, like the author of a thread. But if you don’t absolutely need that relationship, just make it optional:

items: EG.hasMany({
    isRequired: false
})

You can read about the other options for a hasMany relationship here.

When it comes to relationships being required or not, I usually have to ask myself: is this relationship required for my application to function? If the answer is no, it should probably be an optional relationship. Most of mine are optional. The reason EG defaults to required is because that’s what Ember-Data does.


#18

Maybe that was a little unclear, but that relationship actually must exist. I just don’t want to load it unless I really have to. If I set isRequired to false, then the attribute gets omitted entirely (at least unless I am doing something wrong)

Basically I have a nested structure like:

cities -> libraries -> books -> pages -> comments

This structure is reflected by a nested navigation. So, if I load the cities, I also retrieve which librarie-ids are in that city. But I don’t retrieve any specific information about the libraries:

{
  'cities': [{
    id: 'NY',
    libraries: ['lib1', 'lib2']
  }],
}

So now I can display a list of cities and a count of libraries in that city. If the city is selected I load all the libraries in that city. This might seem a little complicated but this is based on the assumption that the client already has almost all the data it needs, or will receive it shortly anyway, so the client only requests what it really needs.

Now if I make the libraries relationship in the city isRequired: false I now longer get the count and which library ids are in that city. Therefore the suggestion of an lazy loading attribute. But that should be solvable by a custom serializer if that doesn’t fit ember graph well.


#19

I’m not sure I completely understand you. Would you mind posting your model and an example server payload?


#20

That actually has been a server payload as I expect it. Quite some RESTful APIs are actually build similarly.

{
  'cities': [{
    id: 'NY',
    name: 'new york',
    //....more data about the city
    libraries: ['lib1', 'lib2']
  }],
}

So, the city has many libraries. However when the cities are queried, the response only contains the details about a city and a list of library ids. If the client wants to know the details about the libraries it has to query for the libraries. The response for a library in turn would contain the details about the library and a list of book ids.

The problem I am having lies here:

deserializeRelationship: function(model, json, name) {
    var meta = model.metaForRelationship(name);
    var value = json.links[name];

    if (value === undefined) {
        if (meta.isRequired) {
    	throw new Em.Error('Missing `' + name + '` relationship: ' + JSON.stringify(json));
        }

        return { name: name, value: meta.defautlValue };
    } 

It expects the related entities under the links attribute of the json response. If the links attribute does not exist and it is not required it uses the default value, which basically overrides my ids. But when thinking about it probably a new serializer would probably be the best option as the current one seems to be very specific to json api. Probably that would not be very smart to mix these changes in.

Well, and I hope I am not bothering you too much with this…