Manual Pagination in Ember Data Relationships

Hey folks :wave:

I asked a question in Discord a few months back about this but it didn’t get any response :tired_face: To be honest it is completely fair because it is a bit of a sticky issue and this is a better medium to have a slightly longer conversation about it.

I’m trying to figure out a good way to properly paginate through a relationship in an ember-data model. I have a slight preference for doing this manually (because I’m actually working with a statically generated backend here), but at this stage I would just like to get anything to work :see_no_evil:

The first thing that I have tried is that I have made sure that I make use of the reference API and I can get the first 10 ids of the objects in my relationship and load them one by one. Here is the code that I’m using to load the data:

let ids = this.args.tag.hasMany('posts').ids().slice(0, 10);

let promise = Promise.all(ids.map(id => this.store.findRecord('content', id)));

this.topPosts = DS.PromiseArray.create({
  promise,
});

(yes it’s a blog system and I know it’s a cliché for examples like this but I’m actually trying to do this with a blog :joy:)

The problem is that over the course of the app lifecycle it seems like the tag.hasMany('posts') array is getting out of order compared to what is coming back from the server :thinking: I have mentioned this to a few people in the Ember Community and they have all said something along the lines of “you can’t rely on the order in the JSON:API response” :upside_down_face:

My question is this: is there any way that I can access this ordered list from the JSON:API response again or is it shredded somewhere in the ember-data internals?

Any help would be greatly appreciated :tada:

2 Likes

Some ideas

  • Would it be possible and easier to query the posts directly?

E.g. store.query('post', { filter: { tag: '123' }, page: { number: '1', size: '10' } });

  • Maybe adding json:api sorting helps as well?

E.g. store.query('post', { filter: { tag: '123' }, page: { number: '1', size: '10' }, sort: 'id' });

Generally I would also not expect that the relationship linkage in the json:api response has a specific order. But if that’s expected from the server/api, then you can probably track the order yourself in a service… maybe after serialization :man_shrugging:

1 Like

JSON:API specification leaves it up to the implementation of they give the order of resource linkage objects a meaning:

Resource Linkage

[…]

Note: The spec does not impart meaning to order of resource identifier objects in linkage arrays of to-many relationships, although implementations may do that. Arrays of resource identifier objects may represent ordered or unordered relationships, and both types can be mixed in one response object.

JSON:API — Latest Specification (v1.1)

Did you considered using related resource links instead?

1 Like

@real_ate From JSON:API’s perspective, this problem is solved by related relationship links, as @jelhan referenced above. You should be able to combine this URL with optional page and sort parameters, and then step through the next links returned in responses.

It’s been a while since I have looked at ember-data’s internals that store linkage ids for has-many relationships. I would be very surprised if you could count on the order of these ids, especially if they’ve been populated through multiple requests/responses. However, if you know that you have a complete set cached, then you could order this collection client-side.

I think a combination of related resource links and next links is a very good fit here - unless I was wrong with my assumption that this is about empress-blog.

The tag resource references the related posts, through a related resource link. This related resource link points to the first page of the resources for that tag. Something like this:

{
  "type": "tags",
  "id": "1",
  "relationships": {
    "articles": {
     "links": {
       "related": "/tags/1/posts-1.json"
     }
    }
  }
}

The related resource link returns the resource documents for the first 5 articles in default sorting order. Additionally it has a next link as described in pagination chapter of JSON:API spec. If there are more than 5 articles for that tag, the links.next points to the second page. If there aren’t more articles for that tag, it’s null. As the first page does not have any previous page links.prev is null. It would look like this:

{
  "data": [
    // resource objects of first five articles for that tag
  ],
  "links": {
    "next": "/tags/1/posts-2.json",
    "prev": null
  }
}

The second page would then contain the link to the third one or null if there aren’t more articles for that tag. Additionally it would have a links.prev pointing to the first page. You may additionally add links.first and links.last if you want to provide that functionality.

The same logic could be used to serve paginated articles by author.

There are a few drawbacks to note compared with the current architecture:

  • The logic to generate the static JSON files at build-time gets more complex.
  • The bundle, which needs to be hosted, gets bigger as the same article would be included in many JSON files (it’s main location + one time per tag used on it).
  • A user may need to download the same article multiple times depending on the usage. This gets as more likely as more different views on the content (e.g. main blog, different tags, different authors).

If you are worried about bundle size or expect a regular user to open multiple views, you could consider using relationship links. While a related resource link must return with a list of resource objects on a GET request, the relationship link returns a list of resource identifier objects. A resource identifier object only contains id and type but not the fields (attributes and relationships). So the payload of a relationship link is way smaller than the one of a related resource link.

It comes with the main drawback that you need multiple requests to fetch the related resources unless they are already in client-side cache: One to get a list of related resources and then one for each related resource to get the actual content. At the end of the day it’s a trade-off between initial render time and less overfetching on intensive usage.

For your specific use case cacheability might also be important. You could fingerprint the articles by using a hash of their content as ID. If serving them as individual files and using their ID as a file name, they could be cached forever thanks to fingerprinting. On deployments invalidating the lightweight relationship links would be enough. To be honest I’m not sure how that would fit with the main page without adding a needless resource just to serve the list of articles used on it through a relationship link.

Disclaimer: Haven’t used relationship links and pagination links in Ember Data for a while. Not sure how well they are supported. But maybe Ember Data would not be needed at all for this simple use case? A service, which fetches the articles and implements a client-side cache might be enough. The rest seems to be lightweight enough that overfetching should not matter that much.