Pagination with Ember Data - can we agree on a single solution?

Ember Data
1

In the spirit of the Modal View topic let’s continue with pagination. It’s been discussed over and over that we need to somehow standardize this, but I think it would be a good idea to make this discussion public, and not just something on gtalk/campfire :slight_smile:

I don’t think this is a question that belongs to SO, because what I’m looking for is a discussion of how different approaches to the problem and which one should be made a standardized solution.

The option people seem to be taking these days is to return metadata about the collection with the data itself, which may look something like this

{
  "meta": { "total": 10, "page": 2 },
  "posts": [
    { "title": "Post 1", "body": "Hello!" },
    { "title": "Post 2", "body": "Goodbye!" }
  ]
}

The result of doing let’s say App.Post.query({ page: 2 }) should be a RecordArray which already knows about the current page and total number of records.

From what I’ve heard the correct approach here would be to use the undocumented extractMeta on the JSON Serializer and do something with it. For example this SO answer sets a global metaDataForLastRequest.

Looking at the implementation for extractMeta, it calls onto the sinceForType on store, which is yet another undocumented method :frowning:

extractMeta: function(loader, type, json) {
  var meta = json[this.configOption(type, 'meta')], since;
  if (!meta) { return; }

  if (since = meta[this.configOption(type, 'since')]) {
    loader.sinceForType(type, since);
  }
}

this makes me wonder if it’s even a good idea to override this method in a custom serializer? Are there any implications to this?

This also seems to be directly related to this PR which makes sideloading ignore anything that isn’t an array, which we probably want for pagination metadata. Should the alternative solution here be to just override the sideload method and ignore the metadata key?

7 Likes
2

We stumbled over this a bit too, and attempted to override the adapter/serializer. The issue we had was where to store the meta once extracted. Initially, we thought that the recordArray was the go, but this has a number of issues:

  1. The recordArray is only exposed in the serializer for findQuery calls
  2. If you call the stores’ filter function, a filteredRecordArray is returned and not the AdapterCreatedRecordArray from the findQuery function (where the meta is set)

We also tried to follow the sinceForType approach, there is no access to the DS.loaderFor after ember data has been initialised, due to an object being returned. You could of course override DS.loaderFor in it’s entirety, but that would be a pain should it change in the core.

In the end, we refactored the sinceForType functionality to allow custom meta parameters to be set on the stores’ typeMap and created a PR.

Would love feedback on this approach and similarly we are interested in how others are solving this problem.

1 Like
3

I tried using extractMeta and failed… ended up using didFindQuery to attach the meta to the record array. It ended up looking something like this:

didFindQuery: function(store, type, payload, recordArray) { var loader = DS.loaderFor(store);

	loader.populateArray = function(data) {
		recordArray.load(data);

		// This adds the meta property returned from the server
		// onto the recordArray sent back
		recordArray.set('meta', payload.meta);
	};

	this.get('serializer').extractMany(loader, payload, type);
},

Pagination is fairly simple once the meta object is attached to those record arrays.

2 Likes
4

Even though the didFindQuery override works I would also like have a standard solution for this. I don’t think the current implementation is coherent. I will expose my use case.

I had my application working with find (findAll) using an infinite scroll interface. With almost no code I was loading a long list of items from my django server using the sinceForType functionality. In a small redesign of the app, now I will be filtering that list. This means that I will be using findQuery instead of findAll. What it should be just adding a parameter to my find call will finally force an implementation of a customized didFindQuery in my adapter.

I guess there must be reasons for the inconsistency of findAll and findQuery working in a different manner. Is there any document that explains them so I can understand this issue better?

Besides using a custom DjangoAdapter, I always find myself adding an additional hack related with findQuery in my apps.

5

I remember this coming up during EmberCamp as slide 44 of @dgeb’s presentation and made a mental note to see where the community was heading on it.

I know I’ll need some form of pagination for a mini-app I’ll be working on soon.

Have any lessons have been learned from Ember List View?

6

shameless bump to get this going again

7

I don’t know if this has any affect on the issue but I’m concerned that the prior examples and discussion in this thread seem to be assuming an outdated way of doing pagination.

It’s common practice to use range based pagination these days for performance with big datasets and in the rest api return the next and previous pages (and possible first and last) as urls in a ‘Link’ header.

For an example see github pagination here: GitHub REST API - GitHub Docs Also for justification see here: Slow pagination over tons of records in mongodb - Stack Overflow

I realize a lot of people still use the older page based pagination system so I would suggest that the premise of this thread of a “single” solution is a bad one.

1 Like
8

I’m not sure which examples you are referring to. The examples above are not for what is sent to the server, but rather what is returned by the rest API. The question is how to get the information about the current page and the total number of pages back to ember, for use in the app.

9

Those are simply to introduce the concept of range based pagination which is widely used, not mentioned previously, and is my direct response to the actual question which is in the title of this thread.

10

My point is that you are not addressing the actual question of the thread. The question is how to receive pagination data from the server, not how to send it to the server. Your comment is not relevant.

11

The question is in the title, I’m simply saying the question and discussion so far seem to me to be missing the fact of range based pagination which might not be familiar to people who have not worked on projects with very large datasets before.

Range pagination (with it’s inherent method of returning page links in the header as I outlined in my first post and which is receiving) is an increasingly common technique that I’m seeing more widely used all the time and therefore I think worthy of consideration in any pagination scheme. I’m bringing it up because I think perhaps the scope of the discussion needs to be widened somewhat before people start discussing specific solutions just to be sure the solution is not short sighted.

I’m not sure what axe you are trying to grind; is there something wrong with discussing this here?

12

It’s off topic. I am currently using range based pagination and it is likely that many other people in this thread are also using range based pagination. However, this is NOT an ember issue. How you implement pagination when sent to the server depends on your server implementation, and while it is a valid concern it is not useful to us here.

We still do not have a standard way to receive pagination data from the server, which is an entirely different component of ember-data. Your comment is adding noise to the signal where we don’t need it.

If you want to contribute to the conversation please stay on topic and suggest a method for serializing the data received from the server so that people can use pagination information in their applications. It makes sense that we receive the total number of pages so that the app can list the page numbers, and the current page so the app knows what page we are currently on.

13

One last time just for you pzuraq:

From the page I linked to:

Link Header

The pagination info is included in the Link header. It is important to follow these Link header values instead of constructing your own URLs. In some instances, such as in the Commits API, pagination is based on SHA1 and not on page number.

Link: https://api.github.com/user/repos?page=3&per_page=100; rel=“next”,
https://api.github.com/user/repos?page=50&per_page=100; rel=“last”

Not relevant?

14

Ok, I understand now that you’re referring to how the REST interface returns the results. Misunderstood the links you posted, my apologies.

I still think that if you followed the actual content of the thread you would find that this conversation has quickly diverged from the plainly stated topic to simply “How do we get meta data from the server?” I’ve found the solution, as it happens, it was implemented in this pull request: https://github.com/emberjs/data/pull/815

The solution is very generic so you can use any pagination system you want, which is all I need personally.

15

FYI: Minimal Pagination Example (Rails + Ember)

16

I do an example like this,but still can’t get pagination

ember.js can't get pagination from metadata - Stack Overflow

17

bump! Is there any ember-team decision on this topic?

4 Likes