Since questions 2 and 3 are pretty straightforward I’ll just take a swing at those before getting into the other stuff which is a longer answer…
The second question: I need to use filters when querying lookup data. However I have a need to filter data after the initial query. I think I need to use peekAll but I’m not sure how to filter “after the fact”
Yes! You’re definitely on the right track. Filtering after the fact is usually done via a combination of peekAll and then computed properties. You could write a single filter function/CP or use all of the fun computed macros to do the filtering. Assuming you’re filtering based on UI controls or something you would probably do this in a controller or component that has the values of your inputs and the data from your peekAll (if it’s a component you could either pass the peekAll into the component or inject the store and use it inside the component. The nice thing about peekAll is that it’s a live array so you can make a CP chain like so:
things: computed(function() {
return this.store.peekAll('some-model');
}),
// returns all "things" with property "truthy" equal to true, updates as the things live-array changes
onlyTrueThings: filterBy('things', 'truthy', true);
otherFilteredThings: computed('things.@each.{name,style}', function() {
// return things filtered by name and/or style
})
The third question: Is there a way I can use a combination of peekAll and query to only issue a network request when there are no targeted models in the store?
If I’m understanding the need here (filter what’s in the store down and if there are no records that meet the filter run a new query) then yes, you could definitely do this. The only part that may be more complex is how to trigger the query. That will kind of depend on your architecture. You could do the filtering in the route and use that to determine whether or not to requery but if the filters are determined by your UI that doesn’t make much sense. You could also do it in a component but then you have to handle the async query states properly, perhaps using ember-concurrency.
But anyway that brings us back around to the first question…
How do people deal with queries that return tens of thousands of records/models?
In my experience (and take this with a grain of salt because this is anecdotal) you probably want to consider not using Ember Data for this particular query/record set. It’s not necessarily obvious from the guides and general community and documentation but Ember Data is not only not required but it’s also not recommended for some things. I worked for a company that made trading software for a while and we stored lots of market data records in ember data (could easily get into the tens of thousands of records, sometimes hundreds of thousands) and while it actually performed pretty well in general on desktop we definitely had a lot of problems keeping it in check and lower power devices really struggled. Memory is probably the biggest thing at that scale. Each ember data record has its own little state machine and keeps a lot of data other than just the model attributes and relationships that we all know and love. This adds up pretty quickly when you’re talking about that many records. There have also been issues, historically at least, safely and effectively unloading records from the store.
Ember Data is great for a lot of things, especially stuff like the canonical blog apps or apps where you’re doing full CRUD operations on heavier data records, but that comes with a tradeoff for situations like yours. Especially if you don’t need much of the “state machine” part of Ember Data or this data won’t be making heavy use of relationships, I think it would make a lot more sense to keep your data outside of Ember Data.
So let’s say you decide not to use Ember Data… there are a lot of directions you could go… you could return your data from a route model hook just the same as usual. If you want to bind the query to query params that might make the most sense. If not though I’d say consider a service and potentially an ember-concurrency task (will make the UI portion simpler if nothing else) or a method that fetches the data via a more standard XHR request and returns a promise or promise proxy. Then you can have filter properties and CPs directly on the service. The advantage to this direction is that all of the logic is contained in one place and you can inject it into any route/controller/component that you need.
Anyway it’s hard and probably confusing to go into more details without more context but my personal recommendation would be to really consider what parts of Ember Data you need in this case and if it’s not much maybe sidestep it altogether. Then determine a broader architecture for the query and filtering needs. Definitely happy to provide thoughts (and again I want to be clear that this is just my take and others may have better feedback) or suggestions for any of the implementation.
