Best way to load a ton of JSON data into an Ember App

Ember Data
1

Building a glossary app with 2000ish definitions. Roughly speaking, each definition includes:

url term speech_type (noun, verb etc) category definition see (cross references) antonyms synonyms img: src, caption

Knowing that, I have 2 questions:

  1. I see a similar topic from '14 that talks about Ember Data moving slow on large payloads. Is that still the case?
  2. How do you recommend deploying the data into production? I’m still pretty new in the Ember world and struggled with the Super-Rentals demo because the basic API is served through Ember Mirage… which is great until you try and deploy it. Does anyone have tips for working with the Amazon API Gateway?

Thanks y’all!

2

Ember shouldn’t have a problem managing 2000 models in it’s store. Earlier I was working on an application that loads about 3000 records on a couple of different pages. Our store grows rather quickly, but there’s no noticeable impact on the UI!

That said, there are many ways large amounts of data can negatively affect a UI. Some questions to ask yourself would be:

  • Is the API/network request for all those records slow?
  • Are you blocking rendering until all 2000 records have been fetched from the server?
  • Are you rendering all records on the screen at once?

In the past I’ve had success with only loading and blocking rendering for the data that’s needed to initially render the page. If you’re only showing one glossary definition you can start rendering as soon as that one definition is loaded and then load the 1999 records in the background.

For example, I’ve got a page that shows support tickets. There’s over 3000 support tickets for this page, but we only display 10 at a time. We still want all of the data loaded into our Ember store to keep sorting, filtering, and search fast.

Since we only need 10 tickets to initially render the page the first step is to fetch all 10 with an API request like:

GET /tickets?page[limit]=10&page[offset]=0

And now we have enough data to start rendering the page.

Next, we have a service that loads the rest of the tickets in the background.

GET /tickets

By the time the user goes to filter or resort the table, all the data has been loaded. The end user has no idea that we’ve split the data loading into two requests to keep the page fast.

This is just one example of many different strategies you use to get your Ember app loading large amounts of of data quickly. I’d recommend starting out with a naive approach, see what’s slow, and then optimize from there.

Last thing, in my experience, SPA have been super awesome for building UIs that depend on lots of data.

3

Thanks for that @ryanto. I just subscribed to Ember Map. :smiley:

Been chatting with @jenweber in Discord about the particular problem I’m trying to solve. She helped me SO MUCH in understanding how to work with the data once it’s loaded into an API, but the question still remains; how to load the data.

Here is the glossary we’re duplicating… turns out it has nearly 5,000 terms/definitions.

I started the day thinking the best approach was JSON+Swagger+Amazon API Gateway+Ember Data. I’m sticking with Ember Data for the asynchronous search features. But, do you have any experience with calling from an Amazon database? I’ve got the JSON here in two forms, just not sure on the best approach as far as where/how to load/query the data.

Any thoughts?

p.s. Amazon is a requirement. Guys in this industry generally don’t trust anything digital, but they do trust Amazon s3/cloud storage.

4

How you load it really depends on your use case. Do you want all of the data to be loaded in the front-end before it’s used? Could you just query it as needed? Or show it in pages? You could fetch it all in one request or fetch in chunks. That’s a pretty hefty payload to grab in one request but maybe it wouldn’t be that bad.

Where the data is served from is all but irrelevant as long as it’s either a standard restful endpoint or websocket. I’ve hit a couple API Gateway endpoints with Ember and it’s really straightforward if you don’t have security concerns (if you do there are obviously some additional complications). You could do it with or without Ember Data, and it may be worth experimenting a little bit. Ember Data is great for many things but it does add a lot of overhead and if you won’t be taking advantage of a lot of the features (it doesn’t sound like you will) it might not be worth said overhead. I’ve seen some pretty big memory impact with like 50k records, and (historically at least) removing data from the store to free the memory can be problematic.

If you’re set on Ember Data you could either a) write a custom adapter for the model and use standard store methods to fetch data or b) fetch with AJAX and do pushPayload into the store so you control the finer points of the fetching (without store/adapter involved) and just use ED for the serializer and store.peek methods. Which serializer you use depends mostly on what your JSON will look like. You may also need to do some extra serialization steps if your payload doesn’t include things like ids.

If you’re not using Ember Data or want to explore what it would look like without ED I’d recommend a service that fetches some/all of the data via ajax and exposes it to your app in a friendly format, optionally with some convenience methods for searching/fetching more/etc. Again the specifics really depend on your use case.

5

Solid. Thank you! Going through a DynamoDB tutorial right now. Seems like it might be a good fit. No manual JSON loading is nice with 5,000 records.

Once I figure out the backend, I’m still trying to conceptualize how to best display the data. Right now, Schlumberger (SLB) has pages for each letter and displays a list of definitions in tables you can click and see the full definition. Creating that many routes is bananas … but I’m not sure how to solve this problem just yet. I’ve thought about maybe having a front page that lists the first handful of words under each letter with a search function for each term… that is good for me because I wouldn’t have to generate all those routes. But it’s not great for the user because having been in the industry as long I have, I know people scroll through the pages looking at the various words and click on ones that jump out at them.

https://www.glossary.oilfield.slb.com/Terms.aspx?filter=a&LookIn=term%20name&searchtype=starts%20with

A collection of Ember Power Selectors for each letter sounds fun… but it also sounds crazy busy.

Just spitballing here, but if you have any examples of ways to display large lists of data in a very small space, I’m all ears.

6

If you wanted to divide it by letter I think you’d still only need a couple routes, probably nested routes. And if you preloaded the data everything would be super fast. You could do something very similar to what they have now, like have a glossary route with a list of letters, clicking on a letter lists all the words with that letter in a nested route (so you still have the letters displayed in the sidebar/top/whatever and can click a different one). Search bar could be a different sub-route or in the parent or child route.

Something like this:

this.route('glossary', function() {
  this.route('letter', { path: '/:letter' });
});

The letter links could look like:

{{link-to glossary.letter "A"}}
{{link-to glossary.letter "B"}}
...

And in the routes/glossary/letter.js you could just return a list of all terms filtered by first letter e.g. <all terms array>.filter(term => term[0] === params.letter);

Then to display a word you could do an subroute of letter, or you could do something else. All that to say I think you’d only need like 3 routes to do what you want.

1 Like
7

Oh wow. Yuge! Thank you!!

8

:+1: definitely feel free to inquire about any other ideas or recommendations or whatever, good luck!

1 Like
9

Okay, working in abstract terms was fun and all, but now struggling with how to develop the adapters and serializers to call our data into the template and been stuck since yesterday morning. Apologize this example is going to be fairly verbose, but it has everything so it’s the example we’re working from.

   "CSG": {
        "url": "https://www.glossary.oilfield.slb.com/en/Terms/c/csg.aspx",
        "term": "CSG",
        "definitions": [
            {
                "speech_type": "n.",
                "category": "Geology",
                "definition": "Abbreviation for coal seam gas. Natural gas, predominantly methane [CH4], generated during coal formation and adsorbed in coal. Natural gas adsorbs to the surfaces of matrix pores within the coal and natural fractures, or cleats, as reservoir pressure increases.\nProduction of natural gas from coal requires decreasing the pore pressure below the coal\u2019s desorption pressure so that methane will desorb from surfaces, diffuse through the coal matrix and become free gas. Because the diffusivity and permeability of the coal matrix are ultralow, coal must have an extensive cleat system to ensure adequate permeability and flow of methane to wellbores at economic production rates.\nCoal seams are typically saturated with water. Consequently, the coal must be dewatered for efficient gas production. Dewatering reduces the hydrostatic pressure and promotes gas desorption from coal. As dewatering progresses, gas production often increases at a rate governed by how quickly gas desorbs from coal, the permeability of the cleat and the relative permeability of the gas-water system in the cleat. Eventually, the rate and amount of gas desorption decreases as the coal seam is depleted of its gas, and production declines.\nCoal seams with no water (dry coal) have been discovered and commercially exploited. In these reservoirs, the adsorbed gas is held in place by free gas in the cleats. Consequently, gas production consists of both free gas from the cleat system and desorbed gas from the matrix.",
                "see": [
                    {
                        "title": "unconventional resource",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/u/unconventional_resource.aspx"
                    },
                    {
                        "title": "seismic trace",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/s/seismic_trace.aspx"
                    },
                    {
                        "title": "trace",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/t/trace.aspx"
                    }
                ],
                "more_details": [
                    {
                        "title": "Learning to Produce Coalbed Methane",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/1991/or1991jan04_methane.aspx"
                    },
                    {
                        "title": "Producing Natural Gas from Coal",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/2003/or2003aut02_gas_from_coal.aspx"
                    },
                    {
                        "title": "Coalbed Methane: Clean Energy for the World",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/2009/or2009sum01_coalbed_methane.aspx"
                    }
                ],
                "synonyms": [
                    {
                        "title": "coalbed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coalbed_methane.aspx"
                    },
                    {
                        "title": "coal bed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal_bed_methane.aspx"
                    },
                    {
                        "title": "coal-bed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal-bed_methane.aspx"
                    },
                    {
                        "title": "CBM",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/cbm.aspx"
                    }
                ],
                "antonyms": [],
                "alternate_forms": "coal seam gas, coal-seam gas",
                "image": {
                    "src": "https://www.glossary.oilfield.slb.com/en/Terms/c/en/~/media/PublicMedia/geology/coalbedMethane01.ashx",
                    "caption": "Gas adsorption and desorption in coal. During coalification, the matrix shrinks, creating orthogonal fractures called cleats. Face cleats tend to be continuous; butt cleats are at right angles to face cleats. Typically, water fills the void spaces of the coal matrix. As the water is produced and the formation pressure decreases, methane\u2014adsorbed on the surfaces of the coal matrix and stored in the micropores\u2014is liberated. The gas then diffuses through the matrix (red arrows), migrates into the cleats and fractures and eventually reaches the wellbore."
                }
            },
            {
                "see": [
                    {
                        "title": "unconventional resource",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/u/unconventional_resource.aspx"
                    },
                    {
                        "title": "seismic trace",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/s/seismic_trace.aspx"
                    },
                    {
                        "title": "trace",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/t/trace.aspx"
                    }
                ],
                "more_details": [
                    {
                        "title": "Learning to Produce Coalbed Methane",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/1991/or1991jan04_methane.aspx"
                    },
                    {
                        "title": "Producing Natural Gas from Coal",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/2003/or2003aut02_gas_from_coal.aspx"
                    },
                    {
                        "title": "Coalbed Methane: Clean Energy for the World",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/2009/or2009sum01_coalbed_methane.aspx"
                    }
                ],
                "synonyms": [
                    {
                        "title": "coalbed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coalbed_methane.aspx"
                    },
                    {
                        "title": "coal bed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal_bed_methane.aspx"
                    },
                    {
                        "title": "coal-bed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal-bed_methane.aspx"
                    },
                    {
                        "title": "CBM",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/cbm.aspx"
                    }
                ],
                "antonyms": [],
                "alternate_forms": "coal seam gas, coal-seam gas",
                "image": {
                    "src": "https://www.glossary.oilfield.slb.com/en/Terms/c/en/~/media/PublicMedia/geology/coalbedMethane01.ashx",
                    "caption": "Gas adsorption and desorption in coal. During coalification, the matrix shrinks, creating orthogonal fractures called cleats. Face cleats tend to be continuous; butt cleats are at right angles to face cleats. Typically, water fills the void spaces of the coal matrix. As the water is produced and the formation pressure decreases, methane\u2014adsorbed on the surfaces of the coal matrix and stored in the micropores\u2014is liberated. The gas then diffuses through the matrix (red arrows), migrates into the cleats and fractures and eventually reaches the wellbore."
                }
            },
            {
                    "speech_type": "n.",
                    "category": "Geophysics",
                    "definition": "Abbreviation for common source gather. A display of seismic\u00a0traces that share a source.",
                    "see": [
                        {
                            "title": "unconventional resource",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/u/unconventional_resource.aspx"
                        },
                        {
                            "title": "seismic trace",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/s/seismic_trace.aspx"
                        },
                        {
                            "title": "trace",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/t/trace.aspx"
                        }
                    ],
                    "more_details": [
                        {
                            "title": "Learning to Produce Coalbed Methane",
                            "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/1991/or1991jan04_methane.aspx"
                        },
                        {
                            "title": "Producing Natural Gas from Coal",
                        },
                        {
                            "title": "Coalbed Methane: Clean Energy for the World",
                            "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/2009/or2009sum01_coalbed_methane.aspx"
                        }
                    ],
                    "synonyms": [
                        {
                            "title": "coalbed methane",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coalbed_methane.aspx"
                        },
                        {
                            "title": "coal bed methane",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal_bed_methane.aspx"
                        },
                        {
                            "title": "coal-bed methane",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal-bed_methane.aspx"
                        },
                        {
                            "title": "CBM",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/cbm.aspx"
                        }
                    ],
                    "antonyms": [],
                    "alternate_forms": "coal seam gas, coal-seam gas",
                    "image": {
                        "src": "https://www.glossary.oilfield.slb.com/en/Terms/c/en/~/media/PublicMedia/geology/coalbedMethane01.ashx",
                        "caption": "Gas adsorption and desorption in coal. During coalification, the matrix shrinks, creating orthogonal fractures called cleats. Face cleats tend to be continuous; butt cleats are at right angles to face cleats. Typically, water fills the void spaces of the coal matrix. As the water is produced and the formation pressure decreases, methane\u2014adsorbed on the surfaces of the coal matrix and stored in the micropores\u2014is liberated. The gas then diffuses through the matrix (red arrows), migrates into the cleats and fractures and eventually reaches the wellbore."
                    }
                }
            ]
        },

Given this JSON input and all of its nested relationships, do you have any thoughts on what the models should look like? Really struggling on this one.

Thanks!

10

Here is the link to the definition in the site we are cloning if it helps at all: CSG | Energy Glossary

11

What a difference a few hours can make. Getting this together into a Twiddle. Now just to figure out how to use mock.json to cue up the example. :thinking:

12

Ok, so the main question is, what kind of search are you doing and how do you want to display data?

I don’t think it’s strictly necessary to display most of this as relationships if it is read-only. I would probably just let it have some attributes that are POJOs (plain old javascript objects). Also look into Ember Data Model Fragments: GitHub - adopted-ember-addons/ember-data-model-fragments: Ember Data addon to support nested JSON documents

If I wanted to, I could load in any json object into a single model and display its deeply nested attributes in a template:

Title: {{model.title}}
Definitions:
{{#each model.data.definitions as |def|}}
{{def.something.whatever}}
{{/each}}

I have done this plenty of times for prototypes. There may be production concerns.

13

Ember Data Model Fragments. Strong!

Doing our best to clone SLB’s glossary, which is very straight-forward. However, the search is one place I would like to drastically improve. Ideally, it would operate like Discourse or the EmberJS API search (when I can get it work). The way it works now is far too clunky and time-consuming so definitely looking for the async sauce.

Will look into fragments and see how far it takes me.

Thank you!

14

@jenweber to the rescuuuuuue!! Oh my gosh. I promised myself I wouldn’t cry!!

It’s just. so. beautiful. :sob:

Thank you Jen!!!

15

Hi James, sorry but I’m on vacation atm and only have my phone so I won’t be able to respond until next Friday. Perhaps someone else can weigh in in the meantime.

1 Like
16

You’re good man, we got it figured out. Have a fantastic vacation!

17

Alright @dknutsen, I’m on this step. Only took a full year… I’m :clap: slow :clap:

When I have a standard route skeleton with nothing else in terms/letter/route.js I’m able to enter {{model.letter}} into the template and successfully return the correct letter for the route. For example, /terms/o correctly displays the letter “o”. As fun as that is, we obviously need every term that starts with “o”, not just the letter.

After toying around this.store.findAll('term') for a while I came back to this thread hoping I might could filter the terms. So now the letter route.js looks like:

import Route from '@ember/routing/route';

export default Route.extend({
  model(params) {
    return this.store.findAll('term').filter(term => term[0] === params.letter);
}
});

This properly sends a get to /api/terms, but the results are not filtered.

44
441323×621 131 KB

What am I doing wrong?

18

If you’re keeping this in mirage for the foreseeable future I’d say this approach is probably fine, but if you are ever querying a real backend you’d probably want to do the “first letter” filtering on the backend (will greatly decrease the amount of data you send over the wire and the amount that the front-end has to cache). So, you have two options:

  1. you could add another filter “letter” to your mirage route handler

  2. I think you just have a minor error in the above code, so while it’s less efficient you could still use that strategy for now (fetch all and filter on front-end) and may just need a slight tweak:

return this.store.findAll('term').filter(term => term[0] === params.letter);

In this code I think you’re saying “filter the 0-th index of the term” but term isn’t actually the term it’s the entire term model, so I think what you want is term.term[0], or rather:

return this.store.findAll('term').filter(term => term.term[0] === params.letter);

Which is like saying “the first character of the term model’s term attribute”

19

Added term.term and it still brings back an array of 29. Added this.get("/terms/:letter") to mirage and it does the same. I’m thinking needs to be a mirage solution. But I’ve tried various adapters and serializers, along with refactoring your search code and not getting anywhere. Sometimes I wonder if I should create a separate letter model but that seems like overkill.

20

Hmmmm in that case I’d definitely just go with the mirage solution, try changing your mirage “/terms” handler to something like this:

  this.get("/terms", function({ terms }, { queryParams }) {
    let { term, letter } = queryParams;
    let results = terms.all();
    if (letter) {
      const match = letter.toLowerCase();
      results = results.filter(t => t.term.toLowerCase()[0] === match);
    }
    if (term) {
      const match = term.toLowerCase();
      results = results.filter(t => t.term.toLowerCase().includes(match));
    }
    return results;
  });

Then in the model hook just query like this:

  model(params) {
    return this.store.query('term', { letter: params.letter });
  }
1 Like