Best way to load a ton of JSON data into an Ember App

If you wanted to divide it by letter I think you’d still only need a couple routes, probably nested routes. And if you preloaded the data everything would be super fast. You could do something very similar to what they have now, like have a glossary route with a list of letters, clicking on a letter lists all the words with that letter in a nested route (so you still have the letters displayed in the sidebar/top/whatever and can click a different one). Search bar could be a different sub-route or in the parent or child route.

Something like this:

this.route('glossary', function() {
  this.route('letter', { path: '/:letter' });
});

The letter links could look like:

{{link-to glossary.letter "A"}}
{{link-to glossary.letter "B"}}
...

And in the routes/glossary/letter.js you could just return a list of all terms filtered by first letter e.g. <all terms array>.filter(term => term[0] === params.letter);

Then to display a word you could do an subroute of letter, or you could do something else. All that to say I think you’d only need like 3 routes to do what you want.

1 Like

Oh wow. Yuge! Thank you!!

:+1: definitely feel free to inquire about any other ideas or recommendations or whatever, good luck!

1 Like

Okay, working in abstract terms was fun and all, but now struggling with how to develop the adapters and serializers to call our data into the template and been stuck since yesterday morning. Apologize this example is going to be fairly verbose, but it has everything so it’s the example we’re working from.

   "CSG": {
        "url": "https://www.glossary.oilfield.slb.com/en/Terms/c/csg.aspx",
        "term": "CSG",
        "definitions": [
            {
                "speech_type": "n.",
                "category": "Geology",
                "definition": "Abbreviation for coal seam gas. Natural gas, predominantly methane [CH4], generated during coal formation and adsorbed in coal. Natural gas adsorbs to the surfaces of matrix pores within the coal and natural fractures, or cleats, as reservoir pressure increases.\nProduction of natural gas from coal requires decreasing the pore pressure below the coal\u2019s desorption pressure so that methane will desorb from surfaces, diffuse through the coal matrix and become free gas. Because the diffusivity and permeability of the coal matrix are ultralow, coal must have an extensive cleat system to ensure adequate permeability and flow of methane to wellbores at economic production rates.\nCoal seams are typically saturated with water. Consequently, the coal must be dewatered for efficient gas production. Dewatering reduces the hydrostatic pressure and promotes gas desorption from coal. As dewatering progresses, gas production often increases at a rate governed by how quickly gas desorbs from coal, the permeability of the cleat and the relative permeability of the gas-water system in the cleat. Eventually, the rate and amount of gas desorption decreases as the coal seam is depleted of its gas, and production declines.\nCoal seams with no water (dry coal) have been discovered and commercially exploited. In these reservoirs, the adsorbed gas is held in place by free gas in the cleats. Consequently, gas production consists of both free gas from the cleat system and desorbed gas from the matrix.",
                "see": [
                    {
                        "title": "unconventional resource",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/u/unconventional_resource.aspx"
                    },
                    {
                        "title": "seismic trace",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/s/seismic_trace.aspx"
                    },
                    {
                        "title": "trace",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/t/trace.aspx"
                    }
                ],
                "more_details": [
                    {
                        "title": "Learning to Produce Coalbed Methane",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/1991/or1991jan04_methane.aspx"
                    },
                    {
                        "title": "Producing Natural Gas from Coal",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/2003/or2003aut02_gas_from_coal.aspx"
                    },
                    {
                        "title": "Coalbed Methane: Clean Energy for the World",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/2009/or2009sum01_coalbed_methane.aspx"
                    }
                ],
                "synonyms": [
                    {
                        "title": "coalbed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coalbed_methane.aspx"
                    },
                    {
                        "title": "coal bed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal_bed_methane.aspx"
                    },
                    {
                        "title": "coal-bed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal-bed_methane.aspx"
                    },
                    {
                        "title": "CBM",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/cbm.aspx"
                    }
                ],
                "antonyms": [],
                "alternate_forms": "coal seam gas, coal-seam gas",
                "image": {
                    "src": "https://www.glossary.oilfield.slb.com/en/Terms/c/en/~/media/PublicMedia/geology/coalbedMethane01.ashx",
                    "caption": "Gas adsorption and desorption in coal. During coalification, the matrix shrinks, creating orthogonal fractures called cleats. Face cleats tend to be continuous; butt cleats are at right angles to face cleats. Typically, water fills the void spaces of the coal matrix. As the water is produced and the formation pressure decreases, methane\u2014adsorbed on the surfaces of the coal matrix and stored in the micropores\u2014is liberated. The gas then diffuses through the matrix (red arrows), migrates into the cleats and fractures and eventually reaches the wellbore."
                }
            },
            {
                "see": [
                    {
                        "title": "unconventional resource",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/u/unconventional_resource.aspx"
                    },
                    {
                        "title": "seismic trace",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/s/seismic_trace.aspx"
                    },
                    {
                        "title": "trace",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/t/trace.aspx"
                    }
                ],
                "more_details": [
                    {
                        "title": "Learning to Produce Coalbed Methane",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/1991/or1991jan04_methane.aspx"
                    },
                    {
                        "title": "Producing Natural Gas from Coal",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/2003/or2003aut02_gas_from_coal.aspx"
                    },
                    {
                        "title": "Coalbed Methane: Clean Energy for the World",
                        "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/2009/or2009sum01_coalbed_methane.aspx"
                    }
                ],
                "synonyms": [
                    {
                        "title": "coalbed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coalbed_methane.aspx"
                    },
                    {
                        "title": "coal bed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal_bed_methane.aspx"
                    },
                    {
                        "title": "coal-bed methane",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal-bed_methane.aspx"
                    },
                    {
                        "title": "CBM",
                        "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/cbm.aspx"
                    }
                ],
                "antonyms": [],
                "alternate_forms": "coal seam gas, coal-seam gas",
                "image": {
                    "src": "https://www.glossary.oilfield.slb.com/en/Terms/c/en/~/media/PublicMedia/geology/coalbedMethane01.ashx",
                    "caption": "Gas adsorption and desorption in coal. During coalification, the matrix shrinks, creating orthogonal fractures called cleats. Face cleats tend to be continuous; butt cleats are at right angles to face cleats. Typically, water fills the void spaces of the coal matrix. As the water is produced and the formation pressure decreases, methane\u2014adsorbed on the surfaces of the coal matrix and stored in the micropores\u2014is liberated. The gas then diffuses through the matrix (red arrows), migrates into the cleats and fractures and eventually reaches the wellbore."
                }
            },
            {
                    "speech_type": "n.",
                    "category": "Geophysics",
                    "definition": "Abbreviation for common source gather. A display of seismic\u00a0traces that share a source.",
                    "see": [
                        {
                            "title": "unconventional resource",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/u/unconventional_resource.aspx"
                        },
                        {
                            "title": "seismic trace",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/s/seismic_trace.aspx"
                        },
                        {
                            "title": "trace",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/t/trace.aspx"
                        }
                    ],
                    "more_details": [
                        {
                            "title": "Learning to Produce Coalbed Methane",
                            "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/1991/or1991jan04_methane.aspx"
                        },
                        {
                            "title": "Producing Natural Gas from Coal",
                        },
                        {
                            "title": "Coalbed Methane: Clean Energy for the World",
                            "link": "http://www.slb.com/resources/publications/industry_articles/oilfield_review/2009/or2009sum01_coalbed_methane.aspx"
                        }
                    ],
                    "synonyms": [
                        {
                            "title": "coalbed methane",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coalbed_methane.aspx"
                        },
                        {
                            "title": "coal bed methane",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal_bed_methane.aspx"
                        },
                        {
                            "title": "coal-bed methane",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/coal-bed_methane.aspx"
                        },
                        {
                            "title": "CBM",
                            "link": "https://www.glossary.oilfield.slb.com/en/Terms/c/cbm.aspx"
                        }
                    ],
                    "antonyms": [],
                    "alternate_forms": "coal seam gas, coal-seam gas",
                    "image": {
                        "src": "https://www.glossary.oilfield.slb.com/en/Terms/c/en/~/media/PublicMedia/geology/coalbedMethane01.ashx",
                        "caption": "Gas adsorption and desorption in coal. During coalification, the matrix shrinks, creating orthogonal fractures called cleats. Face cleats tend to be continuous; butt cleats are at right angles to face cleats. Typically, water fills the void spaces of the coal matrix. As the water is produced and the formation pressure decreases, methane\u2014adsorbed on the surfaces of the coal matrix and stored in the micropores\u2014is liberated. The gas then diffuses through the matrix (red arrows), migrates into the cleats and fractures and eventually reaches the wellbore."
                    }
                }
            ]
        },

Given this JSON input and all of its nested relationships, do you have any thoughts on what the models should look like? Really struggling on this one.

Thanks!

Here is the link to the definition in the site we are cloning if it helps at all: https://www.glossary.oilfield.slb.com/Terms/c/csg.aspx

What a difference a few hours can make. Getting this together into a Twiddle. Now just to figure out how to use mock.json to cue up the example. :thinking:

https://ember-twiddle.com/8fa056c3d3a381084ad6bfffefe0b86f

Ok, so the main question is, what kind of search are you doing and how do you want to display data?

I don’t think it’s strictly necessary to display most of this as relationships if it is read-only. I would probably just let it have some attributes that are POJOs (plain old javascript objects). Also look into Ember Data Model Fragments: https://github.com/lytics/ember-data-model-fragments

If I wanted to, I could load in any json object into a single model and display its deeply nested attributes in a template:

Title: {{model.title}}
Definitions:
{{#each model.data.definitions as |def|}}
{{def.something.whatever}}
{{/each}}

I have done this plenty of times for prototypes. There may be production concerns.

Ember Data Model Fragments. Strong!

Doing our best to clone SLB’s glossary, which is very straight-forward. However, the search is one place I would like to drastically improve. Ideally, it would operate like Discourse or the EmberJS API search (when I can get it work). The way it works now is far too clunky and time-consuming so definitely looking for the async sauce.

Will look into fragments and see how far it takes me.

Thank you!

@jenweber to the rescuuuuuue!! Oh my gosh. I promised myself I wouldn’t cry!!

It’s just. so. beautiful. :sob:

Thank you Jen!!!

https://ember-twiddle.com/7bc08693bab528e5b93e902e970437e5

Hi James, sorry but I’m on vacation atm and only have my phone so I won’t be able to respond until next Friday. Perhaps someone else can weigh in in the meantime.

1 Like

You’re good man, we got it figured out. Have a fantastic vacation!

Alright @dknutsen, I’m on this step. Only took a full year… I’m :clap: slow :clap:

When I have a standard route skeleton with nothing else in terms/letter/route.js I’m able to enter {{model.letter}} into the template and successfully return the correct letter for the route. For example, /terms/o correctly displays the letter “o”. As fun as that is, we obviously need every term that starts with “o”, not just the letter.

After toying around this.store.findAll('term') for a while I came back to this thread hoping I might could filter the terms. So now the letter route.js looks like:

import Route from '@ember/routing/route';

export default Route.extend({
  model(params) {
    return this.store.findAll('term').filter(term => term[0] === params.letter);
}
});

This properly sends a get to /api/terms, but the results are not filtered.

What am I doing wrong?

If you’re keeping this in mirage for the foreseeable future I’d say this approach is probably fine, but if you are ever querying a real backend you’d probably want to do the “first letter” filtering on the backend (will greatly decrease the amount of data you send over the wire and the amount that the front-end has to cache). So, you have two options:

  1. you could add another filter “letter” to your mirage route handler

  2. I think you just have a minor error in the above code, so while it’s less efficient you could still use that strategy for now (fetch all and filter on front-end) and may just need a slight tweak:

return this.store.findAll('term').filter(term => term[0] === params.letter);

In this code I think you’re saying “filter the 0-th index of the term” but term isn’t actually the term it’s the entire term model, so I think what you want is term.term[0], or rather:

return this.store.findAll('term').filter(term => term.term[0] === params.letter);

Which is like saying “the first character of the term model’s term attribute”

Added term.term and it still brings back an array of 29. Added this.get("/terms/:letter") to mirage and it does the same. I’m thinking needs to be a mirage solution. But I’ve tried various adapters and serializers, along with refactoring your search code and not getting anywhere. Sometimes I wonder if I should create a separate letter model but that seems like overkill.

Hmmmm in that case I’d definitely just go with the mirage solution, try changing your mirage “/terms” handler to something like this:

  this.get("/terms", function({ terms }, { queryParams }) {
    let { term, letter } = queryParams;
    let results = terms.all();
    if (letter) {
      const match = letter.toLowerCase();
      results = results.filter(t => t.term.toLowerCase()[0] === match);
    }
    if (term) {
      const match = term.toLowerCase();
      results = results.filter(t => t.term.toLowerCase().includes(match));
    }
    return results;
  });

Then in the model hook just query like this:

  model(params) {
    return this.store.query('term', { letter: params.letter });
  }
1 Like

Man this is EXACTLY what I’ve been trying to do for days. Once again, thank you sir!!

Oh goodness. If it’s not one thing it’s another.

Guessing I need to do one of these things to solve this problem?

  1. Create an onclick action that forces a refresh.
  2. Add a transitionTo for the links.

Fixed it! Thanks to another one of your answers. :nerd_face:

Added another route above terms to separate term/letter from terms/letter/term_id.

Router.map(function() {
  this.route("home", { path: "/" });
  this.route("term", function() {
    this.route("letter", { path: "/:letter" })
  });
  this.route("terms", function() {
    this.route("letter", { path: "/:letter" }, function() {
      this.route("term_id", { path: "/:term_id" });
    });
  });
});

Then just added a new this.get to the mirage config.

export default function() {
  this.namespace = "api";
  this.get("/term", function({ terms }, { queryParams }) {
    let { letter } = queryParams;
    let results = terms.all();
    if (letter) {
      const match = letter.toLowerCase();
      results = results.filter(t => t.term.toLowerCase()[0] === match);
    }
    return results;
  });
  this.get("/terms", function({ terms }, { queryParams }) {
    let { term, letter } = queryParams;
    let results = terms.all();
    if (letter) {
      const match = letter.toLowerCase();
      results = results.filter(t => t.term.toLowerCase()[0] === match);
    }
    if (term) {
      const match = term.toLowerCase();
      results = results.filter(t => t.term.toLowerCase().includes(match));
    }
    return results;
  });
this.get("/terms/:id");
}

Works like a charm. POW!

Have a great weekend!!

Woohoo! :tada: glad you got it working, that’s always a great feeling.

1 Like

'Twas outstanding. Now to make it play nice with this.