How to normalize this response?


#1

I’m having trouble understanding how to normalize this since each date is different…?

{
    "Meta Data": {
    "1. Information": "Monthly Adjusted Prices and Volumes",
    "2. Symbol": "MSFT",
    "3. Last Refreshed": "2018-08-01 12:26:23",
    "4. Time Zone": "US/Eastern"
    },
    "Monthly Adjusted Time Series": {
    "2018-08-01": {
    "1. open": "106.0300",
    "2. high": "106.4460",
    "3. low": "105.4200",
    "4. close": "105.9550",
    "5. adjusted close": "105.9550",
    "6. volume": "11014901",
    "7. dividend amount": "0.0000"
    },
    "2018-07-31": {
    "1. open": "98.1000",
    "2. high": "111.1500",
    "3. low": "98.0000",
    "4. close": "106.0800",
    "5. adjusted close": "106.0800",
    "6. volume": "569013453",
    "7. dividend amount": "0.0000"
    },
    "2018-06-29": {
    "1. open": "99.2798",
    "2. high": "102.6900",
    "3. low": "97.2600",
    "4. close": "98.6100",
    "5. adjusted close": "98.6100",
    "6. volume": "602585341",
    "7. dividend amount": "0.0000"
    },

#2

Guess it depends on what you’re trying to normalize into and what you want to do with it but I’d probably normalize the time series data something like:

{
    "date": "2018-08-01",
    "open": "106.0300",
    "high": "106.4460",
    "low": "105.4200",
    "close": "105.9550",
    "adjustedClose": "105.9550",
    "volume": "11014901",
    "dividendAmount": "0.0000"
}

#3

I think I might approach this as a very custom serializer. Relevent code posted here with a working demo at ember-twiddle.

import { camelize } from '@ember/string';
import DS from 'ember-data';
const { Serializer } = DS;

const META_KEY = 'Meta Data';
const DATA_KEY = 'Monthly Adjusted Time Series';

let nextID = 0;

export default Serializer.extend({
  normalizeKey(keyName) {
    return camelize(keyName.replace(/\d+\.\s+/, ''));
  },
  
  normalizeResponse(store, primaryModelClass, payload, id, requestType) {
    let meta = {};
    let data = [];
    for (let key of Object.keys(payload[META_KEY] || {})) {
      meta[this.normalizeKey(key)] = payload[META_KEY][key];
    }
    for (let date of Object.keys(payload[DATA_KEY] || {})) {
      let payloadItem = payload[DATA_KEY][date];
      payloadItem.date = date;
      data.push(this.normalize(primaryModelClass, payloadItem));
    }
    return { data, meta };
  },
  
  normalize(primaryModelClass, item) {
    nextID++;
    let id = nextID;
    let type = 'time-data';
    let attributes = {};
    for (let key of Object.keys(item)) {
      attributes[this.normalizeKey(key)] = item[key];
    }
    return { id, type, attributes };
  }
});

#4

@sukima Thanks for the in-depth example, this is super helpful. This is my first time really using ember-data to this extent. Obviously you can see here, I’m trying to work with stock data. What would change if I wanted to request data for a multiple stocks, given your example? This data isn’t coming from my server, but a third-party URL. I see how a serializer is ideal to normalize responses in this situation, but I’m having a hard time wrapping my brain around on how a single model and adapter can work for unique sets of data for each stock?

@dknutsen Hope this kind answers your question:
Ideally my investments model would look something like this after running it through the serializer. I apologize for this noob question, but given that I know I’ll only be working with 20 different stocks for this project, should each stock be it’s own model because each stock will have it’s own historical data record?

{
  "investments": {
    "StockA": {
      "historical-data": {
        "2017-07-31": {
          "adjClose": "10.05"
        },
        "2017-07-30": {
          "adjClose": "10.05"
        }
      }
    },
    "StockB": {
      "historical-data": {
        "2017-07-31": {
          "adjClose": "10.05"
        },
        "2017-07-30": {
          "adjClose": "10.05"
        }
      }
    }
}

#5

Off the top of my head I’d think you’d want a model for stock and a model for historical-data. Each stock (StockA, StockB, etc.) would be an instance of your “stock” model. The stock model would have a hasMany(‘historical-data’). Then if you had a model that contained your “investments”, user or portfolio or whatever, that would have a key called investments which would be hasMany(‘stock’).

Each ember model should be pretty flat (e.g. not have nested data that isn’t a relationship). There are exceptions but it’s a good rule of thumb. So when thinking about how to break data down into models and relationships consider what each “chunk” of data is and how those chunks relate. When looking at this I see an investment as any number of “stocks” (so naturally a hasMany relationship) and each “stock” as having any number of historical data items (so naturally another hasMany).

You’re using a 3rd party API so you don’t have control over the responses and you’re going to have to do a lot of wrangling. I’d say your Ember Data schema is going to look completely different from the API. What you want to do is try and extract some reasonable models from the data you have access to, and then figure out how to munge the responses from the API into a JSON API formatted response that will normalize into your ember models. You’ll also have to figure out how you want the requests to be made. You could customize some adapters if that fits the API at all, or you could perform raw ajax requests and do a pushPayload and let the serializer layer take over. You’ll also have to consider what to do if your app will be writing data to the API or if it’s “read only”. If you’re going to be making PUT/POST/PATCH requests you’ll have to do more adapter and serializer customization.

Another thing to watch out for in your serializer is that Ember Data expects ids for each model, so if the API doesn’t give you one you’ll have to manufacture one in the normalization process. (for example in the data above you could combine the historical data date with the stock name or id e.g. “StockB_2017-07-31”.

Point is, this is a fairly complex use case for serialization and there are a lot of paths you could go down. Making good high-level decisions will help you a lot. That mostly comes down to breaking the data down into sensible models, which again should be pretty flat. The sample you just posted would definitely not be serialized enough for ember data to consume. You’ll need model types, ids, etc and they’ll need to be munged into a more specific format (e.g. JSON API).

You can also always forego ember data. Really depends on how much you’re trying to do with the data and whether or not you’ll derive enough benefit from the persistent store and model system.

Sorry that was kind of long and rambly, if you have some sample API responses or other specific questions we’d be happy to try and point you in the right direction. One thing that can be helpful when approaching problems like this is having the API response and then having some general idea of what you want to render. Then it’s just a matter of filling in the middle with what makes the most sense.


#6

@dknutsen No need to apologize, thank you for the long explanation, I am a financial planner and not a professional developer after all, so I need all the help I can get lol. I’m actually kind of glad you acknowledged that this is pretty complex, because I was thinking I just wasn’t smart enough for this stuff! Everything you said made sense though. I did get hung up on whether ember-data was even good for this project, but I think I’m stuck without it? The idea of making a model for the historical data really makes sense.

This is a real API response I’d be working with: https://www.alphavantage.co/query?function=TIME_SERIES_MONTHLY_ADJUSTED&symbol=MSFT&apikey=demo

So one specific question I have is where to store Portfolio Allocation percentages, because each portfolio is made up of different funds. It’ll probably also help to understand what I’m really trying to create, which is a table that lists time-interval performance/returns for a handful of portfolios where each portfolio has a number stocks/mutual funds, etc as you guessed. So right now this is how I do this without using ember-data…

// performance.js route    
model() {        
        return RSVP.hash({
            SLGAX: this.get("ajax") // SEI LARGE CAP
                .request("https://www.alphavantage.co/query?function=TIME_SERIES_MONTHLY_ADJUSTED&symbol=SLGAX&apikey=**", {
                })
                .then(SLGAX => {   
                    return SLGAX;
                }),
            // and so forth forth for each mutual fund

Then I created a computed property for each time period performance in my component (I know now this should probably be done in a controller). 5-6 lines of code for each computed property, times a couple dozen stocks/mutual funds… that’s a lot of repeated code!

SLGAXMonthToDate: computed('model.SLGAX', function() {
    let SLGAX = this.model.SLGAX;
    let listOfDates  = Object.keys(SLGAX['Monthly Adjusted Time Series'])
    let finalAdjClose = parseFloat(SLGAX['Monthly Adjusted Time Series'][listOfDates[0]]['5. adjusted close']);
    let lastMonthAdjClose = parseFloat(SLGAX['Monthly Adjusted Time Series'][listOfDates[1]]['5. adjusted close']);
    return ((((finalAdjClose - lastMonthAdjClose) / lastMonthAdjClose) * 100).toFixed(2))
}),

It get’s even more complex knowing that each portfolio is made up different mutual funds, but they all have a different weight in the portfolio. So Portfolio A has Stock A and Stock B, but it isn’t 50/50. Stock A might make up 60% of the portfolio, for example.

Ideally a user could create a Portfolio in the app, add funds to it and specify the % weight of each fund, and the app would add that portfolio to the table and calculate the returns for each time period. Thanks so much for being willing to provide me with some guidance here, especially with the conceptual questions! I really appreciate it.

Here’s the table I’ve already created just for monitoring returns of each individual fund. It works well for what it is, albeit the amount of computed property code is over the top.


#7

I have questions. In order to use a third-party API they would need to enable CORS. I find it a little hard to believe that they have taken the time to enable CORS but not the time to reconsider their choices in that JSON payload. The sample data is rather… concerning? It does not reflect typical/standard JSON schemas.

Unless there is a proxy server in between in which case the proxy server could simple convert the payload into something a bit more sane to work with (i.e. JSON:API). If this is the case then the conversion could be handled in any language one wishes without the constraints of ember-data and client side JavaScript.

Maybe this reply is bike-shedding?


#8

@sukima, you’re not wrong and I think most people agree: