Ember search engine


#1

Hi all

I am creating an Ember app that has a search engine built into it say for houses. My results change a lot as houses are found / added or removed / sold. Therefore my search results change all the time.

I also have pages for each house which has a similar houses section on it that shows always changing similar houses to this one.

I am trying to find the best way to make this app crawlable to search engines.

I could like discourse use noscript tages for each page but as all my houses pages can hold different information and structure depending on the agent/ seller this would be a lot more work basically duplicating what the client is doing!

I could go down the phantomjs route and cache all my pages and serve via the escaped_fragment method but i am thinking this would be a resource intensive approach with content changing so much. Also with my house pages having similar houses on them that can change depending on the user / location etc, i am not sure how to cache these sections.

Another method i am toying with is to convert my page / section templates into a serverside template so i can render it on the server. For example when a customer creates a house page via my ember app in the format they require they click publish and i convert the rendered html into serverside template with placeholders etc for data.

Anyone help with this ? Any ideas / suggestions / advice would be great !

thanks a lot Rick


#2

You might want to checkout this video about Bustle.com and how they handled searchability.

If you want to explore the idea of rendering on the server side further, you could checkout Embersmith. Embersmith is my project that allows you to use same toolset as you use with Ember to build statically generated content. It can be used to generate content for search engines. In the near future, I’m going to experiment with using the Ember Router as a basis for page generation. I haven’t gotten there yet, but it can generate content from files in the file system.

Also, there is this Ember Ember JS & SEO - Make your App crawlable with the Hashbang Location


#3

thanks for the great reply and help tarasm.

I would love to go with the phantomjs approach but the only thing stopping me is some of my pages are very visiter specific / dynamic. For example if a visiter comes from the US i would present them with houses from the US area and if from UK house from the UK. So i would have to cache these pages often and for each each country.

Unless there is a better way for this ?

thanks


#4

Why not use sub-domains? and actually keep the content different. I think that’s actually better architecturally speaking.


#5

yeah i think i might do that for each language / country thanks.

My main problem i cant quite solve is say i have 100,000 product pages and i cache them all using phantom. That would be fine if the pages content does not change much but … i want to have a similar products section at the bottom of each of these product pages that will show say 10 similar products but that change each each page view or at least very often.

This now becomes a nightmare with caching each page every second/ minute or even hour.

What do you propose to do in this scenario when most of the page doesnt change but you have a very dynamic section at the bottom ?

thanks a lot for all the help Rick


#6

I’ve been thinking about this scenario quite a bit. I’m not sure if this is a viable solution, but its one that I’m exploring right now. I’m thinking about creating components that will query information for that section of the page. There are lots of options available for delivering widget specific area to the page. I would like explore some of those options.


#7

thanks a lot can you tell me some of these options please ?


#8

I haven’t played with any of them and these are very early thoughts, but I’m thinking about pull or streaming data from a service, either my own or a BaaS platform. I’d want to play around with it first before talking about it. Sorry, I don’t have anything specific.


#9

ok no problem thanks a lot again anyway.

Only problem with that approach is its not for the search engines unfortunately.


#10

Do you know how much value those widgets bring to your page? I ask because I’m actually curious. If you could find out, that could be very helpful in understanding what’s important.


#11

well they bring a lot of value to the user as they are similar products to the product that they are interested in. They could also bring value to the SEO of the page as its more keywords etc about the product. I could not display these similar products to search engines but i would then be displaying different content, which maybe seen as bad?


#12

For this, you can serve content after render.

This is the part that I’m curious about. With semantic content, I’m not sure how much emphasis search engine adds to secondary content.

There has to be a limit. For example, users might see a Twitter feed that never shows up to the search engine. Is that bad?


#13

it’s difficult as in my existing app i randomly change the order of similar or matching products on each page view so it’s fair to my customers selling the products. But how do you do that with a Ember app and caching!! ?


#14

Yeah this is the bit i am not sure about as with the thin content etc updates to the google algorithm it crushed traffic to a lot of these pages but i plan on showing more data for each similar product this time.