Different robots.txt for production and staging


#1

I now have a static robots.txt file in my public folder. I am thinking about writing a deploy-pack for ember-cli-deploy-couchdb. My question is how I can provide a different robots.txt depending on deploying for production or staging? When I take a look at this ember-cli-build.js seems to be te place to specify which robots.txt must be used. The question then is where to store the 2 versions? Any help or pointers are welcome.


Loading assets (images, ...) depending on environment
#2

robots-staging.txt along with robots-production.txt inside of the public folder.

Then within ember-cli-build.js conditional move and remove one or the other to robots.txt using something like require('broccoli-stew').mv and require('broccoli-stew').rm


#3

Thanks! So there is a broccoli tree when ember-cli-build is “run” and not after it. Will try this.

Or perhaps do this in ember-cli-deploy-build-plus. Have to think about that because then context.distFiles returned by ember-cli-deploy-build-plus must also be changed. Is it possible to somehow “extend” ember-cli-build.js or change the broccoli tree before L44 to do the moves?


#4

I’ll throw something together tonight.


#5

#6

Many thanks! Learning a lot. I think I will not add this to ember-cli-deploy-build-plus because then you would always need to use the deploy pipeline to get the correct robots.txt and that feels wrong.


#7

I don’t know anything about ember-cli-deploy-build-plus but since it’s an addon, implement a treeForPublic method and move that logic to that hook.

As for whether it belongs in your project, only you can answer that :slight_smile:


#8

Good point. I will try to add it via the treeForPublic hook.


#9

I have added to ember-cli-deploy-build-plus index.js:

  treeForPublic: function(tree) {
    console.log('treeForPublic ' + tree);
    return this._super.treeForPublic.call(this, tree);
  },

in a project that has robots.txt in the public folder but tree is undefined? Do you know what I am missing?


#10
treeForPublic: function() {
  var tree = this._super.treeForPublic.apply(this, arguments);

  // awesome for debugging trees, don't keep it in your production code :)
  // it will write the tree to the file system in your project dir
  tree = require('broccoli-stew').debug(tree, { name: 'public' });

  return tree;
  ...
}

Yeah, it trolled me a bit as well when starting out using these hooks. Also, broccoli-stew


#11

It seems tree is stil undefined Mapper: Expected Broccoli node, got undefined for inputNodes[0] I am still missing something.


#12

I see this https://github.com/Gaurav0/tomster-cli-ember-addon/blob/master/index.js suggesting sometimes you have a tree and sometime you don’t. tree is always undefined in my case. Tested with 1.13.13 also. Perhaps addons are always called first. Will explore more.


#13

Guard against an undefined tree, before trying to access it. I’ve seen this as well in my own project:


#14

I wrote this:

https://github.com/martinic/ember-cli-deploy-build-plus/blob/robots/index.js#L14-L23

The stew.log shows the correct files but they are not merged. Do you know what the correct way to do this is? Also do I need to call _super?


#15

Perhaps something like this will work.

treeForPublic: function() {
  var publicTree = this._super.treeForPublic.apply(this, arguments);

  // if super is invoked and cannot produce a tree then noop
  if (publicTree) {
    var env = this.app.env;

    if (env === 'production') {
      publicTree = stew.rm(publicTree, 'robots.txt');
      publicTree = stew.rename(publicTree, 'robots-production.txt', 'robots.txt');
    } else {
      publicTree = stew.rm(publicTree, 'robots-production.txt');
    }

    return publicTree;
  }
}

#16

I tried that already but publicTree is always undef. I am still missing something.


#17

Found something. If I create a public folder in the root of my addon publicTree is not undef? Perhaps it is a bug?


#18

Released v0.1.5 by using postBuild cleanup.

// Cleanup env specific robots.txt
  postBuild: function(result) {
    cleanupRobotsTxt(result.directory);
  },

It works around the possible bug.


#19

Added it to my Example