Different robots.txt for production and staging

Ember CLI
1

I now have a static robots.txt file in my public folder. I am thinking about writing a deploy-pack for ember-cli-deploy-couchdb. My question is how I can provide a different robots.txt depending on deploying for production or staging? When I take a look at this ember-cli-build.js seems to be te place to specify which robots.txt must be used. The question then is where to store the 2 versions? Any help or pointers are welcome.

2

robots-staging.txt along with robots-production.txt inside of the public folder.

Then within ember-cli-build.js conditional move and remove one or the other to robots.txt using something like require('broccoli-stew').mv and require('broccoli-stew').rm

3

Thanks! So there is a broccoli tree when ember-cli-build is “run” and not after it. Will try this.

Or perhaps do this in ember-cli-deploy-build-plus. Have to think about that because then context.distFiles returned by ember-cli-deploy-build-plus must also be changed. Is it possible to somehow “extend” ember-cli-build.js or change the broccoli tree before L44 to do the moves?

4

I’ll throw something together tonight.

5
2 Likes
6

Many thanks! Learning a lot. I think I will not add this to ember-cli-deploy-build-plus because then you would always need to use the deploy pipeline to get the correct robots.txt and that feels wrong.

7

I don’t know anything about ember-cli-deploy-build-plus but since it’s an addon, implement a treeForPublic method and move that logic to that hook.

As for whether it belongs in your project, only you can answer that :slight_smile:

8

Good point. I will try to add it via the treeForPublic hook.

9

I have added to ember-cli-deploy-build-plus index.js:

  treeForPublic: function(tree) {
    console.log('treeForPublic ' + tree);
    return this._super.treeForPublic.call(this, tree);
  },

in a project that has robots.txt in the public folder but tree is undefined? Do you know what I am missing?

10 
treeForPublic: function() {
  var tree = this._super.treeForPublic.apply(this, arguments);

  // awesome for debugging trees, don't keep it in your production code :)
  // it will write the tree to the file system in your project dir
  tree = require('broccoli-stew').debug(tree, { name: 'public' });

  return tree;
  ...
}

Yeah, it trolled me a bit as well when starting out using these hooks. Also, broccoli-stew

11

It seems tree is stil undefined Mapper: Expected Broccoli node, got undefined for inputNodes[0] I am still missing something.

12

I see this tomster-cli-ember-addon/index.js at master · Gaurav0/tomster-cli-ember-addon · GitHub suggesting sometimes you have a tree and sometime you don’t. tree is always undefined in my case. Tested with 1.13.13 also. Perhaps addons are always called first. Will explore more.

13

Guard against an undefined tree, before trying to access it. I’ve seen this as well in my own project:

14

I wrote this:

https://github.com/martinic/ember-cli-deploy-build-plus/blob/robots/index.js#L14-L23

The stew.log shows the correct files but they are not merged. Do you know what the correct way to do this is? Also do I need to call _super?

15

Perhaps something like this will work.

treeForPublic: function() {
  var publicTree = this._super.treeForPublic.apply(this, arguments);

  // if super is invoked and cannot produce a tree then noop
  if (publicTree) {
    var env = this.app.env;

    if (env === 'production') {
      publicTree = stew.rm(publicTree, 'robots.txt');
      publicTree = stew.rename(publicTree, 'robots-production.txt', 'robots.txt');
    } else {
      publicTree = stew.rm(publicTree, 'robots-production.txt');
    }

    return publicTree;
  }
}
16

I tried that already but publicTree is always undef. I am still missing something.

17

Found something. If I create a public folder in the root of my addon publicTree is not undef? Perhaps it is a bug?

18

Released v0.1.5 by using postBuild cleanup.

// Cleanup env specific robots.txt
  postBuild: function(result) {
    cleanupRobotsTxt(result.directory);
  },

It works around the possible bug.

19

Added it to my Example

https://github.com/broerse/ember-cli-blog/commit/dccefb95b95fbcf4c62ec5a2d2739eb5638b31a1