# Different robots.txt for production and staging

**URL:** <https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358>\
**Category:** Ember CLI\
**Created:** [December 29, 2015, 11:23am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358 "2015-12-29T11:23:37Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [December 29, 2015, 11:23am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/1 "2015-12-29T11:23:37Z")

</div>

I now have a static `robots.txt` file in my `public` folder. I am thinking about writing a deploy-pack for [ember-cli-deploy-couchdb](https://github.com/martinic/ember-cli-deploy-couchdb). My question is how I can provide a different `robots.txt` depending on deploying for `production` or `staging`? When I take a look at [this](http://ember-cli.com/ember-cli-deploy/docs/v0.5.x/fingerprinting/) `ember-cli-build.js` seems to be te place to specify which `robots.txt` must be used. The question then is where to store the 2 versions? Any help or pointers are welcome.

---

<div class="post-metadata">

**Author:** ![jasonmit](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/jasonmit/32/182_2.png) [@jasonmit](https://discuss.emberjs.com/u/jasonmit)\
**Post date:** [December 30, 2015, 2:18am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/2 "2015-12-30T02:18:59Z")

</div>

`robots-staging.txt` along with `robots-production.txt` inside of the public folder.

Then within `ember-cli-build.js` conditional move and remove one or the other to `robots.txt` using something like `require('broccoli-stew').mv` and `require('broccoli-stew').rm`

---

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [December 30, 2015, 7:17am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/3 "2015-12-30T07:17:03Z")

</div>

> [@jasonmit](#):
>
> broccoli-stew

Thanks! So there is a broccoli tree when ember-cli-build is “run” and not after it. Will try this.

Or perhaps do this in [ember-cli-deploy-build-plus](https://github.com/martinic/ember-cli-deploy-build-plus). Have to think about that because then `context.distFiles` returned by [ember-cli-deploy-build-plus](https://github.com/martinic/ember-cli-deploy-build-plus) must also be changed. Is it possible to somehow “extend” `ember-cli-build.js` or change the broccoli tree before [L44](https://github.com/martinic/ember-cli-deploy-build-plus/blob/master/index.js#L44) to do the moves?

---

<div class="post-metadata">

**Author:** ![jasonmit](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/jasonmit/32/182_2.png) [@jasonmit](https://discuss.emberjs.com/u/jasonmit)\
**Post date:** [December 30, 2015, 8:18am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/4 "2015-12-30T08:18:59Z")

</div>

I’ll throw something together tonight.

---

<div class="post-metadata">

**Author:** ![jasonmit](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/jasonmit/32/182_2.png) [@jasonmit](https://discuss.emberjs.com/u/jasonmit)\
**Post date:** [December 30, 2015, 8:29am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/5 "2015-12-30T08:29:12Z")

</div>

> <https://github.com/jasonmit/conditional-robots-txt/blob/master/ember-cli-build.js#L13-L23>

---

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [December 30, 2015, 1:26pm UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/6 "2015-12-30T13:26:09Z")

</div>

Many thanks! Learning a lot. I think I will not add this to [ember-cli-deploy-build-plus](https://github.com/martinic/ember-cli-deploy-build-plus) because then you would always need to use the deploy pipeline to get the correct robots.txt and that feels wrong.

---

<div class="post-metadata">

**Author:** ![jasonmit](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/jasonmit/32/182_2.png) [@jasonmit](https://discuss.emberjs.com/u/jasonmit)\
**Post date:** [December 30, 2015, 8:09pm UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/7 "2015-12-30T20:09:38Z")

</div>

I don’t know anything about `ember-cli-deploy-build-plus` but since it’s an addon, implement a `treeForPublic` method and move that logic to that hook.

As for whether it belongs in your project, only you can answer that 🙂

---

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [December 31, 2015, 7:38am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/8 "2015-12-31T07:38:48Z")

</div>

Good point. I will try to add it via the treeForPublic hook.

---

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [December 31, 2015, 10:20am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/9 "2015-12-31T10:20:34Z")

</div>

I have added to ember-cli-deploy-build-plus index.js:

```auto
  treeForPublic: function(tree) {
    console.log('treeForPublic ' + tree);
    return this._super.treeForPublic.call(this, tree);
  },

```

in a project that has robots.txt in the public folder but tree is undefined? Do you know what I am missing?

---

<div class="post-metadata">

**Author:** ![jasonmit](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/jasonmit/32/182_2.png) [@jasonmit](https://discuss.emberjs.com/u/jasonmit)\
**Post date:** [December 31, 2015, 10:32am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/10 "2015-12-31T10:32:54Z")

</div>

> [@broerse](#):
>
> treeForPublic: function(tree) { console.log('treeForPublic ’ + tree); return this.\_super.treeForPublic.call(this, tree); },

```js
treeForPublic: function() {
  var tree = this._super.treeForPublic.apply(this, arguments);

  // awesome for debugging trees, don't keep it in your production code :)
  // it will write the tree to the file system in your project dir
  tree = require('broccoli-stew').debug(tree, { name: 'public' });

  return tree;
  ...
}

```

Yeah, it trolled me a bit as well when starting out using these hooks. Also, broccoli-stew

---

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [December 31, 2015, 1:01pm UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/11 "2015-12-31T13:01:46Z")

</div>

It seems tree is stil undefined `Mapper: Expected Broccoli node, got undefined for inputNodes[0]` I am still missing something.

---

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [December 31, 2015, 3:21pm UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/12 "2015-12-31T15:21:54Z")

</div>

I see this [tomster-cli-ember-addon/index.js at master · Gaurav0/tomster-cli-ember-addon · GitHub](https://github.com/Gaurav0/tomster-cli-ember-addon/blob/master/index.js) suggesting sometimes you have a `tree` and sometime you don’t. `tree` is always undefined in my case. Tested with 1.13.13 also. Perhaps addons are always called first. Will explore more.

---

<div class="post-metadata">

**Author:** ![jasonmit](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/jasonmit/32/182_2.png) [@jasonmit](https://discuss.emberjs.com/u/jasonmit)\
**Post date:** [December 31, 2015, 11:23pm UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/13 "2015-12-31T23:23:54Z")

</div>

Guard against an undefined tree, before trying to access it. I’ve seen this as well in my own project:

> <https://github.com/ember-intl/ember-intl/blob/master/index.js#L108-L110>

---

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [January 6, 2016, 11:08am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/14 "2016-01-06T11:08:54Z")

</div>

I wrote this:

[https://github.com/martinic/ember-cli-deploy-build-plus/blob/robots/index.js#L14-L23](https://github.com/martinic/ember-cli-deploy-build-plus/blob/robots/index.js#L14-L23)

The stew.log shows the correct files but they are not merged. Do you know what the correct way to do this is? Also do I need to call \_super?

---

<div class="post-metadata">

**Author:** ![jasonmit](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/jasonmit/32/182_2.png) [@jasonmit](https://discuss.emberjs.com/u/jasonmit)\
**Post date:** [January 7, 2016, 5:56am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/15 "2016-01-07T05:56:16Z")

</div>

Perhaps something like this will work.

```auto
treeForPublic: function() {
  var publicTree = this._super.treeForPublic.apply(this, arguments);

  // if super is invoked and cannot produce a tree then noop
  if (publicTree) {
    var env = this.app.env;

    if (env === 'production') {
      publicTree = stew.rm(publicTree, 'robots.txt');
      publicTree = stew.rename(publicTree, 'robots-production.txt', 'robots.txt');
    } else {
      publicTree = stew.rm(publicTree, 'robots-production.txt');
    }

    return publicTree;
  }
}

```

---

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [January 7, 2016, 9:05am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/16 "2016-01-07T09:05:03Z")

</div>

I tried that already but `publicTree` is always undef. I am still missing something.

---

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [January 7, 2016, 9:28am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/17 "2016-01-07T09:28:53Z")

</div>

Found something. If I create a public folder in the root of my addon `publicTree` is not undef? Perhaps it is a bug?

---

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [January 7, 2016, 9:03pm UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/18 "2016-01-07T21:03:25Z")

</div>

Released [v0.1.5](https://github.com/martinic/ember-cli-deploy-build-plus/tree/v0.1.5) by using postBuild cleanup.

```auto
// Cleanup env specific robots.txt
  postBuild: function(result) {
    cleanupRobotsTxt(result.directory);
  },

```

It works around the possible bug.

---

<div class="post-metadata">

**Author:** ![broerse](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.emberjs.com/broerse/32/9497_2.png) [@broerse](https://discuss.emberjs.com/u/broerse)\
**Post date:** [January 8, 2016, 11:28am UTC](https://discuss.emberjs.com/t/different-robots-txt-for-production-and-staging/9358/19 "2016-01-08T11:28:01Z")

</div>

Added it to my Example

[https://github.com/broerse/ember-cli-blog/commit/dccefb95b95fbcf4c62ec5a2d2739eb5638b31a1](https://github.com/broerse/ember-cli-blog/commit/dccefb95b95fbcf4c62ec5a2d2739eb5638b31a1)