Feature for adding noindex, nofollow meta tags in non-production (OSOE-621)

[Piedone]

Staging and other non-production environments can be open to the public internet. To prevent search bots from indexing these (and thus duplicating production content), let's add a new module that when enabled, adds a noindex, nofollow meta tag when the app is not running in a production environment.

<meta name="robots" content="noindex, nofollow">

The environment can be checked with IHostEnvironment.IsProduction(). However, add a separate configuration coming from a configuration provider that indicates if the current environment is production or non-production, in case ASP.NET Core should be told it runs in Production but not actually (like in staging, that should be as close to production as possible).

Jira issue

[DemeSzabolcs]

Any good name suggestions? SearchBotBlocker, NoIndexNoFollow, EnvironmentMetaTag. I think SearchBotBlocker is fairly accurate.

[DemeSzabolcs]

Must we use the meta tag? I mean this is easily solvable with a middleware, that adds a response header:

context.Response.Headers.Add("X-Robots-Tag", "noindex, nofollow");

The difference is, that this way there will be no

<meta name="robots" content="noindex, nofollow">

tag in the HTML file.

According to my research, there are no meaningful differences. The only difference is: "A robots header can also override a meta tag if they have conflicting values."

Actually I think it's better, since we can only use the meta tag for HTML files.

[Piedone]

For the name, EnvironmentRobots?

I didn't know about the header, but I'd use both to be sure (adding the meta tag is just simply an IResourceManager call from a filter).

[DemeSzabolcs]

Okay, I agree. After some more research I discovered, that using both method is the best, since there are some cases where one is not supported.