How does Instart tell crawlers to skip Ad Recovery pages?
In addition to taking a number of proactive steps to ensure that Ad Recovery technology is never activated for crawlers, Instart also ensures that an Ad Recovery page won’t be indexed, in the unlikely event a crawler directly accesses it. Instart uses multiple common methods to make sure ad-recovered pages are excluded from crawling and indexing:
- We send the HTTP response header x-robots-tag: noindex, no follow with any ad-recovered pages which directs crawlers to exclude them.
- We also exclude the /g00/* path in your
robots.txtfile, which provides an extra layer of protection, indicating that recovered pages should not be crawled.
- Finally, we take the additional step of putting the original non-g00 URL in the canonical tag. This is a largely unnecessary precaution, as the no-index directives mean that Googlebot would never get far enough to pick this up.