Product and Feature Updates about Kinja

Updated Article Exclusions (robots.txt and NOINDEX) - Submitted URL blocked by robots.txt & Indexed, though blocked by robots.txt

We learned that we need to No Index and request re-crawl BEFORE excluding from robots.txt and removing a page from the sitemap.

Because we have historically done it all at once or maybe just one and not the other, we’ve caused these issues:

  • Indexed, though blocked by robots.txt > if it’s blocked by robots.txt, we should NOINDEX and we’re not
  • Submitted URL blocked by robots.txt > if it’s blocked by robots.txt, we should NOINDEX and we’re not
Advertisement

We updated excluded articles so that they are removed from the sitemap and have the NOINDEX robot tag. All errors have been submitted for validation.

Share This Story

About the author

Caiti Vincel

Director Of Editorial Product. #DiscoTeam4Lyfe