Melbourne-based, delivering Australia-wideBusiness websites from $499 · Call 0422 222 267
0422 222 267 Get instant quote
Technical

XML sitemaps and robots.txt explained

By the YAVI Solutions team · Published

What each file is for, what belongs in them and why robots.txt is not a security tool.

Key points

  • Sitemaps list what to index.
  • Robots.txt guides crawling, not access.
  • Use logins for private pages.

Sitemap

A list of the pages you want search engines to know about. It should contain only live, canonical pages that are allowed to be indexed.

Robots.txt

A file at the root of your site that asks crawlers not to fetch certain paths. Well-behaved crawlers follow it; it is a request, not a lock.

Not a security measure

Anything private must sit behind a login. Listing a secret path in robots.txt simply advertises it.

Noindex is different

To keep a page out of search results, use a noindex tag on the page and let crawlers fetch it so they can see the tag.

Submit and check

Add the sitemap in Google Search Console and Bing Webmaster Tools, then watch the reports for pages that are excluded and why.

About our SEO work

Want to talk it through? Call 0422 222 267 or request a callback.

Call 0422 222 267Get instant quote