XML sitemaps and robots.txt explained
By the YAVI Solutions team · Published
What each file is for, what belongs in them and why robots.txt is not a security tool.
Key points
- Sitemaps list what to index.
- Robots.txt guides crawling, not access.
- Use logins for private pages.
Sitemap
A list of the pages you want search engines to know about. It should contain only live, canonical pages that are allowed to be indexed.
Robots.txt
A file at the root of your site that asks crawlers not to fetch certain paths. Well-behaved crawlers follow it; it is a request, not a lock.
Not a security measure
Anything private must sit behind a login. Listing a secret path in robots.txt simply advertises it.
Noindex is different
To keep a page out of search results, use a noindex tag on the page and let crawlers fetch it so they can see the tag.
Submit and check
Add the sitemap in Google Search Console and Bing Webmaster Tools, then watch the reports for pages that are excluded and why.
Want to talk it through? Call 0422 222 267 or request a callback.