Back to Free Tools

Robots.txt Checker


What it is

Robots.txt is a file in text form that instructs bot crawlers to index or not index certain pages. It is also known as the gatekeeper for your entire site. Bot crawlers’ first objective is to find and read the robots.txt file, before accessing your sitemap or any pages or folders.

How to fix it

Create a robots file for your site using a utility, Wordpress plugin, or your CMS's robots.txt creation process. Follow our guides below.

SEOptimer Report Preview

SEOptimer - SEO Audit & Reporting Tool.
Improve Your Website. Win More Customers.

Get a Free Website Audit Instantly