Check which page URLs a crawler may access using a public robots.txt URL or pasted rules, with matching rules explained.
Useful to know
Does robots.txt prevent indexing or protect private files?
No. These rules request crawler behaviour. A blocked URL can still be known to a search engine, and robots.txt is not access control.
Where are my files and inputs processed?
Pasted text and opened files are processed on your device. If you request a public sitemap or robots.txt URL, OnlineTools Kenya receives that URL and downloads it through its server. Sitemap indexes also download their child URLs. The remote website sees server requests. This feature does not save document contents; hosting and security logs may retain request metadata. Passing an extracted URL list to another tool uses temporary browser-tab storage, expires after ten minutes, and is consumed when opened.
Rate this tool
Tell others how well Robots.txt Rule Tester worked for you. No account needed.