A file placed in the root directory of a website to instruct search engine bots on which pages to crawl or exclude from crawling.