Create a valid robots.txt file to instruct search engine robots which pages or files they can and cannot crawl.