Robots.txt is a file that exists on the root directory of each Site and can be employed to instruct engines like google on which directories/files of the web site they are able to crawl and incorporate in their index. Should you be expecting them for being 100% on customer function https://news.rafeeg.ae