Robots.txt is a file that exists on the basis Listing of every Web-site and can be used to instruct search engines like google and yahoo on which directories/files of the website they might crawl and contain of their index. According to Rubright, an even better technique is hand-picking your prospective https://news.rafeeg.ae