In order to avoid unwanted written content inside the look for indexes, webmasters can instruct spiders to not crawl specified files or directories from the common robots.txt file in the foundation directory with the area. Furthermore, a site may be explicitly excluded from the search engine's database through the use https://news.rafeeg.ae