Notice: Trying to access array offset on value of type bool in /home/admin/domains/robotstxt.info/public_html/maincore.php on line 205
Discussion Forum - Robots.txt
Navigation
Users Online Now
Guests Online 1

Members Online 0


Total Members: 2
Newest Member: robots-txt
Forum Categories
thumbnail
Robots.txt
A robots.txt file tells search engine robots what they may and may not access from your site. A robots.txt file does not enforces access to your site. It is merely a suggestion to search engine robots that respect these robots.txt rules. These search engine robots, also called bots, that crawl the web are fully automated. Before they access pages of your site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For these bad boys you can setup a bot trap that is making use of robots.txt. Anything that needs to be confidential is best to password protected.
thumbnail
Bots (good and bad) that crawled your site
Web Robots (also known as Web Crawlers, or Spiders), are automated programs that rummage the Web. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, ransack your site and many other uses.
Sign In
Not a member yet? Click here to register.