What is robots.txt?

  • 0 replies

Offline mechanic

  • Engineer
  • *****
  • 3420
  • Gender: Male
  • Engineer
    • View Profile
    • Engineering Forum
What is robots.txt?
« on: April 27, 2019, 09:41:26 AM »
What is robots.txt?

It’s a way of telling the Search Engine Bots about the web pages on your website that you don’t want to open or visit by others.
If in case you want to block all search engines robots for crawling your website, then just put the
Code: [Select]
follow code:
User-agent: *
Disallow: /
If you want to block Google for crawling your website, just put the follow code:
Code: [Select]
User-agent: Googlebot
It’s important to install Robots.txt file very carefully, or else errors might occur while displaying the website.