Robots.txt is a very important file if you want to have a good ranking on search engines, It defines which paths are off limits for spiders to visit. What is Robots.txt :-- When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored and this is always located in your Web server's root directory..... How to Create Robots.txt :-- The format for the robots.txt file is special. It consists of records. Each record consists of two fields : a User-agent line and one or more Disallow : lines. The format is :-- User-agent: * Disallow: This one keeps all robots out. User-agent: * Disallow: /