How to create and Generate Robots.txt
| About Robots.Txt |
 |
 |
 |
|
Some pages and directories on websites do not require indexing. These include files like the admin file which is irrelevant to the search engine. If you want you can disable the search engine from indexing the pages. This is carried out by the Robots.txt is a file. It is a file created to command search engines to exclude the pages or directories from being crawled. By creating a robots.txt file you can make the process of website indexation, easier. Firstly, you need to create a file called robots.txt. It must be maintained in the root directory for your domain. This means that if your site is "www.yourdomain.com" the robots.txt file must be located at ROOT directory of your website! If kept anywhere else, it will be ineffective. How to create a robots.txt file: * All pages are excluded: User-agent: * Allow: / * No Page is included: User Agent: * Disallow: * One page is excluded: User Agent: * Disallow: /directory/path/page.html * All pages from a directory and its subfolders are excluded: User Agent: * Disallow: /directory/ Where to put robots.txt file: In the root directory of your website!
|
|||
| Last Updated on Tuesday, 01 September 2009 20:00 |
