Search engines will look in your root
domain for a special file named "robots.txt"
(http://www.mydomain.com/robots.txt). The file tells
the robot (spider) which files it may spider (download).
This system is called, The Robots Exclusion Standard.
The format for the robots.txt file is special. It
consists of records. Each record consists of two fields
: a User-Agent line and one or more Disallow: lines.
The format is:
":"
The robots.txt file should be created in Unix line
ender mode! Most good text editors will have a Unix
mode or your FTP client *should* do the conversion
for you. Do not attempt to use an HTML editor that
does not specifically have a text mode to create a
robots.txt file.
For more Information to create a Robots.txt please
go here
|
