The robots.txt file is used by website owners to give instructions about their site to web robots. The web robot will read the robots.txt file before visiting a site, to check whether any directories should be excluded from the visit.
In the example file below, the 'User-agent' is specifying ALL robots, and 'Disallow' is instructing them not to visit the /customers directory.
Customer Self Service allows for easy maintenance of the robots.txt file by site Administrators. Simply follow the procedure below.
To View and Update the robots.txt file:
- Login as an Administrator.
- Navigate to 'Settings' --> 'SEO' (/zSearchEngineOptimisationMaintenance.aspx).
- Click the 'Robots.txt' tab.
- The contents of the current robots.txt file will be displayed in the dialogue box.
- Edit the file contents directly on screen, or copy and paste data from another file.
- In the event of an error, click the 'Reload Robots.txt' button to start again.
- Click the 'Submit Changes' button.
The revised file will be sent to Commerce Vision for review, and you will be advised once the new file has been uploaded.
For further information on robots.txt, see http://www.robotstxt.org/robotstxt.html.