Search Engines such as Google, Yahoo and Microsoft Live Search (or Bing) are very useful at getting your site noticed. However, sometimes the search engine 'robot' that finds your site and catalogues your web pages can be a little bit over ambitious. On sites with lots of content that is fetched from a database, the attention of a search engine robot can cause significant load on a database as the robot repeatedly pulls all the content from your database.
To improve performance, we recommend that every site has a 'robots.txt' file which indicates to the search engine where it can search and where it can not. This not only helps with your performance, reducing CPU load, but can also improve your search ranking as the results in the search engine do not contain irrelevant pages.
The following article provides many useful examples on how to optimize for search engines with popular packages such as phpBB and Wordpress.
http://www.askapache.com/seo/seo-with-robotstxt.html
© 
