- BotKnowledge.com
 - Directory of intelligent software agents, knowbots, and bots. Includes FAQ and newsletter. - BotSpot
- Directory of bots and bot resources for Windows and Mac.
- Collegebot
- Dedicated to indexing and searching education and academic related pages.
- Google Sitemaps
- Collaborative crawling system that enables webmasters to communicate directly with Google to keep the search engine informed of all their web pages, and when changes are made to these pages.
- Googlebot
- Web-crawling robot that collects documents from the web to build a searchable index for the Google search engine.
- Grub
- Open source, distributed Internet crawler.
- ht://Dig
- Indexing and searching system for a small domain or intranet.
- MSNBot
- Web-crawling robot developed by MSN Search that collects documents from the web to build a searchable index.
- Peregrinator: A Web-Indexing Robot
- A robot for traversing and indexing sections of the Web.
- RoboGen
- Programme to generate a robot exclusion file, robots.txt, for your web site, which controls the files appearing in search engines.
- Robotcop
- An open source module for webservers which helps webmasters prevent spiders from accessing parts of their sites they have marked off limits.
- Searchbots
- Offers bots with different search programmes, or allows users to customise their own.
- SG-Scout Home Page
- Developed for the Xerox Palo Alto Research Centre as part of a project involving the development of a new form of directed WWW browser.
- Spider Cloak
- Offers search engine cloaking script that hides the source code to prevent theft and generates optimized marketing pages for submission.
- SpiderHunter.com
- Demonstrates how to write cloaking scripts and track spiders.
- Web Robots Pages, The
- Lists Web Robot FAQs, databases, and mailing lists.
- Webglimpse
- Search engine software that consists of Glimpse, an engine for indexing and searching text files on a Unix system and Webglimpse, the spider and manager for the files to be searched.
- WWWMM Robot (W3M2)
- A wanderer robot written for several experimental purposes, but more especially to study throughput/latency on paths in the net.
|
