Accurate and Efficient Crawling for Relevant Websites Martin Ester, Hans-Peter Kriegel, and Matthias Schubert   View Paper (PDF)   Return to RESEARCH SESSION 10:MANAGING WEB INFORMATION SOURCES Abstract Focused web crawlers have recently emerged as an alternative to the well-established web search engines. While the well-known focused crawlers retrieve relevant web pages, there are various applications which target whole websites instead of single web pages. For example, companies are represented by websites, not by individual web pages. To answer queries targeted at websites, web directories are an established solution. In this paper, we introduce a novel focused website crawler to employ the paradigm of focused crawling for the search of relevant websites. The proposed crawler is based on a two-level architecture and corresponding crawl strategies with an explicit concept of websites. The external crawler views the web as a graph of linked websites, selects the websites to be examined next and invokes internal crawlers. Each internal crawler views the web pages of a single given website and performs focused (page) crawling within that website. Our experimental evaluation demonstrates that the proposed focused website crawler clearly outperforms previous methods of focused crawling which were adapted to retrieve websites instead of single web pages. ©2005 Association for Computing Machinery