BibFinder/StatMiner: Effectively Mining and Using Coverage and Overlap Statistics in Data Integration Zaiqing Nie, Subbarao Kambhampati, and Thomas Hernandez   View Paper (PDF)   Return to Distributed Information Management (Session D2) Abstract Recent work in data integration has shown the im- portance of statistical information about the cover- age and overlap of sources for efficient query pro- cessing. Despite this recognition there are no ef- fective approaches for learning the needed statis- tics. In this paper we present StatMiner, a sys- tem for estimating the coverage and overlap statis- tics while keeping the needed statistics tightly un- der control. StatMiner uses a hierarchical classi- fication of the queries, and threshold based vari- ants of familiar data mining techniques to dynam- ically decide the level of resolution at which to learn the statistics. We will demonstrate the ma- jor functionalities of StatMiner and the effective- ness of the learned statistics in BibFinder, a pub- licly available computer science bibliography me- diator we developed. The sources that BibFinder integrates are autonomous and can have uncon- trolled coverage and overlap. An important fo- cus in BibFinder was thus to mine coverage and overlap statistics about these sources and to ex- ploit them to improve query processing. ©2004 Association for Computing Machinery