The role of variance in term weighting for probabilistic information retrieval Warren R. Greiff, William T. Morgan, and Jay M. Ponte   View Paper (PDF)   Return to Information retrieval models Abstract In probabilistic approaches to information retrieval, the occurrence of a query term in a document contributes to the probability that the document will be judged relevant. It is typically assumed that the weight assigned to a query term should be based on the expected value of that contribution. In this paper we show that the degree to which observable document features such as term frequencies are expected to vary is also important. By means of stochastic simulation, we show that increased variance results in degraded retrieval performance. We further show that by decreasing term weights in the presence of variance, this degradation can be reduced. Hence, probabilistic models of information retrieval must take into account not only the expected value of a query term's contribution but also the variance of document features. DiSC'03 © 2003 Association for Computing Machinery