When a search system decides which documents match a query, not all matching words should count equally. A document that shares the word “the” with your query tells you almost nothing; a document that shares a rare technical term tells you a lot. In a 1972 paper in the Journal of Documentation (volume 28, pages 11-21), Karen Sparck Jones gave this intuition a statistical footing and proposed weighting terms by what is now called inverse document frequency, or IDF.
Her argument was that term specificity should be treated statistically, as a function of how often a term is used across a collection rather than its meaning. A term that appears in few documents is specific and should be weighted heavily; a term that appears everywhere is unspecific and should be weighted lightly. Combined with how often a term appears within a given document (term frequency), this produces the TF-IDF weighting that dominated information retrieval for decades. Her experiments on test collections showed the scheme improved retrieval performance.
TF-IDF remains a baseline taught in every information retrieval course and is still used in production search, text mining, and as a feature in machine learning. A 2015 survey found the majority of text-based recommender systems in digital libraries relied on it. The paper is one of the most influential in the history of search.
For a general reader, this is the simple idea behind why search engines pay attention to your distinctive words and ignore the common ones, a principle that quietly shapes nearly every search you run.