The Machine Learning Forum

Information Theory and Machine Learning

There is a fundamental conceptual affinity between information theory and machine learning. Both are concerned with exploiting regularities and patterns in data sources to accomplish certain tasks in the presence of uncertainty and noise. In the case of information theory, the tasks of interest involve reliable representation, transmission and storage of information; in the case of learning, we are interested in making accurate predictions about the future on the basis of previously seen data.

Active Learning

The term active learning applies to a wide range of situations in which a learner is able to exert some control over its source of data. For instance, when fitting a regression function, the learner may itself supply a set of data points at which to measure response values, in the hope of reducing the variance of its estimate. Such problems have been studied for many decades under the rubric of experimental design [C72,F72]. More recently, there has been substantial interest within the machine learning community in the specific task of actively learning binary classifiers. This task presents several fundamental statistical and algorithmic challenges, and an understanding of its mathematical underpinnings is only gradually emerging.