The Learning Problem
The supervised learning problem statement.
The Supervised (Inductive) Learning Problem Statement
Let us start with a classic formal definition of the supervised learning problem.
Vapnik’s formulation of the learning problem (enhanced with notation from the Deep Learning book)
The description below is taken from Vadimir Vapnik’s classic book Statistical Learing Theory, albeit with some enhancements on terminology to make it more in line with our needs.
The generator is a source of situations that determines the environment in which the target function (he calls it supervisor) and the learning algorithm act. Here we consider the simplest environment: the data generator generates the vectors
The vector
The learning algorithm observes data that is drawn randomly and independently from the joint distribution function
During what is called training, the learning algorithm constructs some operator which will be used for prediction of the supervisor’s answer
The ability to optimally predict, according to a criterion, when observing data that we have never seen before, the test set, is called generalization. Note that in the literature supervised learning is also called inductive learning. Induction is reasoning from observed training cases to general rules (e.g. the final hypothesis function), which are then applied to the test cases.
In summary, to learn we need three components:
- Data that may be stored (batch) or streamed (online).
- An algorithm that optimizes an objective (or loss) function
- A hypothesis set
We now have everything in place to start addressing learning tasks such as regression and classification.