Introduction to Statistical Learning Theory

Class Times: Fridays 14:00 - 15:30 (Sometimes Wednesdays 11:30 - 13:00 see syllabus).
Location: Ground floor lecture theater, Gatsby Computational Neuroscience Unit, Sainsbury Wellcome Centre, 25 Howland Street.
Instructor: Carlo Ciliberto
TAs: Stephen Pasteris
Office Hours: Thursdays 14:00 - 15:00. 3rd floor Hub, 66 Gower street
Email Contact : cciliber (a) gmail.com, stephen.pasteris (a) gmail.com

This course represents half of Advanced Topics in Machine Learning (aka COMP GI13 / COMP M050) from the UCL CS MSc on Machine Learning.

The other half is Reproducing kernel Hilbert spaces in Machine Learning (Taught by Prof. Arthur Gretton).

Course Description

Statistical Learning Theory (SLT) studies the problem of learning from empirical observations (data) to predict and/or understand the behavior of an unknown phenomenon (e.g. the dynamics of the stock market or the activations patterns in the human brain). SLT provides a mathematical framework within which it is possible rigorously address questions such as “How to design a learning algorithm”, “what does it mean for an algorithm to ‘solve’ a learning problem” or “How to compare two learning algorithms”.

The goal of this course is to introduce students to the ideas behind most well-established learning algorithms and provide fundamental insights on how to use them in practice or to design new ones.

Prerequisites

Linear Algebra, Probability Theory, Calculus.

Grading

Final grades will depend on one project assignment (50%) and a final exam (50%).

Syllabus

Class	Date	Topic
1	Fri Oct 06	Course Overview
2	Fri Oct 13	Overfitting and Regularization
3	Wed Oct 18	Tikhonov Regularization
4	Fri Oct 20	Computational Regularization I: Iterative Regularization
5	Fri Nov 03	Computational Regularization II: Sampling
6	Fri Nov 10	Generalization Error and Stability
7	Fri Nov 17	Model Selection
8	Fri Nov 24	Notes on the Approximation Error
9	TBA	Going further with Regularization
10	TBA	Structured Prediction

Reading List

There is no required text for the course. Below is a number of useful references:

O. Bousquet, S. Boucheron and G. Lugosi Introduction to Statistical Learning Theory (Tutorial).
T. Poggio and L. Rosasco course slides and videos.
P. Liang course notes.
N. Cristianini and J. Shawe-Taylor. Kernel Methods for Pattern Analysis . Cambridge University Press, 2004.
I. Steinwart and A. Christmann. Support Vector Machines Springer, 2008.
S. Shalev-Shwartz and S. Ben-David Understanding Machine Learning: From Theory to Algorithms (Online Book). Cambridge University Press , 2014.