Insegnamento a.a. 2023-2024

The main topics of the course are:

- probability and combinatorics review

- notions of entropy and mutual information

- entropy and data compression

- applications to predictions

- channel capacity and error-correcting codes

- continuous analogs of discrete channels

- applications to statistics

- the maximum entropy principle

- applications to finance: portfolio theory

- understand how entropy models optimal data compression and how data compression is related to modeling and predictions

- understand how to model the errors introduced by a noisy channel and how the capacity of a channel limits the rate of information transmission

- describe and analyze the main families of algebraic error-correcting codes

- connect information theory concepts to applications in statistics, finance, and machine learning

- define probabilistic models and compute information-theoretic quantities related to such models

- choose the error-correcting scheme most appropriate to a particular applications

- apply information-theoretic techniques to compute tail bounds of discrete probabilistic models

- apply information-theoretic techniques to machine learning applications

- use entropy and KL divergence in applications, perform calculations and estimate asymptotic bounds

- apply information-theoretic techniques to financial models

• Lezioni frontali
• Esercitazioni (esercizi, banche dati, software etc.)
• Lavori/Assignment individuali

Students will practice their modeling and calculation skills with in-class exercises. Those will take place a few times during the semester during class. Students will aply the theoretical knowledge from previous lectures to solve the exercises individually or in small groups. The correct solution will then be presented by the instructor. The goal of this activity is for students to practice the applications of the theoretical knowledge imparted in class and to self-evaluate their progress in the class.

Students will also be given take-home individual assignments, which will similarly be questions that test the ability of the students to apply the theoretical knowledge of the course. These assignments will be graded. The goal of this activity is for students to practice their problem-solving abilities on more challenging questions than the ones given as in-class exercises, and it will provide an ongoing assessment of the progress of each student in their understanding of the subject. 

Students will be given individual take-home assignments to test their progress during the course, there will be a written midterm and a written general exam. The final grade will be determined by assignments (5%), midterm (40%) and general (55%)

Cover and Thomas, "Elements of Information Theory", 2nd edition, Wiley