Optimization (ACM 40990 and ACM 41030)
Description: In Spring 2025, I will be teaching the first seven weeks of Optimization in Machine Learning (ACM 40990) and all of Optimization Algorithms (ACM 41030). For the first seven weeks, these two modules share exactly the same content. This content will be posted here. These are advanced modules which introduce students to the theory behind optimization algorithms - looking at the setup of the algorithms, their convergence, and the practical implementation of same. The method of instruction will be lectures about theory, and practical coding classes.
Lecture Notes:
Course Documents:
- Introduction to ACM 40990 (January 2025)
- Introduction to ACM 41030 (January 2025)
- Side note on Theorem 2.10
- Side note Section 1.3 (Convexity of Polyhedra)
Handwritten Lecture Notes:
Exercises #1: Line-search methods.
Code repository:
Exercises #2: Newton iteration and the Strong Wolfe Conditions.
Code repository:
Exercises #3: BFGS revisited and the Trust-Region Method.
Exercises #4: Global Optimization and Simulated Annealing
Code repository:
Exercises #5: Constraints (ACM 41030 only)
Exercises #6: More Constraints (ACM 41030 only)