Optimization (ACM 40990 and ACM 41030)
Description: In Spring 2024, I am teaching the first seven weeks of Optimization in Machine Learning (ACM 40990) and all of Optimization Algorithms (ACM 41030). For the first seven weeks, these two modules share exactly the same content. This content will be posted here. These are advanced modules which introduce students to the theory behind optimization algorithms - looking at the setup of the algorithms, their convergence, and the practical implementation of same. The method of instruction will be lectures about theory, and practical coding classes.
 
Lecture Notes:
Course Documents:
- Introduction to ACM 40990 (January 2024)
- Introduction to ACM 41030 (January 2024)
- Side note on Theorem 2.10 (6th February 2024)
- Bonus Assignment (Due date: March 26th 2024)
- Topics for midterm exam (Exam date: March 26th 2024)
- Side note Section 1.3 (Convexity of Polyhedra)
- Topics for second exam (Exam date: May 15th 2024, ACM 41030 only)
Handwritten Lecture Notes:
- Week 1, Lecture 1
- Week 1, Lectures 2-3
- Week 2, Lecture 1: ONLINE LECTURE: notes and video
- Week 2, Lectures 2-3
- Week 3, Lecture 1: ONLINE LECTURE: notes and video
- Week 3, Lectures 2-3
- Week 4, Lecture 1: ONLINE LECTURE: notes and video
- Week 4, Lecture 2
- Week 5, Lecture 1: ONLINE LECTURE: notes and video
- Week 5, Lectures 2-3
- Week 6, Lecture 1: ONLINE LECTURE: notes and video
- Week 6, Lectures 2-3
- Week 7, Lecture 1: ONLINE LECTURE: notes and video
- Week 6, Lectures 2-3
- Week 7, Lecture 1: ONLINE LECTURE: notes and video
- Week 7, Lectures 2-3
Exercises #1: Line-search methods.
Code repository:
Exercises #2: Newton iteration and the Strong Wolfe Conditions.
Note: The model answers were corrected after the lecture on 22/02. The corrections are shown in red.
Code repository:
Exercises #3: BFGS revisited and the Trust-Region Method.
Exercises #4: Global Optimization and Simulated Annealing
Code repository:
Exercises #5: Constraints (ACM 41030 only)
Note: The model answers were corrected after the lecture on 09/04. The corrections are shown in red.
Exercises #6: More Constraints (ACM 41030 only)
Foundations of Data Science: Materials for Short Course on 8/12/2023: