Optimization (ACM 40990 and ACM 41030)
Description: For Optimization in Machine Learning (ACM 40990, Spring 2025), refer to the Brightspace page. For Optimization Algorithms (ACM 41030, Spring 2025), refer to this page.
Lecture Notes:
- Complete set of typed notes, v1: January 2025
- Complete set of typed notes, v2: February 2025 (Updated Sections 8.1-8.3)
- Week 2, Lecture 1: Lecture notes and video
- Week 3, Lecture 1: Lecture notes and video
- Week 4, Lecture 1: Lecture notes and video
- Week 5, Lecture 1: Lecture notes and video
- Week 6, Lecture 1: Lecture notes and video
- Week 7, Lecture 1: Lecture notes and video
- Week 10, Lecture 2: Lecture notes
Course Documents:
- Introduction to ACM 41030 (January 2025)
- Side note Section 1.3 (Convexity of Polyhedra)
- Week 6, Bonus Assignment
- Exam 1 Guidelines
Examinable results - Part 2:
- Week 8: The projection operator, Section 12.4
- Week 8: First-order optimality conditions in case of a single inequality constraint, Section 13.2
- Week 9: Nothing!
- Week 10: Showing that if the LICQs are satisfied, then the Lagrange Multipliers are unique - Section 15.1, Theorem 15.1
- Week 10: Showing that the tangant cone is inside the set of LFDDs; showing that the tangant cone and the LFDDs are the same when the LICQs are satisfied - Section 15.2, Theorem 15.3
- Week 11: Assuming Farkas's Lemma, prove the necessary conditions (KKT conditions) for a feasible point x_{star} to be a minimizer - Section 16.4
- End of list!
Exercises #1: Line-search methods.
Code repository:
Exercises #2: Newton iteration and the Strong Wolfe Conditions.
- Questions
- Model answers (Correction to Model Answers, Question 2 on 20/02)
- Data file for the 10-dimensional OP
Code repository:
Exercises #3: BFGS revisited and the Trust-Region Method.
Exercises #4: Global Optimization and Simulated Annealing
Code repository:
Exercises #5: Constraints (ACM 41030 only)
Exercises #6: More Constraints (ACM 41030 only)
Foundations of Data Science: Materials for Short Course on 8/12/2023: