Calculus and Optimization

Machine learning reduces to finding parameters that minimize a loss function. This chapter covers the mathematics of minimization: multivariable calculus, convex and non-convex optimization, and constrained optimization.

• Multivariable Calculus -- Partial derivatives, gradient, directional derivative, Taylor expansion
• Convex Optimization -- Convex sets and functions, convergence guarantees, strong convexity
• Non-Convex Optimization -- Saddle points, loss landscape geometry, learning rate schedules
• Constrained Optimization -- Lagrange multipliers, KKT conditions, duality