A central challenge in machine studying is methods to prepare fashions on delicate person knowledge. Trade has broadly adopted a easy algorithm: Stochastic Gradient Descent with noise (a.ok.a. Stochastic Gradient Langevin Dynamics). Nonetheless, foundational theoretical questions on this algorithm’s privateness loss stay open — even within the seemingly easy setting of clean convex losses over a bounded area. Our foremost consequence resolves these questions: for a wide variety of parameters, we characterize the differential privateness as much as a relentless issue. This consequence reveals that each one earlier analyses for this setting have the mistaken qualitative conduct. Particularly, whereas earlier privateness analyses improve advert infinitum within the variety of iterations, we present that after a small burn-in interval, operating SGD longer leaks no additional privateness.
Our evaluation departs fully from earlier approaches based mostly on quick mixing, as a substitute utilizing strategies based mostly on optimum transport (particularly, Privateness Amplification by Iteration) and the Sampled Gaussian Mechanism (particularly, Privateness Amplification by Sampling). Our strategies readily prolong to different settings, e.g., strongly convex losses, non-uniform stepsizes, arbitrary batch sizes, and random or cyclic selection of batches.
*=Equal Contributors
