CFEM Seminar - Title:"Assumption free forecasting and anomaly detection: From theory to algorithms" | Michael Rabadi | Spotify
An effective anomaly detector can save a company millions of dollars every month by immediately localizing problems and alerting relevant engineers. Spotify uses anomaly detection for everything from monitoring revenue health to detecting bugs after releasing updates. In some cases it can also be used to detect fraudulent activity. While there are many specialized anomaly detectors, a major open question is how to design a general anomaly detection system that can work out-of-the-box on thousands of time series without making any distributional assumptions. In this talk, we will discuss the foundations of anomaly detection and how it should be decoupled into a forecasting step, followed by a thresholding step. We will go into new discoveries from statistical learning theory and explain how this theory has been used to design state-of-the-art algorithms. We then discuss how we can develop simple, scalable algorithms that can solve the general anomaly detection problem and provide some experimental results.
Michael Rabadi is a Machine Learning Researcher and Engineer at Spotify. He develops novel machine learning algorithms to address many of the unsolved challenges that current machine learning algorithms are ill-equipped to solve. He applies these algorithms to many projects within Spotify, though he focuses on generating revenue.
Before joining Spotify, Rabadi studied Brain Machine Interfaces at NYU. BMIs are brain controlled systems that give paraplegic patients the ability to consciously control robotic arms. There he statistically analyzed the time series algorithms common to the field and discovered a number of statistical violations that explained the limitations of modern BMIs. He used his analysis to create a novel BMI algorithm that had much better empirical results. One notable accomplishment of this work was a general formalization of the co-adaptive learning problem that allowed him to prove feasibility of the problem and derive a statistical guarantee for evaluating such algorithms.