Netflix Prize
In plain wordsA 2006–2009 contest in which Netflix offered a million dollars to anyone who could predict customers' movie ratings 10 percent better than its own system. It popularized machine-learning competitions and model blending.
An open competition run by Netflix from October 2006 to September 2009, offering US$1 million to the first team to improve the root-mean-square error of its Cinematch recommendation algorithmrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on. by 10 percent on held-out ratings. Contestants received about 100 million ratings given by some 480,000 anonymized customers to 17,770 films. The contest demonstrated the strength of latent-factor matrix factorizationmatrix factorization, n.: Breaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together. for collaborative filteringcollaborative filtering, n.: Recommending things based on what people with similar tastes liked, without needing to know anything about the things themselves, as in 'people who bought this also bought…'., the value of modeling how tastes change over time, and above all the gains from blending hundreds of diverse models into an ensembleensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.. The prize went to BellKor's Pragmatic Chaos, a merger of teams from AT&T Labs, Austria and Canada, which tied the rival team The Ensemble on the test score but had submitted 20 minutes earlier. Netflix never deployed the winning blend in full, judging the engineering cost too high.
The Netflix Prize taught a generation of data scientists that the winning model is usually a blend.
Origin
Announced by Netflix on 2 October 2006 and awarded on 21 September 2009. Its public-leaderboard format foreshadowed competition platforms such as Kaggle, founded in 2010.
Usage note
A planned sequel was cancelled in 2010 after Arvind Narayanan and Vitaly Shmatikov showed that the anonymized ratings could be linked back to individuals, prompting a privacy lawsuit and a US Federal Trade Commission inquiry — an early landmark for data privacy in machine learning.
Related
- collaborative filteringcollaborative filtering, n.: Recommending things based on what people with similar tastes liked, without needing to know anything about the things themselves, as in 'people who bought this also bought…'.
- matrix factorizationmatrix factorization, n.: Breaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together.
- recommender systemrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on.
- ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
Sources
- Bennett, J. & Lanning, S. (2007). The Netflix Prize. Proceedings of KDD Cup and Workshop 2007.
- Narayanan, A. & Shmatikov, V. (2008). Robust De-anonymization of Large Sparse Datasets. IEEE Symposium on Security and Privacy 2008.
Mentioned in
leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models., collaborative filteringcollaborative filtering, n.: Recommending things based on what people with similar tastes liked, without needing to know anything about the things themselves, as in 'people who bought this also bought…'., ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model., matrix factorizationmatrix factorization, n.: Breaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together., Arvind NarayananArvind Narayanan, prop. n.: An Indian-American computer scientist at Princeton who studies AI's real-world effects and argues that AI is a 'normal technology' whose impact will unfold gradually., restricted Boltzmann machinerestricted Boltzmann machine, n.: A two-layer network, one layer for the data and one hidden, that learns the patterns in data by trying to generate it. Stacks of them helped start the deep-learning revival around 2006.