Amazon Mechanical Turk
In plain wordsA website Amazon opened in 2005 that acts as a hiring hall for tiny online jobs, such as labeling images for AI, which registered workers pick up and finish for a few cents each.
A crowdsourcingcrowdsourcing, n.: Getting many people over the internet to each do a small piece of a big job, such as labeling pictures for a few cents apiece, with anyone free to join in. marketplace operated by Amazon that divides work into Human Intelligence Tasks (HITs): a 'requester' posts batches of small, self-contained jobs — transcribing a receipt, judging whether two product listings match, tagging an image — and sets a price per task, 'workers' choose which to complete, and Amazon takes a commission. From the late 2000s it was the leading source of cheap labels for machine-learning and language-processing research, including much of ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom., and a major participant pool for social-science experiments. Its weaknesses shaped later practice: uneven worker quality, spam and, by the 2020s, workers quietly using language models to do text tasks. These pushed requesters toward redundancy, qualification tests and, eventually, managed vendors and expert platforms.
We ran the pilot on Mechanical Turk with a qualification test and three workers per HIT.
Origin
Launched publicly on 2 November 2005. The name recalls the Mechanical Turk, Wolfgang von Kempelen's 18th-century chess 'automaton' that hid a human player inside; Amazon promoted the service as 'artificial artificial intelligence' — human labor behind a software interface — and an MTurk product director credited Jeff Bezos with the concept.
Usage note
A 2018 study of worker earnings found median pay of about two dollars an hour. A 2023 study pointedly titled 'Artificial Artificial Artificial Intelligence' estimated that a third or more of workers on one summarization task used language models.
Related
- crowdsourcingcrowdsourcing, n.: Getting many people over the internet to each do a small piece of a big job, such as labeling pictures for a few cents apiece, with anyone free to join in.
- data labelingdata labeling, n.: The work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them.
- ghost workghost work, n.: The hidden human labor behind AI: people, often poorly paid and far away, who label data, check outputs and handle cases the software cannot, while the product appears fully automatic.
- ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom.
- annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it.
Sources
- Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C. & Bigham, J. P. (2018). A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. CHI 2018. arXiv:1712.05796. arxiv.org/abs/1712.05796
- Veselovsky, V., Horta Ribeiro, M. & West, R. (2023). Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks. arXiv:2306.07899. arxiv.org/abs/2306.07899
Mentioned in
crowdsourcingcrowdsourcing, n.: Getting many people over the internet to each do a small piece of a big job, such as labeling pictures for a few cents apiece, with anyone free to join in., data labelingdata labeling, n.: The work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them., Mechanical TurkMechanical Turk, prop. n.: A famous fake chess-playing machine from 1770: a robed figure that seemed to play chess by itself but hid a human player inside. Its name now stands for 'AI' that is really people working behind the scenes.