Google have actually followed a similar model for like ten years now with like 10,000 temp workers following this 172 manual to essentially as the basis to train a lot of their ranking models.
Gmail's spam filter is basically community trained by people clicking spam/not spam (at least, that was what they said many years ago, it might have silently changed). What's different about search?
Wikipedia shows it can work at scale.