It seemed like a fanciful idea, but the title caught my eye, “Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design” (Jeffrey Heer, Michael Bostock ACM Human Factors in Computing Systems (CHI), 203–212, 2010).
OK, google “Mechanical Turk” – it is a service from Amazon that matches “workers or Turkers” with “requestors” for small tasks and asked “workers” to assess different visualization designs. If you know this already, you are way ahead of where I was last Tuesday. The authors got 186 respondents to assess some visualizations for a total study cost of $367.77, and the authors were able to compare their study results to published literature to conclude that the method was viable. If “Turkers” can assess visualizations, then they presumably they can respond to other types of questionnaires or surveys about a range of issues.
This piqued my curiosity, and I searched for more information on this issue. One of the first questions of concern to any epidemiologist would be the degree and types of selection biases associated with using crowdsourcing and related approaches to obtain what are essentially survey samples and responses. I was surprised to see a growing body of literature in this arena, the most recent of which is Jennifer Jacquet’s article,on the Scientific American blog (July 7, 2011), “The Pros and Cons of Amazon Mechanical Turk for Scientific Surveys” (aren’t surveys some of what we do?).
I found several other studies characterizing the demographics of samples of respondents on Mechanical Turk:
“The New Demographics of Mechanical Turk”, March 9, 2010 Panos Ipeirotis, NYU School of Business, www.behind-the-enemy-lines.com/2010/03/new-demographics-of-mechanical-turk.html
Ross, Irani, Silberman, Zaldivar and Tomlinson, “Who are the Crowdworkers? Shifting Demographics in Mechanical Turk” CHI 2010, www.ics.uci.edu/~jwross/pubs/RossEtAl-WhoAreTheCrowdworkers-altCHI2010.pdf
Going further, some folks are pursuing the idea of using this methodology for subject recruitment in experimental research: “Using Mechanical Turk as a Subject Recruitment Tool for Experimental Research”, Berinsky, Huber and Lenz, October 7, 2011,
Clearly, some very bright minds are exploring the potential of this mechanism. I would think that we epidemiologists will have a lot to contribute in this arena, and will see great benefits, as well.