Panos Ipeirotis, a researcher, writes about crowdsourcing, Mechanical Turk, online advertising, and data experiments.
HN commenters cite it for its follow-up on ad viewability, its crowdsourcing and worker reputation work, and a post on a surprise bandwidth bill.
Humans being the error-prone creatures we are, I wonder how it could work better to a longer word-based code, but allow for certain kinds of errors: for example, 1) word order doesn't matter, 2) it tries to correct a word not in the built-in word list to the "closest" word in it, 3) maybe you only need, say, seven of the eight words to be right to get in (then it might remind you of the real code). You can do a little calculation to see how many words you need to get to the desired number of bits then.
Pretty different from passwords as we know them (#3 means you can't simply store a bcrypt/scrypt'd code--you could build something complicated tricks to square "allow one word wrong" with not storing plain PWs: each word is 10 bits and what you tell the user is really their secret code + a parity/ECC word, and correction happens before a convnetional password check--that rabbit hole goes deep, though.) But if all this gets average folks remembering more entropy than before, that makes it kind of interesting.
On the study (which I admit just skimming), survey in 2010 suggested about half of Mechanical Turk users were located in the US (http://www.behind-the-enemy-lines.com/2010/03/new-demographics-of-mechanical-turk.html)--folks who learned English at school would probably do better at memorizing a password in their native language. (I studied French many years in school, but I doubt I'd memorize a set of random French words as easily as English words.)
Turkit, Javascript code for achieving consensus among AMT workers, especially with multiple stages. See the Find-Fix-Verify pattern: http://www.behind-the-enemy-lines.com/2011/04/want-to-improve-sales-fix-grammar-and.html
By doing multiple stages of crowdsourcing, you can do complicated tasks with high quality, like: Improve the grammar of these reviews. Zappos did this and saw a lift in conversion.
Crowdflower, which is good for achieving high-quality one-shot annotation, but not so much for pipelines of annotations.
MobileWorks, which is good for achieving high-quality one-shot annotation, and says in the docs that it has pipelines. I haven't figured this feature yet out.
CrowdControl, which supposedly solves every problem, but is priced as an enterprise solution.
If you want to build something cool, implement pipelines of work. i.e. build a crowd-programming layer that has subroutines. Look at what CrowdControl says they are doing.
[edit: If you also want to build something cool, implement a reputation system. Don't just assign workers a single number. Figure out what kind of tasks the workers are good as, and do a per-task reputation system. For bonus points, solve this correctly, by dynamically gauging the skillset and difficulty for each task, rather than simply grouping tasks into N clusters, where N is low.]
Email me if you want to discuss. I've been thinking about this for a while.
Apart from the small sample size, if they collected data by mechnical turk then I wonder what sort of biases that introduces? In the paper you linked, Frances and Mialon state:
"Samples of mTurk workers have been found to be more representative of
the US population than in-person convenience samples, standard internet samples, and typical
college samples"
I am somewhat sceptical of this, and there seems to be some evidence to back-up my scepticism [1][2][3]. In particular, [2] states:
"In sum, the MTurk sample is younger, more male, poorer, and more highly educated than Americans generally. This matches the image of who you might think would be online doing computer tasks for a small amount of money."
Which are some of the factors that the marriage study itself seeks to examine. This looks like lazy data collection methodology to me.
[1] http://www.culturalcognition.net/blog/2013/7/10/fooled-twice-shame-on-who-problems-with-mechanical-turk-stud.html
[2] http://themonkeycage.org/2012/12/19/how-representative-are-amazon-mechanical-turk-workers/
[3] http://www.behind-the-enemy-lines.com/2010/03/new-demographics-of-mechanical-turk.html
Amazon's not amazing at announcing useful timelines for their products/services. There are presumably a bunch of internal factors at play, but spending limits aren't the only thing that's been "in the works" for ages. If you're holding out for something that may or may not be coming (eg DevPay in your region, hard spending limits, long-term-queues, scheduling, ability to cloudfront s3 'subfolders' etc) it's frustrating, but on the other hand, they do tend to release well thought out services that make sense.
If it's any consolation, I have heard of individuals getting huge bills and having them cleared by Amazon when it became apparent that they hadn't willingly racked up that many fees (like the guy who included his S3 images in a google spreadsheet causing several TB of requests - http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self-attacked.html ).
I imagine that must be their thinking. Clearly a lack of technical resources it's not. Moreover, Amazon has already shown [1] good will when negligence led to an unexpectedly high bill.
I'm surprised (though empirically without much reason) not to hear stories from NewHotness et al about "how AWS burned five times our monthly budget in one day of frenzy". Even if those new visitors may very well be welcome for the business, I imagine there are many folks who would like some control (say, to double the limit), maybe at the risk of leaving the site offline for a few minutes.
[1] http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self-attacked.html
"TL;DR: There are about 100K-200K unique workers on Amazon. On average, there are 2K-5K workers active on Amazon at any given time, which is equivalent to having 10K-25K full-time employees. On average, 50% of the worker population changes within 12-18 months. Workers exhibit widely different patterns of activity, with most workers being active only occasionally, and few workers being very active. Combining our results with the results from Hara et al, we see that MTurk has a yearly transaction volume of a few hundreds of millions of dollars."
https://www.behind-the-enemy-lines.com/2018/01/how-many-mechanical-turk-workers-are.html