Untyped, a UK web development consultancy, writes about Scheme and Racket, A/B testing, bandit algorithms, and Dynamo.
HN mostly links it for the post on bandit algorithms as an alternative to A/B testing, and mentions it as a Racket consultancy.
This makes sense if you are doing a one-time split test, but not if you are doing ongoing tests akin to the scheme described in the article I link to below (which comes up on HN from time to time). The problem being that in these cases you never really stop testing: instead, you always assign some random proportion of incoming traffic to your various buckets based on your current statistical certainty.
http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/
This not only has the advantage mentioned where you limit your regret for making a bad decision on too little evidence, but with some modifications to the algorithm (such as discounting old evidence from your inference) you can also deal with situations where the world in which you are running the different variations is itself changing over time, or where different types of users coming from different browsers may have different behaviors.
From the perspective of someone searching for your site, however, the results might be nearly identical: even if you have massive changes to the phrasing (having versions written in various regional dialects of English, for example), the resulting page will still largely be identical, possibly sentence-for-sentence (though not word-for-word). In this case, you will still want to get one of those variations indexed, and it might not matter which one.
As described, of course, this is both "cloaking" and not: the site is not treating GoogleBot differently…
My approach was to read the Dynamo paper. It's presents the goals of Dynamo and the technology from which it is constructed in a very compact form. From there you have a basis for understanding the engineering tradeoffs made by the other NoSQL databases. Then you can actually make decisions based on what fits your use case, rather than just choosing Mongo because it's popular. (Don't get me wrong, Mongo is great in its niche.)
Reading the Google BigTable paper would also be a good idea, as it represents another major strand of work.
My blog post on Dynamo: http://untyped.com/untyping/2011/01/21/all-about-amazons-dynamo/
This description of content optimization using bandit algorithms sounds like an even better approach: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/
That company has already made a web app and service to optimize content using that approach, Myna, at http://www.mynaweb.com/. A simulated experiment showed their approach to be better than A/B testing: http://www.mynaweb.com/blog/2011/09/13/myna-vs-ab.html. Though Myna's website doesn't say whether it is currently free or not, or what its pricing will be when it goes out of beta.
Noel Welsh at untyped has been working on a cool implementation of this adaptive design.
Check it out http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/
And the HN discussion: http://news.ycombinator.com/item?id=3867380
Not quite what you're asking about, but an interesting related innovation:
http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/
You know that "Racket" is the former "PLT Scheme", right? Some boutique consultants use it, notably Untyped: http://untyped.com/