Optimizely Blog

Thanks for your comment. This is Darwish, the Product Manager working on Stats Engine. You are correct "classic statistics" is the method we used in the past. It also what is most commonly used in industry (the main reason we started with this method). This was not an easy project for us to take on, but after talking to customers and looking at our historical experiment data, it was clear how important this problem was to solve, and thats why we spent a lot of resources on fixing this. Just for those following along on this comment, its not that "classic statistics" on their own that are incorrect, but rather the misuse of these statistics that can be costly. When used "incorrectly" (not using a sample size calculator, running many goals and variations at a time etc..), you can meaningfully increase your chance of making a bad business decision or commit yourself to unnecessarily long sample sizes. Using statistics correctly is an industry-wide problem that many have tried to solve with education (i.e. give statistics crash courses). We hope that our solution shows how important we think it is that statistics drive day-to-day decisions in organizations and that there are different ways (change the math, not the customer) to get customers to this point. Many companies have data science teams and in-house statisticians that are very aware of these problems, but many don't and thats really where we wanted to help out. You can read more about why we thought this was a serious…

[Disclaimer: I previously worked for Optimizely as predictive analytics PM - but no longer work there, and don't speak for the company.]

Optimizely has a bandit based 'traffic auto-allocation' feature in production on select enterprise plans [1]; bandits are excellent in a wide range of situations, and have many advantages, but like anything, have design parameters and there are some caveats you have to be aware of to make sure you are using them effectively.

On Frequentist and Bayesian:
Optimizely's stats engine combines elements of both Frequentist and Bayesian statistics. They have a blog that tries to touch on this issue [2]
But this is subtle stuff - and there are a lot of trade-offs, and different perspectives; look at the Bayesian/frequentist debate which has been going on for decades among statisticians.

But, FWIW, I definitely saw Optimizely as an organisation make a big investment to produce a stats engine which had the right trade-offs for how their customers were trying to test; and I think the end result was way more suitable than 'traditional' statistics were.

[1] https://help.optimizely.com/hc/en-us/articles/200040115-Traffic-Allocation-Limit-experiment-traffic-or-launch-winning-variations
"Traffic Auto-allocation automatically adjusts your traffic allocation over time to maximize the number of conversions for your primary goal. [...]
To learn more about how algorithms like this work, you might want to read about a popular statistics problem called the…

Hey,

If you are interested in learning more about the problems we solved, we lay them out in much more detail in our blog post. It’s a meaty topic and so the post is not short. http://blog.optimizely.com/2015/01/20/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/

In a few sentences: in the past if you didn’t use a sample size calculator properly (set a sample size up front and only evaluate test results at that time) and had tests with a lot of goals and variations, you could increase that chance of making an incorrect declaration. With Stats Engine, we allow you to monitor your results real-time and test as many hypothesis as you would like and give an accurate representation of the likelihood your test is actually a winning or losing test. We’re definitely not trying to claim to have invalidated traditional statistics. If you used the proper testing procedure in the past, then Stats Engine will simply give you an easier workflow than before (no need to pick an appropriate sample size, minimum detectable effect, limit the number of hypotheses being tests). Many companies have data-scientists, statisticians, or are otherwise well-informed on the topic, however many are not. Stats Engine allows you to have an accurate Statistical Significance measure without requiring you to set a sample size because it accounts for the errors that are introduced by looking at your results as experiment data comes in.

In this case you're probably right but in general, latency can make a big impact on profit and revenue.

For example Marissa Mayer claimed (though this was in 2006) that a 500ms delay directly caused a 20% drop in traffic and revenue: https://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html

Optimizely found that for The Telegraph, adding a 4s delay led to a 10% drop in pageviews: https://blog.optimizely.com/2016/07/13/how-does-page-load-time-impact-engagement/

100ms cost Amazon 1% of sales: http://blog.gigaspaces.com/amazon-found-every-100ms-of-latency-cost-them-1-in-sales/

Akamai has a number of claims: https://www.akamai.com/us/en/about/news/press/2009-press/akamai-reveals-2-seconds-as-the-new-threshold-of-acceptability-for-ecommerce-web-page-response-times.jsp

Of course the impact of latency is going to depend on your particular circumstances but there are certainly circumstances where it can make a big impact.

Thanks!

If you look back you can see we've been growing at a pretty steady clip every since we launched: http://trends.builtwith.com/analytics/Optimizely

However you are right that there does seem to be an inflection point in May 2012. I can only speculate why this happened. Here are a few possible reasons:

1. We got really lucky with a bunch of great press at the end of April including a phenomenal piece in Wired: http://www.wired.com/business/2012/04/ff_abtesting/ and a piece on CNN: http://www.youtube.com/watch?v=TEpsfQHNUxU

2. We announced our funding: http://blog.optimizely.com/2012/05/30/optimizely-surpasses-google-website-optimizer-and-raises-money-to-battle-omniture/

3. Probably in some part because of #1 we started ranking very high for organic search terms. For example we are now the first ranked result for "website a/b testing" on Google.

Hope that answers your question!

Author here. So yeah: site seals link to useful info, but their UI doesn't encourage users to click on it: instead it expects users to use the presence of an image in the browsers content area as a source of trust.

Most of HN already knows that, so more importantly:

- The reason the site seal uses JS (rather than a simple link) is that the link is actually to the CA's sales page, not the site report - there's no 'nofollow' so it's a massive search engine rank boost to the CA.

- There's a bunch of studies from non-security industry sources about how 'site seals' actually impact conversions. Some are positive, some are negative, but the biggest takeaway is that 'site seals' increasing conversations is by no means a foregone…