Evan Miller

Hi

Have you looked into these two?

- Trustworthy Online Controlled Experiments by Kohavi, Tang, and Xu

- Statistical Methods in Online A/B Testing by Georgi Georgiev

Recommended by stats stackexchange (https://stats.stackexchange.com/questions/546617/how-can-i-learn-statistical-a-b-testing)

There's a bunch of other books/courses/videos on o'reilly.

Another potential way to approach this learning goal is to look at Evan's tools (https://www.evanmiller.org/ab-testing/) and go into each one and then look at the JS code for running the tools online.

See if you can go through and comment/write out your thoughts on why it's written that way. of course, you'll have to know some JS for that, but it might be helpful to go through a file like (https://www.evanmiller.org/ab-testing/sample-size.js) and figure out what math is being done.

Thanks for sharing this post. Bret mentioned this in his "Media for thinking unthinkable" talk that Mathematical notation itself is a great tool for making mathematical structure visible. They are like user interface, it tell things that language can't express. The birth of mathematical notation is considered to be birth of modern mathematics. With dynamic media provided by computer, there should be a better way to dynamically represent mathematical structure. With that in mind, I would also recommend Evan Miller's comment's on Bret Victor's "Kill Math" article called "Don't Kill Math". Evan empathize that besides dynamic simulation, we should also embrace "theory and calculation" to see the underlying structure.

http://worrydream.com/MediaForThinkingTheUnthinkable/
http://www.evanmiller.org/dont-kill-math.html

"practical" as in: the programing model fits the problem domain model of modern systems.

But, "practical" doesn't show up on everybody's radar much these days with so much effort focused around fads-of-the-half-decade.

Erlang is made by professionals with decades of experience who grow the system towards being the most stable, scalable, safe, and as understandable as possible. People use Erlang, solve problems, stay quiet, then make money using it.

On the other hand, the thought leader these days seems to be Go (even though it's made by out of touch weirdos (review: http://www.evanmiller.org/four-days-of-go.html)) and has a creepy cult/fanatical following more suited to tribal sportsball teams than what we would expect from professional technical work needing to be stable over the long term.

The procedure I suggested starts with setting the maximum number of conversions before beginning the test. It is emphatically not the procedure that Evan Miller criticizes.

In fact see http://www.evanmiller.org/sequential-ab-testing.html for Evan Miller suggesting a similar procedure to mine, based on past conversations that I have had with him. The difference between that one and mine is that I focus on "make the best decision available, even though it may be only little better than a coin flip" while he tries to only decide when you can decide with confidence. In past conversations we have agreed that each other's procedures are reasonable, but they achieve different goals.

So damn true, it's so difficult to get statistically significant A/B test results, and the popular tools actively lead you astray. It's rare enough for a startup to have the traffic to meaningfully get results on their homepage before the heat death of the universe, let alone random landing or in-app pages.
I'd recommend anyone reading this who does or wants to do A/B tests read: https://www.evanmiller.org/how-not-to-run-an-ab-test.html It was one of the trickiest lessons I had to learn.
Once you realize you need to set your sample size in advance, you actually have to do the math to figure out the traffic you'll need. That's when reality hits you.

I respect the desire to work on low level language like C. Programmers should not be afraid of it.

But that said, building a proper HTTP stack is not trivial.

If you want to use C language, then why not create Nginx module?

Nginx already solved the hard problems:

* HTTP parser

* Distribute work via event loop on multiple workers

* Useful load balancing strategies (not as great as HAProxy, but i am satisfied with it)

* Serious effort in dealing with CVE

* Widely used and battle tested

Here's a fine guide on how to write Nginx module in C:
http://www.evanmiller.org/nginx-modules-guide.html