Steve Hanov

Most or all of the points suffer from:

Let's address them one by one.

is that actually true?

In every case, yes.

does regular a/b testing not also face that issue?

For the big ones, regular A/B testing does not face that issue. For the more complicated ones, A/B testing does face that issue and I know how to work around it. With a bandit approach I'm not sure I'd have noticed the issue.

was it suggested that you must "set it and forget it"?

Not "must", but it was highly recommended. See paragraph 4 of http://stevehanov.ca/blog/index.php?id=132 - look for the words in bold.

are there no mechanisms for mitigating these issues?

There are mechanisms for mitigating some of these issues. The blog does not address those. As soon as you go into them, you get more complicated. It stops being the "20 lines that always beats A/B testing" that the blog promised.

I was doing some back of the envelopes on different methods of mitigating these problems. What I found was that in the best case you turn into

would using 20% or 30% mitigate the issues?

That would lessen the issue that I gave, at the cost of permanently worse performance.

The permanent performance bit can benefit from an example. Suppose that there is a real 5% improvement. The blog's suggested approach would permanently assign 5% of traffic to the worse version, for 0.25% less improvement than you found.

Now suppose you tried a dozen things. 1/3 of them were 5% better, 1/3 were 5% worse, and 1/3 did not…

Effectively, you need to keep a small heap and if possible have "simple" data structure. In my case, I implemented the Python code of a trie based levenshtein distance algorithm[0]. I used maps because it was basically a direct translation of the Python code. This resulted in millions of maps and oft the index would just stop in the garbage collector for a couple of seconds. I solved it using my own map which is travelled linearly.

type SpellCheckerMap struct {
Runes []rune
Nodes []*SpellCheckerNode
}

with only few runes, I can just go through the list to find the index of the following node. You save the computation of the map hash key. It is pretty fast, here is an example of a spelling suggestion over a 500,000 words corpus[1].

Basically, the garbage collection cost forced me to think about a better data structure for my case. Maybe not that bad all in all.

[0]: http://stevehanov.ca/blog/index.php?id=114
[1]: https://www.chemeo.com/search?q=asparine

One of the draw backs that typically gets mentioned about current distributed systems is the lack of locking which makes managing un-merge-able (binary) files impossible to manage correctly. This is why you will not hear of many game dev studios using git, version controlling art assets is fairly common in such situations and perforce handles it like a champ. DVCSs in general tend to have performance issues with large files, I know mercurial warns you about the performance implications of storing anything over 10MB.

A bit old(2009) so it may have changed, but here is a rant about the scalability of git vs perforce.
http://stevehanov.ca/blog/index.php?id=50
and a little newer post on git scalability issues:
http://news.ycombinator.com/item?id=3548824

Just because a problem has been "solved" for one platform (Fast PC's w/ Gigabytes of ram/disk) doesn't mean it is solved for all platforms (Web apps for phones/tablets, etc).

John Resig (of jQuery and Khan Academy fame) had a really interesting thread on creating a fast-loading (i.e. small, efficient) dictionary for a JavaScript app. http://ejohn.org/blog/revised-javascript-dictionary-search/

The comments have some really clever ideas for encoding the dictionary and Resig weighs in on his favorite solution with informative data. There's an interesting "Succinct Trie" data structure that turns a 620K dictionary into a 220K string that doesn't need to be decoded to be searched.
http://stevehanov.ca/blog/index.php?id=120

The trie is my favorite data structure. It allows you to build inverted indices (i.e. hashmaps) that can also be used for prefix lookups of tokens and for spelling/typo correction.

With a trie, it's very efficient to calculate Levenshtein distances. Here's a post (http://stevehanov.ca/blog/?id=114) that has a good primer on that.

If you love this sort of thing, here's a fast and neat memory efficient Trie variant: https://db.in.tum.de/~leis/papers/ART.pdf (Adaptive Radix Tree).

I've used ART as a building block to create an open source typo tolerant search engine called Typesense (https://github.com/typesense/typesense).

I think practical solution to this is to return all words that are within a Levenshtein distance[1] of 2*number_accented_characters of the original word, ranked by the distance.

Levenshtein distance can be calculated using a trie[2] so it's fast enough for most purposes.

If you wanted to get fancy you could modify the Levenshtein to reduce the distance penalty on accented characters, and/or only apply the Levenshtein calculation within 1 character of an accented character or the equivalent ASCII substitution sequence.

[1] https://en.wikipedia.org/wiki/Levenshtein_distance

[2] http://stevehanov.ca/blog/?id=114