Chris Stucchio

That's rather simplistic, not to mention completely wrong. US markets are highly visible (you just saw a free visualization of HFT activity since 2007), very tightly regulated and well controlled. Things go wrong, as they do in all complex systems, but rarely do they go wrong to the extent that ordinary investors are harmed.

Read this, before jerking your knee any further. Think of it as turning on the bedroom light, showing the monster in the corner to be nothing more than a pile of clothes:

http://www.chrisstucchio.com/blog/2012/hft_apology.html
http://www.chrisstucchio.com/blog/2012/hft_apology2.html

Financial markets have been moving towards zero latency trading since their inception. First, men gathered under trees or in coffee houses, within shouting distance of each other. They fought over the best spots. Then, traders became some of the earliest adopters of telegraph technology, running private lines from remote cities to the exchanges. Ticker tapes appeared in distant offices. Computerized price dissemination followed, together with electronic trading connections to the exchanges themselves. All in the name of getting information and acting on it before the next guy. It's the reason you, as an individual, can trade global markets cheaply and instantly. It helps you take control over your retirement and savings, rather than paying through the nose for some managed scheme where only a privileged few can access the markets.

Where is your arbitrary line where…

Humans are a thousand times worse than algorithms. Unattractive defendants get prison sentences twice as long as attractive ones. Judges give significantly harsher sentences just before lunch, when they are hungry. And of course the classic biases against race and gender and other protected classes. I bet there are many other biases that we haven't even discovered yet. Some studies have shown that job interviews are worse than just looking at resumes.

Humans should never ever be in charge of sentencing decisions, or anything else you want to be "fair". There's nothing remotely fair about human judgement.

One particularly weird bias humans have, is we strongly distrust machines. Psychologists have studied this and named it "Algorithm Aversion": http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf Even when we know machines do better than humans, we are much more distrustful of them. We can watch humans make mistakes and an algorithm make many fewer mistakes, and still end up trusting the human far more.

And it's weird, because in almost every domain that's been studied, algorithms do vastly better than humans. For any statistical problem where there are a few features available to predict the outcome, even simple linear regression typically beats human experts. We suck at probabilistic and statistical problems. Often the predictions of experts are barely better than chance. And this has been known for a long time. The…

The problem of HFT is two.

First, HFT is very difficult. It requires the talents of extremely intelligent individuals. These individuals are a scarce resource. Now, being as they are near the generation and handling of money, HFT firms are best placed to outbid all others for these people. So R&D and other such lose out on top minds. Additionally, if much of the money of high net people is being invested in funds without finding its way into risky high tech research, this too can be a drag on the economy in the long term. http://sirc.rbi.org.in/downloads/4Cecchetti.pdf

The second is a problem of misaligned priorities. It's one thing to suck in all the best minds and it's another to have them all expending a massive amount of effort trying to shave down microseconds so their order goes in first. Chris Stucchio here http://www.chrisstucchio.com/blog/2012/hft_whats_broken.html argues very well that the so called sub-penny rule (abs(changeinprice) >= 0.01) is antiquated and from a time when humans reigned. This rule is a source of friction for the bots and is very likely a key player in the current misallocation of talent. It's removal would make HFTing cheap enough that not so many people would be needed. Some would leave to cure cancer or mine asteroids. The rest could then work on more sophisticated pricing techniques - which is a net boon, maybe those techniques could make prices more stable or be reused for optimal donor matching or something.

I seem to remember a blog post one day about someone interviewing for several "Big Data" positions - where the data turned out to be some Excel sheets. Or fit on a thumb drive. The hiring companies that thought they had "big data" really didn't, was the point. The blog post was widely shared around my office at the time and I see it as the origin of this idea.

Edit: I think it was https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html, HN discussion https://news.ycombinator.com/item?id=6398650

But the idea wasn't that data was big as soon as it didn't fit in Excel anymore - Excel was a reductio ad absurdum. If it still fits in Excel, it's laughably far away from being "big". That was the idea.

Big data was about the problems you get when you need to join data together that you can't fit well in one database server, not even with the amounts of memory and disk space you can get these days. It was Google type problems, the stuff that map-reduce needed to be invented for. The kind of problem that most companies just don't have.

So the term mostly disappeared from job descriptions, "data science" became popular instead, and people keep using Excel because it is great.

(that blog post is from 2013 so it can't be the source of that quote from 2012. But please don't define big data in terms of Excel in a 2021 MSc thesis)

They can actually work together.

Thompson Sampling / Multi-armed bandit algorithms are traffic allocation strategies to use the least amount of traffic to find the optimal variant.

So say you have version A/B/../N variants of a website, you can split the traffic equally or use a a bandit algorithm and find the best performing variant of it.

But that only helps you test N variants. If you want to test for 4 titles and 3 color buttons, 3 layouts and 5 images you already have 4x3x3x5=180 different variations and you can't test them all. Evolutionary algorithms can help you search through a much bigger search space:

First you try 10 variations, and allocate traffic equally or through a bandit algorithm. Then you combine the top performing candidates multiple times to create a new generation and start over again.

Evolution helps you find which candidates to try in a large search space and then a bandits algorithm can help you allocate traffic optimally.

You have to be careful with A/B testing with Bandit algorithms though. If the conversion rate changes over time or if you have visitors that don't always convert instantly you have to take that into account: https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html

I tend to agree that the value of a company doesn't actually change substantially from one second to the next. But to focus on that is to ignore the value of having liquidity available. For an example of what can happen when there isn't enough liquidity, take a look at the graph of RSP on the morning of August 24th.

For a more detailed description of the mechanics and motivation behind market making, I recommend "A High Frequency Trader's Apology": https://www.chrisstucchio.com/blog/2012/hft_apology.html

If traders, be they high-frequency or otherwise, can't make money by making liquidity available, they won't do it. Less liquidity means wider spreads, which means higher costs especially for smaller transactions (i.e. individual investors).

You could reasonably argue that HFT has caused expenses to increase for large investors who need to buy or sell large volumes of shares. But to the extent that is true, it is like saying that large investors used to get on average an unreasonably good deal at the expense of their counter parties. As in any free market, an especially good price for one party is by definition an especially bad price for the other party.