Cal Paterson

> This has never happened and never will. You simply are not omniscient. Even if you're smart enough to figure everything out the requirements will change underneath you.

My best project to date was a largely waterfall one - there was somewhere around 50-60 pages of A4 specs, a lot of which I helped the clients engineer. As with all plans, a lot of it changed during implementation, actually I figured out a way of implementing the same functionality, but automating it to a degree where about 15 of those could be cut out.

Furthermore, it was immensely useful because by the time I actually started writing code, most of the questions that needed answers and would alter how it should be developed had already come up and could be resolved, in addition to me already knowing about some edge cases (at least when it came to how the domain translates into technology) and how the overall thing should work and look.

Contrast that to some cases where you're just asked to join a project and help out and you jump into the middle of ongoing development, not going that much about any given system or the various things that the team has been focusing on in the past few weeks or months.

> It’s not hard to see that if they had a few really big systems, then a great number of their problems would disappear. The inconsistencies between data, security, operations, quality, and access were huge across all of those disconnected projects. Some systems were up-to-date, some were ancient. Some…

There is a surprising amount of important data in various Mongo instances around the world. Particularly within high finance, with multi-TB setups sprouting up here and there.

I suspect that this is in part due to historical inertia and exposure to SecDB designs.[0] Financial instruments can be hideously complex and they certainly are ever-evolving, so I can imagine a fixed schema for essentially constantly shifting time series universe would be challenging. When financial institutions began to adopt the SecDB model, MongoDB was available as a high-volume, "schemaless" KV store, with a reasonably good scaling story.

Combine that with the relatively incestuous nature of finance (they tend to poach and hire from within their own ranks), the average tenure of an engineer in one organisation being less than 4 years and you have an osmotic process of spreading "this at least works in this type of environment" knowledge. Add the naturally risk-averse nature of finance[ß] and you can see how one successful early adoption will quickly proliferate across the industry.

0: This was discussed at HN back in the day too: https://calpaterson.com/bank-python.html

ß: For an industry that loves to take financial risks - with other people's money of course, they're not stupid - the players in high finance are remarkably risk-averse when it comes to technology choices. Experimentation with something new and unknown carries a potentially unbounded downside with limited, slowly emerging…

A few months ago I wrote an article testing out the relative speed of Python's async options for web apis/sites (https://calpaterson.com/async-python-is-not-faster.html).

My findings were a bit controversial at the time as I found that uWSGI + sync Python is about the best option, largely because replacing interpreted Python code with native code is a very significant factor in the performance of a Python program.

In the following discussion (and private emails) I was stunned by the number of async enthusiasts who proposed raising the number of database connections in the connection pool by two or three orders of magnitude (I had 20 conns in a pool for a 4 cpu machine) for reasons along the lines of "the async app is starved of connections".

In my opinion that suggestion betrays a misunderstanding of what is likely going on in the database when you have 100s or 1000s of connections all contending for the same data: in SQL even reads create locks. Async applications yield a lot, by design, and under hundreds or thousands of requests per second there is a considerable buildup of "work in progress" in the form of async tasks that were yielded from and which have not yet been returned to and completed. Many hundreds of database connections is going to create an enormous volume of bookkeeping on the database side and is very likely to slow things down to a absolute crawl.

Even idle connections are known to have detrimental effect in postgres. Someone at Microsoft is…

Yea and this created "bank python" informally a good article is here.

https://calpaterson.com/bank-python.html

The best description about how it became a problem is one of the paragraphs.

"New starters take an exceptionally long time to get up to speed - and that's if they don't resign in fit of pique as soon as they see the special, mandatory, in-house IDE (as I nearly did). Even months in, new starters are still learning quite fundamental new things: there is a lot that is different."

I think it took me til I was there around two and a half years to fully comprehend it when I was working on it. Not much modern training til they figured out they had to teach it again that was better. The worst part is to make an UI around it coding it and it wasn't approved for new projects.

Huh, information in https://calpaterson.com/mozilla.html are, delicately speaking, disturbing.

One thing surprised me as well:

"Firefox barely exists on phones, with a market share of less than half a percent. This is baffling given that mobile Firefox has a rare feature for a mobile browser: it's able to install extensions and so can block ads."

Yes, indeed, FF is pretty great on the phone, it looks like someone fails to advertise it as such. I use it precisely because I can use add blocker. The person with 2.4M compensation should notice that something wrong is going on.

I am afraid that Firefox/Mozilla at some point will become something like Yahoo or Commodore, a name that is being bought and sold by investment founds to extract some income and sell it further...

Mozilla wastes all its money on its CEO and bullshit DEI programs, while neglecting to develop Firefox, their flagship product. When they spend some money on the browser, they use it to implement user-hostile garbage like Facebook's client-side tracking bullshit.

Here's a cute graph of CEO pay vs Firefox market share: https://calpaterson.com/assets/mozilla-boss-pay.svg

Mozilla is completely fucked. I just hope the company goes bankrupt so its niche of an open source browser can be filled by an organisation that actually cares about its product, and is not just a sham for execs to get millions of dollars from Google for being faux-competition, while fucking over their users.