Aaron Drake

It feels like that dev time cost is instead being crowdfunded by energy, disk, CPU and RAM that every user has to pay with and eventually that cost is paid by Earth itself. An app might be cheap or free but in the long time cheaper for the planet and each user would be an app costing a dollar or two more per install. I was using a 2011 laptop with only an HDD, 2 gigs of RAM and no discrete GPU until it actually went and finally broke last year so it irks me especially when I see this handwaving of "computers are cheap", especially from Westerner millenials or developers from SF. The fact that well off people who change machines every few years can even dare to call less well off people with older and shittier hardware "entitled" for wanting performant snappy software just like they got 5-10 years ago when they bought their machine is baffling. Not everyone needs a crazy and new machine, writers, reviewers, sales people, admins, etc. Case in point - G.R.R. Martin uses a DOS machine to write - https://www.youtube.com/watch?v=X5REM-3nWHg .

I.e. Slack and Atom got absolutely lambasted for performance, sluggishness and resource use (while VS Code was applauded, so it's clearly not an Electron specific thing) despite being made by companies valued in billions and based in the most expensive region of the world, one of them even being a paid product.

Or a game with pixel art (I do like it and I understand that particular indie dev optimizing for time with such a niche product so…

Thanks for writing this. I think many of your observations are correct (and agree with them). I'm not as sold on your solutions though. Here's some thoughts:

Abstractions are a correct way to do things when we don't know what "correct" is or need to deal with lots of unknowns in a general way. And then when you need to aggregate lots of different abstractions together, it's often easier to sit yet another abstraction on top of that.

However, in many cases we have enough experience to know what we really need. There's no shame in capturing this knowledge and then "specializing" again to optimize things.

In the grand 'ol days, this also meant that the hardware was a true partner of the software rather than an IP restricted piece of black magic sealed behind 20 layers of firewalled software. (At first this wasn't entirely true, some vendors like Atari were almost allergic to letting developers know what was going on, but the trend reversed for a while). Did you want to write to a pixel on the screen? Just copy a few bytes to the RAM addresses that contained the screen data and the next go around on drawing the screen it would show up.

Sometime in the late 90s the pendulum started to swing back and now it feels like we're almost at full tilt the wrong way again despite all the demonstrations to the contrary that it was the wrong way to do things, paradoxically this seemed to happen after the open source revolution transformed software.

In the meanwhile, layers and layers…

While awk is indeed under-appreciated, there are many instances where using grep to pre-filter your input is helpful because the grep family (GNU grep, agrep, etc.) can match strings much much faster than awk's regex engine.

For example: GNU grep uses a heavily optimized implementation of the Boyer-Moore string search algorithm [1] which (it is claimed) requires just 3 (x86) cycles per byte of input. Boyer-Moore only searches for literal strings, but grep will extract the longest literal from the search regex (e.g. the string "foo" from the regex /foo(bar|baz)+/) and use Boyer-Moore to find candidate matches that are then verified using the regex engine.

So if you have a large corpus of text to search, grep is your friend. It can easily saturate your I/O bandwidth and perform an order of magnitude faster than typical "big data" platforms. [2]

[1] https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html

[2] https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html

In this order, I recommend:

1. Bash. Can you get the answer you need with awk, sed, sort, etc.? Linux commands behave nicely with CPU and RAM, and "Command-line Tools can be 235x Faster than your Hadoop Cluster" (https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html).

2. SQLite. The maximum database size is 140 terabytes, and SQLite can join different database files together. First I would recommend using bash to cull your data down before import, if possible (https://www.sqlite.org/limits.html).

3. PostgreSQL. Again, I would recommend using bash to cull your data down before import, if possible (https://www.postgresql.org/about/).

Maximum Database Size Unlimited
Maximum Table Size 32 TB
Maximum Row Size 1.6 TB
Maximum Field Size 1 GB
Maximum Rows per Table Unlimited
Maximum Columns per Table 250 - 1600 depending on column types
Maximum Indexes per Table Unlimited

More: "Bane's Law (Going Deep)" (https://news.ycombinator.com/item?id=8902739)

>It's that in some scenarios python can't be made fast.

Can you give some examples of this? I mean, obviously with enough effort you can "make python fast" since it has good C bindings, and can just be a thin wrapper around fast stuff. Similar to how command line tools can be ridiculously fast[^1] despite, ostensibly, running in bash.

So I'm a bit confused about what you're claiming. Organizational issues, it's difficult to get management on board with an optimization pass?

[^1]: https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html

Hi HN! I was working on a project some time ago where I created a web page (http://tinysite.aadrake.com) deliverable in a single IP datagram and needed a way to see the size of compressed data I was using for testing. I created a utility for this, and now it has a frontend (not for mobile) and lives at https://compresstest.com.

The utility is a custom TCP server written in Go, for those who are curious, and the more detailed writeup is available on my website (https://aadrake.com/posts/2017-10-29-the-biggest-smallest-website.html).

Hope it's helpful!