Evan Jones

> I really don't get what people mean when they say they're against frameworks. Your app that you're making with no framework...how are things structured? Do you just repeat yourself everywhere?

So, there are frameworks and there are libraries. With a framework, you use the structure provided by the framework, and hook pieces of your code into it; with a library, you structure your code to your problem, and use libraries to provide functionality.

Frameworks are generally easier to get started with: they provide a lot, and their pieces generally work well together. But inevitably, the day will come when you need to do something not envisioned by your framework, and then you're stuck: it can be arbitrarily difficult, painful and/or low-performance to do what you need. That last point is also worth mentioning: because a framework tries to be good at providing the same structure for a lot of different problems, it inevitably is a jack of all trades and a master of none. This can end up meaning poor performance, due to needing to call lots of functions to do simple things (imagine some hellacious Java framework with AbstractFactoryVisitorSingletons), or it can be a pain on the programmer, or both.

Meanwhile, libraries are just that…libraries of code that the programmer can use to implement needed functionality. How he stitches them together is up to him.

With a framework, your code lives within someone else's structure; using libraries, other people's code is called by…

I'm glad you've at least looked into this, but you have a number of things wrong:

First of all, it is not a single algorithm, it is a family of algorithms

This argument holds for SHA1, but all modern cryptographic hash functions are also families. The SHA3 family has SHA3-224, SHA3-256, SHA3-384, SHA3-512, SHAKE128, and SHAKE256

For a CRC of size N, you have to also choose a N-bit
polynomial, N-bit starting value, and N-bit output XOR
value. There is no single standard, there are tens or
hundreds of popular options

Apparently you've never heard of CRC32C!

http://www.evanjones.ca/crc32c.html

Most programming languages have some CRC32 implementation
- but do not document what parameters they use, or use
different notation for the same parameters (forward and
reverse), or do not let you change the parameters, or do
not let you change the CRC size, or all of these. There is
no easy way to get a "standard CRC64 or CRC128" compatible
across platforms without putting it together from github
snippets and example code yourself.

Here's CRC32C implementations in a handful of popular languages:

- C: https://software.intel.com/en-us/node/503522

- Java: http://download.java.net/java/jdk9/docs/api/java/util/zip/CRC32C.html

- JavaScript: https://www.npmjs.com/package/fast-crc32c https://www.npmjs.com/package/crc32c

- Go: https://github.com/jacobsa/gcloud/blob/master/gcs/gcsutil/crc32c.go

- Python: https://github.com/ludios/pycrc32c

- Ruby:…

A few links gathered from a quick google search as a primer:

http://stevehanov.ca/blog/?id=97

https://www.evanjones.ca/portable-linux-binaries.html

https://insanecoding.blogspot.com/2012/07/creating-portable-linux-binaries.html

https://rpg.hamsterrepublic.com/ohrrpgce/Portable_GNU-Linux_binaries

https://github.com/wheybags/glibc_version_header

In other words: there are a lot of steps and a lot of gotchyas to doing this that you're glossing over. Linux userland libraries are generally designed with the intention that an army of third-party maintainers will integrate all of this desperately developed software together and place it in a repo. Naturally every distribution wants to do things a little differently too, and they have a habit of changing it up every couple years. When you try to step out of that mold, even just to static link, things unsurprisingly become more difficult. Whereas Windows, Mac, Android, etc. have been designed since the beginning not to require that sort of thing and it is consequently a much, much more straightforward process.

I'm curious why, since you seem to believe the process is so straight-forward, you think it is that so few people distribute a simple binary? Why were Flatpak and AppImage invented?

I plan to cover CPython's memory management in the future posts. In a nutshell, an object gets destroyed when its reference count hits 0. In this case, CPython calls `tp_dealloc` [1] of the object's type. The `tp_dealloc` slot releases all the resources the object owns and frees the memory. The implementation of `tp_dealloc` differs for different types. Eventually, the `free` function of the memory allocator is called to free the memory. A memory allocator is a set of functions to manage memory. The default memory allocator for objects is pymalloc. It allocates small objects (<= 512 bytes) using the arena allocator [2] and falls back to the raw memory allocator otherwise. The latter calls the `free()` library function to free the memory.

The Python/C Reference Manual has a great section on memory management [3].

[1] https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_dealloc

[2] https://www.evanjones.ca/memoryallocator/

[3] https://docs.python.org/3/c-api/memory.html

I can see where my statement was ambiguous, I'd intended to say that the GC developers spend tons of time tuning their implementations to fit their users' workloads.

I will say that "government project which serves the whole country now" doesn't really specify how many requests you're doing or what kind of pauses are acceptable. If it's a traditional website, a 300ms GC pause might be invisible. A 2s pause might even be invisible.

For a service doing 30,000 requests/s (~1-2k/s per machine), a single STW pause of 2s is _very_ disruptive. Even 300ms is high. A 300ms pause is 600 queued requests @ 2k/s. We certainly tune this service to eliminate most STW GC. We were even hit by http://www.evanjones.ca/jvm-mmap-pause.html .

Its actually very useful, although very weak. The link layer check, at least in the case of Ethernet, really only protects your data "on the wire". It doesn't protect it inside the switches. It turns out there are failure modes, such as the one that caused this issue, where the packets get corrupted inside the switch (probably due to bad RAM). Details about how this can happen: http://www.evanjones.ca/tcp-and-ethernet-checksums-fail.html

After this kernel bug was fixed, the TCP checksum did its job and discarded all the corrupt packets.