Dave Tf, a software and infrastructure engineer, writes about Kubernetes, networking, low-level systems details, and LLM coding.
HN commenters discuss his Kubernetes essay, with some sharing his operational concerns and others disputing that it is inevitable, and they link his jumbo frames post.
It's so saddening to see how the Kubernetes hype-cycle follows OpenStack and all the fundamental problems still seem unsolved. I sometimes feel like its just the same story playing out 5 years later, one layer up to the stack (IaaS -> CaaS) and with other fools to fall for it (with OpenStack it was sysadmins trying to run a control plane, with Kubernetes its devs trying to run infrastructure).
The abstractions we have available to build and run distributed systems may have improved, but they still suck in the grand scheme of things. My personal nightmare is that nothing better comes along soon.
> - Is it the networking model that is simple from the consumption standpoint but has too many moving parts for it to be implemented?
Many poor sysadmins before us have tried to implement Neutron (OpenStack Networking Service) with OvS or a bunch of half-assed vendor SDNs. Or LBaaS with HAProxy.
> - Is it the storage model, CSI and friends?
I mean, the most popular CSI for running on-premise is rook.io, which is just wrapping Ceph. Ceph is just as hard to run as ever, and a lot of that is justified by the inherent complexity of providing high performance multi-tenant storage.
> - Is it the bunch of controller loops doing their own things with nothing that gives a "wholesome" picture to identify the root cause?
Partially. One advantage the approach has is that it's conceptually simple, consistent and feels easy to compose complex behavior. The problem is that Kubernetes enforces…
> As far as I can see, the best choice for writing bootstrap-related software in 2024 is still C99
That seems a bit silly, since going from ISO C99 to C++ is only one more build away[0] and from there you can practically build the whole world as you please.
> CMake is absolutely out
Considering how hostile Meson is to platforms other than *nix[1], I doubt there is any realistic alternative here.
> I still haven’t forgiven it for deciding that the space-separated output of pkg-config meant it was a “list”, then emitting that list as a semicolon-separated string into my build commands.
I'd be curious where the issue opened by the author is, since this would be a pretty big problem if it was truly an issue. The CMake module processing the output of pkg-config is also just using an existing CMake command[2], so the impact would be greater than just pkg-config related code if there really was an issue.
[0]: https://blog.dave.tf/post/finding-bottom-turtle/
[1]: https://github.com/mesonbuild/meson/issues/8153
[2]: https://gitlab.kitware.com/cmake/cmake/-/blob/f8ba5b12ca842ce1c02c22bf25ff52e2bc18e7b6/Modules/FindPkgConfig.cmake#L163
IMO, Kubernetes isn't inevitable, and this seems to paint it as such.
K8s is well suited to dynamically scaling a SaaS product delivered over the web. When you get outside this scenario - for example, on-prem or single node "clusters" that are running K8s just for API compatibility, it seems like either overkill or a bad choice. Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs.
There are also folks who understand the innards of K8s very well that have legitimate criticisms of it - for example, this one from the MetalLB developer: https://blog.dave.tf/post/new-kubernetes/
Before you deploy something, actually understand what the pros/cons are, and what problem it was made to solve, and if your problem isn't at least mostly a match, keep looking.
Besides my local cluster of virtual box cluster, I have tried Kubernetes on three clouds with at least a dozen different installers/distributions and operational pain would be a factor going forward has always been my gut feeling.
That's where the author also has following to say:
>My conclusion at this point is that if you can afford it, both in terms of privacy/GDPR and dollarinos then managed is the way to go.
And I agree. Kubernetes managed is also really hard for those of offering it and have to manage it for you behind the scenes.[0]
[0]. https://blog.dave.tf/post/new-kubernetes/
There's some weird legacy parsing for IPv4 addresses with less than four components. 1.2.3.4 can be represented equivalently as 1.2.772, 1.131844, or 16909060. (Note that 772 = 0x0304, 131844 = 0x020304, and 16909060 = 0x01020304.)
TL;DR: the second "1" in "1.1" is treated as a 24-bit integer.
Longer explanation: https://blog.dave.tf/post/ip-addr-parsing/
> Now I want the story of the 9000 bytes MTU.
* 2018: https://blog.dave.tf/post/jumbo-frames/
* April 2023: https://news.ycombinator.com/item?id=35647779