InfluxData Blog

InfluxDB 3 just went alpha, and they similarly have very severe limitations on what their open-core product will do, very strongly drive folks to upgrade to enterprise.

https://www.influxdata.com/blog/influxdb3-open-source-public-alpha/ https://news.ycombinator.com/item?id=42684524 https://news.ycombinator.com/item?id=42703113

While I admit that I don't think losing open source developers is actually that big a harm to many projects (there's just not enough people out there to drive by big valuable amazing features), I feel like the open core approach shuts yourself off from most people who are looking for open source solutions. The core is not enough.

No one's going to be happy running a 500x slower python project knowing there's the real deal running elsewhere, with a hip new runtime they can't get.

I recognize that for some of these companies, this probably is a necessary move. They need revenue to do what they do and it's hard to get revenue in open source. But these are both interesting products that I was hopeful for that I can't imagine adopting anymore. That's fine, I don't demand being served by anyone, but it is really sad to see, and I wonder how many awesome projects that would have grown big stop these technologies will never be created, because of these shifts.

Matrix especially feels like a brutal loss, because we are so short of good communication systems. I regret not seeing DataFusion & Arrow being out to use & integrate on with InfluxDB 3 but at least…

The blog post series seems to bury the lede -- it isn't until part 3 [1] that we get to the insight that MQTT and Kafka solve different problems and therefore can have complementary roles in the same system.

We use this architecture for IoT: MQTT for edge because it's standard and super good enough, and Kafka because it turns out we want to do more than one thing with the data as it streams in so not using Kafka would end up being more complicated.

Here's a key insight though: for IoT you don't want to use an actual MQTT broker, like Mosquitto or HiveMQ. If you do, it's hard to avoid data loss. Server side you have something that subscribes to those MQTT topics and pushes the data into Kafka. What do you do when that thing needs to be restarted? Ok, you can use persistent sessions in your MQTT broker. But how much memory does your MQTT broker need? What if your MQTT broker crashes? Oops, now your MQTT broker needs its own persistent database to keep track of all those messages in limbo.

What you want is an MQTT gateway -- something that looks like an MQTT broker to the devices, but the server side does something different with received messages. When it gets the MQTT PUBLISH command, it sends the message to Kafka, waits for the ack from Kafka, and only then sends PUBACK back. Presto, the MQTT thing is now stateless and horizontally scalable. Your clients just need some retry logic.

Maybe Kafka's mqtt-proxy does this, I don't know. I don't think it's mentioned in part 3.…

The link to Slack in this post by "developer advocate" Jay Clifford is a rabbit hole worth diving into a bit: https://community.influxdata.com/t/getting-weird-results-from-gcp-europe-west1/30615/12

Find your way into that Slack workspace via https://www.influxdata.com/blog/introducing-our-new-influxdata-community-slack-workspace/ and you'll discover some interesting things:

* That the only reason being given in that thread for the shutdown is "The region did not get enough usage or growth to make it economically viable to operate, so it became necessary for InfluxData to discontinue service in those regions.". i.e. there's no regulatory issue here like other answers speculated - just pure cost-cutting.

* That on July 5th, a couple of hours before they started shutting everything down (based on the shutdown timeline at https://status.influxdata.com/), that same "Developer Advocate" announced that they were suspending their live "office hours" sessions for July.

* Multiple people are asking for help after finding that they can't connect and getting ABSOLUTELY NOTHING in the way of support from the company. It's literally falling to _other users_ to tell them that all their data is gone.

* One person chiming in, Matthew Allen, DID have a colleague who saw the notification email, but notes that...

- it was a pain for him to migrate has data to a different region due to InfluxDB Cloud's rate limiting, but that he did it anyway

- ... but that the documented migration…

I have been currently exploring the option of open sourcing a project of mine and have researched CLAs a bit, but would love to be corrected if mistaken.

It is as simple - and it is not. You need to take care of special cases, like contributions from company employees in their free time (their company could still own rights to this work), people contributing other people's code (SO answers), patents and whatnot. Fortunately there are existing CLA agreements (Apache for instance).

As for this being the social issue, I don't know yet how big a problem that is. It's never bothered me before, as I recognize that maintainer might want to take the project in another direction in the future and since I don't want to maintain it, I am just happy that they are doing it for me. I will use CLA for my project and if someone doesn't like it, it's fine too - I don't mind forks (if they are well maintained, I'll just switch to them ;), and if someone doesn't want to contribute because of this, I don't want their contribution to be in my code anyway, because it limits my options in the future. There's a new post today on HN [0] that presents options pretty nicely, and it is very aligned with the conclusions I came to.

[0] https://www.influxdata.com/blog/its-time-for-the-open-source-community-to-get-real/

So far working with Rust has been great. We haven't had to build any special build tooling. The build times are greater than Go, but we're able to keep things manageable by breaking the project into separate Crates (all still within the same repo and directory tree), which can take advantage of incremental compilation.

Rust has the memory control we're looking for with the safety of a higher level language. For programming that requires concurrency, as most server software including this project does, it's fantastic as the compiler will prevent data races. The way Rust has you deal with errors also makes it easier to write correct code at compile time rather than finding problems at runtime when they can be harder to track down.

Most of what I wrote here a few years ago still feels true:
https://www.influxdata.com/blog/rust-can-be-difficult-to-learn-and-frustrating-but-its-also-the-most-exciting-thing-in-software-development-in-a-long-time/

I also talk about using Rust in the announcement blog post here: https://www.influxdata.com/blog/announcing-influxdb-iox/

I run a whole bunch of different servers and apps in Docker on an old laptop with an i7 processor and 16GB RAM.

One of my favourite self-hosted apps is WikiJs (https://wiki.js.org/). It takes a git repository of markdown files and turns it into an editable wiki and syncs back changes to the git repository.

I also use InfluxDB with Chronograf and Telegraf (https://www.influxdata.com) to collect and analyse logs. It's not quite as full featured as Elastic or Prometheus but is easy to use, rock-solid and nice to look at. Plus they sent me a free pair of socks (all the way to NZ) for filling in a survey! :-)

And to help manage all the Docker images I spin up and down, I use Portainer - https://portainer.io/

Forgot to add that just recently I've started to run Apache Guacamole to give me remote access to my local LAN while I'm not at home. This is so great - HTTPS access to RDP, VNC, and SSH sessions. https://guacamole.apache.org/