InfluxData's company blog announces releases and explains the design of InfluxDB, its open source time series database, including its storage engine.
HN commenters link it when discussing time series databases. One finds InfluxDB very nice for modest data but limited in clustering; another notes slow HTTP data retrieval.
Why would you choose a LSM Tree based storage mechanism for a message queue?
The only reason I can come up with would be because it's a read-to-use library you can just plug in which gives OK performance and some handy features because you can use the KV store for other things. But it doesn't scale well and backups with LevelDB are not really easy either (close DB, copy all files).
Message queues when they are ordered (at least on the local node/queue level) usually just need some kind of append-only log file. You don't do random reads or writes into the middle of the queue, you only modify the head and tail.
InfluxDB, albeit being a time series db has similar write patterns to a message queue, learned it the hard way when they first tried to use a LSM Tree database (LevelDB), then switched to a B+Tree (BoltDB/LMDB) but that also doesn't scale once the DB gets big and the tree has quite some depth.
They kindly did a nice writeup of their journey: https://influxdb.com/docs/v0.9/concepts/storage_engine.html
Why not do it simple and use append-only files without complex structure and management?
Check out Kafka for a better storage format for message queues of this kind.
PS: every message queue should first clearly explain what guarantees it provides.
Hi and thanks for the new release!
With the new distinction between tags and fields, the documentation isn't clear about how to use them. I assume tags are indexed, but that you can still query by field, and that there are no limitations?
We're basically just throwing documents (containing nested documents) at InfluxDB, with no predefined schema, so we don't know beforehand what users will be performing ad-hoc queries on; it's completely dynamic in the UI. Someone, perhaps it was you, mentioned in another thread that InfluxDB only supported 255 tag names per series, which we'd probably exceed with our schema. At the same time we'd like to benefit from the performance we'd presumably get from using tags.
PS. This page is giving me an error "NoSuchKey": http://influxdb.com/docs/v0.9/concepts/aggregate_functions.html
This post it's quite old and graphite ecosystem improve, some examples:
Grafana from my POV is the best dashboard at the moment:
http://grafana.org/
http://grafana.org/blog/2014/05/25/monitorama-video-and-update.html
http://play.grafana.org/
About alert system I'm using cabot:
http://cabotapp.com/
About system metrics, at the moment I'm using Diamond.
https://github.com/BrightcoveOS/Diamond
In the other hand, Influxdb is growing, maybe I'll switch to InfluxDB next year, some features are better than graphite, and it's statsd compatible ;-)
http://influxdb.com/
Regards ;-)
They've gone through a few engines. They have a post about the newest one which isn't quite out yet:
https://influxdb.com/blog/2015/10/07/the_new_influxdb_storage_engine_a_time_structured_merge_tree.html
We use them for storing "permanent" data. They slice up the data into weekly files, so if you don't access it often, it's no biggie to store a lot of data. If you constantly access data across time, you may run into performance degradation from a caching point of view (having to pull in the files from past, blowing caches of new stuff constantly).
Looks interesting. This answers a lot of questions about primary data, but I'm interested to know how continuous queries [1] are handled.
If they're stored like any other data, the "inserts are far more likely than updates" assumption doesn't really make sense. For example, what if you have a continuous query grouped by year?
PS. "challenges" is mis-spelled in the last sentence.
1. http://influxdb.com/docs/v0.8/api/continuous_queries.html
I have used http://influxdb.com/, I have used it with a few million records. Getting the data out is a bit slow because it goes over HTTP. Also make your InfluxDB library of choice can deal with HTTP chunking. I found that if you request a lot of data from InfluxDB and the system does not have enough memory, the process will silently die.
If you have mega huge data http://opentsdb.net/ seems pretty decent, however I have not tried it out.