The blog of Graylog, an open source log management platform built on Elasticsearch, covering releases and logging features.
HN commenters recommend Graylog as a log management option, noting its dashboards are less mature than Splunk's, and compare it with Grafana Loki.
LXC Containers:
- Koel (Web-based music player; https://github.com/phanan/koel)
- Aria2 (generic downloader; https://aria2.github.io/)
- GPodder (podcast downloader; https://gpodder.github.io/)
- PiHole (mostly for curiosity, I now use full-blown DNS & DHCP servers; https://pi-hole.net/)
- OpenVPN (https://openvpn.net/)
- Pritunl (OpenVPN failover; https://pritunl.com/)
- PlexPy (Plex monitoring; https://github.com/JonnyWong16/plexpy)
- BIND (DNS server; https://wiki.debian.org/Bind9)
- ISC-DHCP (https://wiki.debian.org/DHCP_Server)
- Smokeping (connection monitoring; https://oss.oetiker.ch/smokeping/)
- Ansible (central host with playbooks; https://www.ansible.com/)
- Graylog (log aggregation; https://www.graylog.org/)
- Wallabag ("read it later"; https://wallabag.org/en)
- Home Assistant (home automation; https://home-assistant.io/)
- Bookstack (documentation; https://www.bookstackapp.com/)
- Observium (network devices monitoring; https://www.observium.org/)
VMs:
- PfSense (firewall & permanent VPN for some containers; https://www.pfsense.org/)
- Rancher (Docker orchestration; http://rancher.com/)
Rancher (Cattle environment):
+ ELK stack (https://www.elastic.co/)
+ Concourse CI (just to experiment, I prefer GitLab's CI; https://concourse.ci/)
+ Dumpster (my own temporary file upload platform, with additional YubiKey support; https://github.com/nmaggioni/dumpster)
+ Own Telegram bot (talks with ELK)
+ Let's Encrypt renewal daemon
+ FaaS (either…
In my experience, i've seen the modular code approach more often than separate deployment approach, quite possibly because the latter is still a bit harder to do when compared to just having 1 instance (or people are just lazy and don't want to take the risk of breaking things that were working previously for future gains), but sooner or later the question of scalability does come up, at least in successful projects.
Let me tell you, as someone who has delivered a critical code fix for business continuity after midnight a few times, slapping N instances of an app runtime in a data center somewhere is way easier than having to struggle with optimizations and introduce more complexity in the form of caches, or write out Hibernate queries as really long and hard to debug SQL because people previously didn't care enough about performance testing or simply didn't have a feasible way to simulate the loads that the system could run into, all while knowing that if your monolith also contains scheduled processes, none of your optimizations will even matter, because those badly optimized processes will eat up all of the resources and crash the app anyways.
In short, the architecture that you choose will also help you mitigate certain risks. Which ones you should pay attention to, however, depends on the specifics of your system and any compliance requirements etc. Personally, as a developer, fault tolerance is up there among the things that impact the quality of my life the most,…
From a comment on the Elastic post earlier today:
"We build something on Elastic - and trying to work with them to get licensing was at best a major PITA. In ~2017 Elastic decided they wanted to go Enterprise Only. They started to charge $100k for x-pack, though one could get a dev license for $50K.
We quickly built a SQL parser for Elastic, an alerting engine, some other bits. These are all bits that just went poof yesterday when Amazon released opendistro - and I couldn't be happier. Sure we'll have to slice out a bunch of code, but at the end of the day it didn't have value to our core business."
https://news.ycombinator.com/item?id=19370085
The trouble is, once an open source company puts essential stuff under a shared source license, and calls it open source, you don't know if that's all they're going to do, or if they're going to keep charging more, or putting more essential stuff under a non open source license.
The simplicity that drew people into using the open source product is gone, and there's an uncertain future.
It's hard to find an open core product that has only enterprise features in its enterprise offering. For instance, in Greylog2, views and a user audit log would be useful to pretty much anyone. [1] And people would probably be happy to implement and contribute it themselves. And the company wouldn't merge it in because it competes with their enterprise offering. With nginx, a feature whose usefulness is not limited to enterprise that is only in…
Neat! Definitely a better solution for single source logs. Splunk is ridiculous and Cisco acquiring it isn't going to make that better.
For others with a bit more complex needs, take a look at the free (or paid) versions of Graylog Open[1].
It's really improved over the years. I had messed with Graylog in it's early days but was turned off by it. A few years back, I encountered someone doing some neat stuff with it. It looked much improved. I stood up a "pilot project" to test, and it's now been running for years and several different people use it for their areas of responsibility.
It does log collection/transforming and graphing and dashboarding and we use the everloving crap out of it at work. I wish I could publicly post some of the stuff we're doing with it.
It takes input from just about any source.
1. https://graylog.org/products/source-available/
Hi snewman,
I am currently using logdna as my splunk alternative, but it's not very good.
Why should I try scalyr next, over the other alternatives ?
My main constraint is the solution should be at least half as good as splunk, and cost less than $50 a month (I generate less than 1G a day).
Your pricing seems to be reasonable. But how do you compare to splunk on features ?
Here are the other tools I am considering trying
https://www.humio.com/
https://www.sumologic.com/
https://sematext.com/logsene/
https://www.datadoghq.com/log-management/
https://logz.io/pricing/
https://www.chaossearch.io/
https://www.papertrail.com/
https://www.loggly.com/
https://www.graylog.org/products/enterprise
I’ve personally used Graylog[1] with success in the past.
However, I’ve had an excellent time with Grafana and have been following their Loki[2] project.
My company uses other solutions, so I haven’t needed it, but the Grafana stack might suit your use case.
[1] https://www.graylog.org/products/open-source
[2] https://grafana.com/docs/loki/latest/