SigNoz Blog

Can't share the company name, but it's a relatively small startup with a few thousand monthly active users.

Here's what our current stack looks like:

- LLM Access: Custom proxy layer to an OpenAI-compatible gateway for chat completion, transcriptions, image generation, embeddings (we started with LangChain but were fighting with it more than it helped us). Now we use the OpenAI SDK + a custom API gateway that handles cost tracking, metered billing, API with model fallbacks, aliases, and feature flags (e.g., supports_structured_output, supports_reasoning, modality: text, image, audio, etc.).

- Workflow Orchestration: We started with a very simple task executor + background job queue to handle retries and durable executions. Now it has become a real limitation, and we are looking for a complete replacement, considering Temporal, trigger.dev, and Conductor. I don't see an obvious winner here, everything looks like a really sophisticated piece of tech that will have a learning curve and require some rethinking of our decisions and serious refactoring to support new flows.

- Observability: OTEL + https://signoz.io, we don't track LLM outputs for security reasons (consumer product, a lot of private stuff).

- Cost Tracking: We embedded cost tracking into our API proxy layer. For each request, we estimate usage and push it into ClickHouse (via TinyBird) to provide analytics of product usage and to our metering/billing provider (Stripe meters, but evaluating Orb to replace…

What I’ve seen happen is that people pay Datadog and, for some reason or another, they:

1. Cannot really get the whole team on Datadog onboarding (Development too, not just “Infra”), so there are these really deep gaps that will never, ever be covered. 2. Don’t use all the services they pay for: “Oh, these CI/CD observability pipelines thing that costs us half our infrastructure? Yeah, I suppose they run fine.”, “Oh, these Profile Hosts? Yeah, just in case we use them, although it was 11 months the last time we did use them...” 3. Pay in excess for services they don’t take advantage of because of #1. 4. Cannot allocate time to follow through with their Account Manager and apply the spending suggestions they sometimes make. “Hey, you’re spending tons on Profile Hosts, but I don’t see usage. Have you looked into that? Oh, yeah, some day we will...” I would definitely not call them useless, but gosh they are expensive for the minimum of things, which I agree if you cannot commit to the points above, you might as well setup your own Prometheus + Grafana + Loki + Mimir + Tempo stack and see what you can come up with, or even try one of the other alternatives I’ve seen around, like SigNoz[1].

-----

[1]: https://signoz.io/

Hi, I want to love this product. This will be the 3rd or 4th time I have looked at the docs to figure out basic setup.

Each time I look into this. I look to see how it can report CPU/Memory/Disk of the systems at large. When I read, all I find on open telemetry and SigNoz is how to integrate with application stacks.

Am I missing something fundamental in that SygNoz/OpenTelemetry do not integrate with host level metrics? I really, really want to use this product. But this extremely BASIC implementation to use the service is 100% missing for documentation as far as I can tell. Even the example page has nothing listed https://signoz.io/docs/tutorials/ ...

So I am either trying to find out how to make SygNoz do something it was not built to do... or the documentation has a huge hole in it.

Let me know how to proceed, if you have time. It's appreciated so that I dont keep walking down the wrong road and hoping to find something useful.

Thanks!

Telescope is primarily focused on log visualization, not on log collection or preparing ClickHouse for storage. The system does not currently provide (and I think will not ever) built-in mechanisms for ingesting logs from any sources.

I will consider providing a how-to guide on setting up log storage in ClickHouse, but I’m afraid I won’t be able to cover all possible scenarios. This is a highly specific topic that depends on the infrastructure and needs of each organization.

If you’re looking for a all-in-one solution that can*both collect and visualize logs, you might want to check out https://www.highlight.io or https://signoz.io or other similar projects.

And also, by the way, I’m not trying to create a "Grafana Loki killer" or a "killer" of any other tool. This is just an open source project - I simply want to build a great log viewer without worrying about how to attract users from Grafana Loki or Elastic or any other tool/product.

am interested in the `dummyBytes` generated by flog[2], in the linked benchmark result[1]. it's random words from `/usr/share/dict/words`, which may not be highly compressible..

i.e., 500 GB -> 207 GB (with zstd data compression + indexes) seems like a worst case scenario. with "real" logs, I am expecting this to be much better (for logs at least)..

does anyone have a similar size comparison with real life examples? (similar data size, interested in compressed logs size and indexes size with clickhouse)

[1] https://signoz.io/blog/logs-performance-benchmark
[2] https://github.com/signoz/flog
[3] https://github.com/tjarratt/babble/blob/cbca2a4833c1dd0e0287d50982e6dda554e95c9d/babble_unix.go#L11

SigNoz | https://signoz.io | Remote (US) | Platform Engineer | Full Time

SigNoz is an open source observability platform based natively on OpenTelemetry

Looking for a Platform engineer to join our team at SigNoz. You will be part of the first few hires in our US team and will have the opportunity to own a significant part of the product.

Why us?

- Opportunity to work in a global dev infra product

- Handle Petabyte scale

- Work on an open source product (22K+ github stars). Engage with the community. Build your GitHub profile

- Fully Remote

Detailed JD and application form here - https://jobs.ashbyhq.com/SigNoz/01ebd081-db0c-4eec-8a8b-e346bc3f14a7