The company blog of Lightstep, an observability vendor, covering distributed tracing, spans and monitoring practices.
HN links it in distributed tracing threads, where commenters pair it with Google's Dapper paper. Staff also use it to mention pricing and jobs.
I'm disappointed that all of the open-source tracing systems have adopted the Dapper [0] model. It's understandable why: it's extremely simple to implement, as it handles scaling challenges by doing client-side sampling.
A bit of background about how Dapper-style distributed tracing works. Things typically start with an RPC call of some kind (typically from an external source like a public load balancer). At that point, you must decide whether to trace this request or not, which is typically done as a random sample (say, 1% of requests). At that point, the request gets assigned a _trace id_, a random identifier for that request.
The trace id is stored in some request context and propagated to each subsequent service. Each service, meanwhile, divides up its request processing flow into a series of "spans" which represent some piece of computation. For example, a span cover an RPC call or a DB query. Spans are identified by a random _span id_. Once a request has been sampled, all spans for that request are sent to a central span collector where they're stored for later querying.
This model is simple but very limited. It's often hard to know whether a trace is interesting at the outset, hence the reliance on random sampling. For example, you might want to understand why your 99p latency is high, but if you're just sampling 1% the 99p requests will only be 0.01% of your sample.
More generally, interesting events (like errors or slow requests) tend to be rare, and sampling a…
LightStep | San Francisco or Remote | https://lightstep.com
LightStep builds observability tools for proactively monitoring software in today's microservices-driven world. We write for other engineering teams, and are the only company that develops observability tools at the scale and complexity that enterprises need them. We're especially looking for people-minded engineers who are excited about shipping useful, meaningful products for the developer community.
To learn more about our diverse, thoughtful team; our engineering culture; and whether this is the right place for you, read our Key Values profile: https://www.keyvalues.com/lightstep
You can also check out some recent blog posts, tech talks, and demo:
* Play with our product for free! https://lightstep.com/play/
* "How we write code at LightStep" https://lightstep.com/blog/how-we-write-code-at-lightstep-lab-notebooks/
* "Trace driven development" https://www.youtube.com/watch?v=NU-fTr-udZg&t=4s
* "How GCS processes cloud functions 4x faster than AWS" https://research.lightstep.com/reports/google-cloud-storage
Our open positions (Hiring multiple roles but actively seeking EM):
* All Roles: https://boards.greenhouse.io/lightstep?gh_src=8d0d8b781
* Engineering Manager (SF) - high priority: https://boards.greenhouse.io/lightstep/jobs/924680?t=8d0d8b781
We are not able to sponsor new H1B visas at this time.
Tech Stack: d3.js/React/Redux single page webapp, Go backend
Email Luis[at]lightstep[dot]com for…
I define a special category of distributed system performance bugs. The fact that proper distributed tracing is not standard yet (especially outside of cloud environments) is a real pain here. The google white paper on Dapper [0] and all its implementations [1][2][3] in one form or another are appearing but usually target the Web environments. That means that for my legacy stack with multiple nodes connected via networks of different kinds I have to profile on multiple systems, aggregate different log types manually, think of time syncs (not everything logs utc..) and so on. To be honest we just reached the point where we try to get a European subsidiary project rolling where we address this problem. The time lost to trace down unexplainable time gaps between first displayed result in app and request are weeks, and in multiple occasions months. Just because we have no easy way to profile our distributed ecosystem.
After identifying who is to blame: network, db nodes, processing nodes, application logic, cache (invalidation); we can talk about fixing.
For fixing of course it would be great if you could extract a meta model of the application and another of the hardware. Changes to those models and some nice prediction of impact would be the ideal end goal. Or, if we are wishing anyway, some nice DSE on those models to optimize would be even better.
[0] https://research.google.com/pubs/pub36356.html
[1] http://opentracing.io/
[2] http://lightstep.com/
[3]…
Lightstep | San Francisco or Remote | https://lightstep.com
Lightstep, recently acquired by ServiceNow, builds observability tools for proactively monitoring software in today's microservices-driven world. We write for other engineering teams, and are the only company that develops observability tools at the scale and complexity that enterprises need them. We're especially looking for people-minded engineers who are excited about shipping useful, meaningful products for the developer community.
Check out our latest blog posts: https://lightstep.com/blog
Our open positions:
* All Roles: https://lightstep.com/about/careers
* Software Engineer, Backend (remote friendly)
* Software Engineer, SRE (remote friendly)
*many other roles including TAMs, PM, AE, SE, etc.
We're big believers in growth and that a career path can take many different shapes and forms; with that being said, for remote candidates, we do request prior professional experience in your discipline/field of interest.
Tech Stack: d3.js/React/Redux single page webapp, Go backend
Email swilson[at]lightstep[dot]com for inquiries.
If you're interested in distributed system tracing there's a lot going on.
As a starting point, I would recommend reading Google's paper on their project "Dapper." [1] It's essentially the core of most distributed tracing systems. At least those I've encountered.
There's a lot of tooling out there that take their cues from Dapper. I've recently been looking into integrating OpenZipkin[2] with our systems. I see at as a more viable alternative (no tie in!) to Yet Another Propriety Thing in AWS (YAPTA). There's other as well, like AppDash.
Recently there has been a push towards an open-standard for the collection side called OpenTracing[3]. I came across it when investigating LightStep[4]. Ideally that means no vendor lock-in, which of course has lots of knock-on effects.
If you know of any, I'd love to be pointed in the direction of _different_ and not just divergent techniques.
[1]: http://static.googleusercontent.com/media/research.google.com/en//archive/papers/dapper-2010-1.pdf
[2]: http://zipkin.io
[3]: http://opentracing.io
[4]: http://lightstep.com - Impressive team behind this.
Lightstep has recently changed their pricing, it's now the most cost effective option out there. You pay $100 per service with basically unlimited data ingestion, it's a fabulous deal.
https://lightstep.com/pricing/
As for scrubbing PII, they are now supporting the OpenTelemetry tracing API which does this as standard. For query endpoints you will see something like "QUERY Users where name=? email=?", i.e. masking with "?" chars as you only care about the keys since those what determine your indexing or lack thereof. (This is handled in the OpenTelemetry application library/plugin level.)
As an aside, PII scrubbing should be done that whether or not you own the cluster, because even if you own the cluster you generally don't want your support staff seeing PII esp. as organization grows larger.