Segment Blog

I helped build an analytics platform that served more than millions of events. Nothing about it is really difficult until you scale heavily.

The tl;dr - It's not worth your time / energy to build from scratch at this scale. Leveraging an existing standard like the analytics.js spec [1] makes web analytics very easy and quick to get started. With this approach you just have to add one JS snippet to your website and never need to update it in your code. If you are interested in the internals, you might enjoy digging deeper into the spec to understand why it was designed this way.

Two services that implement this spec are Segment [2] and MetaRouter [3] [full disclosure: I helped build the product that became MetaRouter at a previous job]. They have different target audiences and pricing models but both are worth a look.

You can think of these types of services as a meta analytics service that routes your events to destination analytics services and data stores of your choice. The great thing about using the standard is you can benefit from all of the many integrations that have already been created with various analytics services, databases, data warehouses, etc [4]. These destination catalogs can also help you decide what services to explore and try next as you need more advanced features.

To get started with a meta analytics service, in the management dashboard, just add your API keys and config values for each service. For a simple service like Google Analytics…

I appreciate and welcome this requirement from Apple.

Some time ago I built something I call “Network Blackhole”.

The project intercepts HTTP traffic from applications that I installed in either my MacBook or iPhone. An excellent example of this is Crashlytics [1], Segment [2], and Sentry [3], which are among a list of popular web services that many developers use to report bugs and crashes in their software, and the famous Google Analytics, which I hope I don’t need to explain what it is used for.

With the help of Little Snitch [4] a popular network monitor for macOS, I detect when an app tries to connect to one of these services, or similar. Then I execute a tool written in Go like so: “blackhole example.com” which does the following:

1. Inserts domain into /etc/hosts

2. Create an HTTP web server (in Go)

3. Adds a match-all endpoint to the server

4. Creates an SSL certificate with mkcert [5]

5. Creates an Nginx virtual-host for the server

6. ???

7. Profit

In the end, and after 1-2 minutes, I have all the traffic to that domain gracefully redirected to a black hole, reducing the amount of data that I leak to 3rd-party websites.

However, don’t get me wrong, I understand the purpose of these services, I haven’t said they are evil or anything like that. I would probably use them myself if I had to, but I certainly would add an alert to ask for explicit consent from the user to send this information to a service that I won’t even have control over. If one of them…

What we've found is that different analytics services help us answer different kinds of questions.

For example, here are the tools we use to find each metric he mentions:

Distribution

1. New users by channel: Google Analytics Source breakdown satisfies what I need to know here. If you want to look at conversion to a Goal broken down by source, there's the "Goals" view in GA.

2. Total users: easiest for us here is to just query our database.

3. Cost of customer acquisition, lifetime value, pay back period: again Google Analytics if you're doing adwords advertising.

Engagement

1. Active users: here a tool like KISSmetrics, Keen IO or Mixpanel will let you see how many people are really succeeding in using your service.

2. % of users using top 3 key features in a given month: same tools as active

Revenue

1. Revenue growth: Stripe or w/e your payments solution is.

2. Conversion to paid: KISSmetrics and Mixpanel are the best tools to help you measure conversion rates.

3. Avg spend per paying customer: Stripe or w/e your payments solution is.

4. Churn rate: Mixpanel probably does this best, but querying your database might be a better option depending on how you measure churn.

5. Burn rate: your bank account!

What we've found is that no single tool answers all these questions well. Most companies we talk to use a mix of Google Analytics for basic stats and distribution measurement. KISSmetrics, Mixpanel or Keen IO for engagement metrics and tracking. Customer.io,…

Segment.io - San Francisco, CA

Mobile Engineer

We're looking for an extremely good mobile engineer to help us change the way people approach mobile analytics.
This is a perfect position for someone who loves building clean APIs for others to use. You enjoy not only building mobile apps, but writing the libraries that power those apps. You're fluent in everything to do with iOS as a platform, and if you have Android chops that's a huge bonus, but definitely not a requirement.

You'd be working on current and new features of the mobile SDKs themselves, as well as helping to write code for where they touch our Node.js backend, where we process tens of millions of events per day. And you'd be in charge of our entire mobile offering, which is used in product by lots of big companies like Live Nation, Nest Labs and The Knot.

You'd be joining a team of very talented engineers that are in love with open-source. Both of those SDKs are open-source, as is tons of our codebase[1]. So if you enjoy working with smart people and getting to make a name for yourself releasing the things you work on to other developers, you've found the right place.

Not only that, but you'd be working on a few of our still-secret mobile projects that we're pretty confident are going to change the way people integrate third party code into every mobile app they write afterwards.

[0]: https://segment.io

[1]: https://github.com/segmentio

---

To apply, email us at jobs@segment.io with:

1. a…

Sure but how about Pixel. Or Google Analytics, Clicky, Alexa (analytics not the talking thing), Adroll , ... I mean the amount of trackers I can toss on a website is mind boggling and super easy, all hoovering as much data as possible. This is not exclusive to Facebook nor do I think they gather the most data.

After sorting through this guys first half dozen tweets I finally realized he's talking about AI and advertising. As an ecommerceguy here's my take. I'd love to be able to upload a product feed, have FB or whomever evaluate those products and push them to whomever the adbot/algorithm/ai/hall9k whatever we're calling it today; as long as it returns good ROI, happy customers and less work for me I'm happy.

This is already somewhat possible with Google Shopping Feed.

So again, besides this helped Trump, why the outrage? All I see is people freaking out at what has been public knowledge for years albeit obfuscated under a massive sheen of PRSpeak.
FWIW I have always actively stayed away from Facebook as much as possible to the point I tell my sister to take photos off her feed of me. But here I am tangentially defending them.

I wanted to add a link to this site, It connects to over 200 trackers.
https://segment.com/

Segment's blog posts on cost optimisation have plenty of detail and tips on this topic:

https://segment.com/blog/the-million-dollar-eng-problem/
https://segment.com/blog/spotting-a-million-dollars-in-your-aws-account/
https://segment.com/blog/the-10m-engineering-problem/

Similarly this Honeycomb writeup is also excellent:
https://www.honeycomb.io/blog/treading-in-haunted-graveyards/

By the sounds of it, you need to take drastic action. It sounds like you will not be able to just optimise your AWS spend to get more runway, though you should definitely do some bill optimisation. You will need to optimise your product itself and maybe even getting rid of unprofitable customers.

If you are not sure exactly who or what is driving the AWS cost, take a look at Honeycomb to get the ability to dive deep into what is eating up resources.