Boundary Blog

A topic I rarely see (at least in depth, or in discussion) in stories/blogs about APIs is ID generation, i.e. best practices for creating access_token, resource ID and especially handling collisions.

The Flickr blog post [1], which featured a simple implementation of a ticket server, is so far one of the easiest and most secure to use.

Personally I don't feel comfortable using "hash of i++", "hash of ...", "i++", as they all "fall apart" when you need ids of different specifications (e.g. a 8-char ID, a 24-char token ID).

Anyone have two cents on this? I have a difficult time imagining that larger APIs just "live with" the chance of collision (even though it can be very low) - they must mitigate it somehow, right? combined probability (such as the nonce check and tokens on oauth1 requests)?

Actual ticket servers (e.g. [2], [3]) either introduce added complexity (and possibly latency) or needs a secondary code base. Regardless, the issue of scaling and latency is rarely touched upon, so if anyone have some input here, I'd greatly appreciate it.

EDIT: I've found two interesting links on the problem, [4] [5] (snippit'ed here [6]).

[1] http://code.flickr.net/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap/

[2] https://github.com/twitter/snowflake

[3] https://github.com/boundary/flake

[4] http://boundary.com/blog/2012/01/12/flake-a-decentralized-k-ordered-unique-id-generator-in-erlang/

[5] https://blog.twitter.com/2010/announcing-snowflake

[6]…

I was actually looking at a bunch of open source time series databases and settled on kairosdb but this looks pretty nice.

I think there is a hackernews rule someplace that a more interesting tech alternative shows up right when you decided to go with something else.

For reference here is the list I created when researching these:

http://opentsdb.net/overview.html
Built on HBASE

http://www.gocircuit.org/vena.html
Built on go + go'circult uses google's leveldb

https://code.google.com/p/kairosdb/
A rewrite of opentsdb which can use Cassandra

http://blueflood.io/
Built by Rackspace, decent but still seems a bit immature

http://graphite.wikidot.com/
Obligatory Graphite reference
(uses whisper, new backend called 'ceres' is being developed)

https://github.com/agoragames/kairos (yet another 'kairos', alternative backends for graphite - SQL, redis, or mongo)

Riak seems to be popular with SASS metric providers (hosted graphite, boundary). There isn’t any code but there are a couple of talks that explain how and why they went with Riak:

http://basho.com/hosted-graphite-uses-riak-to-store-all-customer-metrics/
http://boundary.com/blog/tag/tech-talks/

http://vimeo.com/42902962

Probably downvoted for not providing a link to the benchmarks that show Java and Scala out-performing Go 1.0?

I know of Collin VanDyck's web service showdown (http://boundary.com/blog/2012/09/17/comparing-go-and-java-part-2-performance/) which showed that Java had lower latency and higher throughput for highly concurrent tasks based on a very simple code base, but net/http performance in Go 1.0 wasn't a high priority. It seems that it has since been bumped up the priority list and a lot of work has been going on (https://groups.google.com/d/topic/golang-nuts/zeLMYnjO_JA/discussion).

The thing is, Go isn't just for writing web apps. If you have a look at some of the real world use cases for Go (there's a list in another comment here somewhere) you'll notice that these aren't just web apps.

Every bit of Go code I've written (web app, mp3 database generator, encryption library) has been blisteringly fast, and it seems Go 1.1 will bring substantial performance improvements again.

Oh, and I didn't down-vote you, I don't yet have that privilege.

(it's hard to install new gigabit fiber pipes to siphon
off the data without anyone noticing)

If you have access to manufacturers that can put in back doors for you, I reckon you don't even need your mole to install stuff for you. Instead you just ask your mole to inform you want is going to be installed and then make sure that the company gets backdoored systems when hardware is installed/upgraded/replaced.

I'm wondering is datacenter monitoring utilities like the stuff Boundary[0] is working on could be used to identify anomalies in how network hardware is behaving versus how it should be behaving. I know that in my conversations with cliff, they are trying to get their monitoring solution to the point where they can visualize "the circulatory system" of a data system with the goal of spotting things that don't look quite right.

[0] http://boundary.com/

We use a lot of services at DNSimple. Here are some of them:

Chargify (http://chargify.com/) - recurring billing

Stripe (http://stripe.com/) - payment processing

Librato (https://metrics.librato.com/) - metrics

Mailgun (http://mailgun.com/) - email service

Boundary (http://boundary.com/) - network analysis

Papertrail (http://papertrailapp.com/) - consolidated logging

PagerDuty (http://pagerduty.com/) - alerting

TravisCI Pro (http://travisci.com/) - continuous integration

CodeClimate (http://codeclimate.com) - static code analysis

BugSnag (https://bugsnag.com) - exception tracking

Also, other services that have already been mentioned (like Github, Intercom, Trello, Dropbox, Sendgrid, NewRelic, etc)

Very nice. This is based on Boundary's Erlang Flake ID service, more details of which are at http://boundary.com/blog/2012/01/12/flake-a-decentralized-k-ordered-unique-id-generator-in-erlang/.

One thing I had to look up again was the definition of k-ordering: K-ordering gives roughly time-ordered id's down to the millisecond when sorted lexicographically. This is usually fine for most purposes but keep in mind it doesn't allow you to reason about causality between timestamps from different machines.

This also relies on your servers clocks to be in sync (something that should be happening anyway).