Jeff Hodges writes about distributed systems, Go, and data structures like the opposite of a Bloom filter.
Commenters recommend his "Notes on Distributed Systems for Young Bloods" as a really good article that changed how one person saw their job.
> Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box that is assumed to be 100% available. These assumptions about reliablity of systems and their consistent behavior under network partitions has bitten countless people before and will continue to claim victims unless application paradigms are fundamentally rethought.
I'd recommend this... except that it's really, really hard [0]. And if you think about it, you have to do this for every new application, which is a bit wasteful. So IMO, doing it at filesystem level is a good trade off. It's abstract enough to be useful over a wide range of use-cases (unlike block storage), because if you think about it file systems are just frameworks.
To be fair, block can work really well. I think block storage vendors have themselves to blame. Half the "standards" aren't, interop can be terrible. I know I'd rather have simple, reliable storage - stop stuffing "enterprise" features into a rotting, increasingly technical-debt laden codebase. All back-up solutions for block suck - you'd think this problem would be fixed, especially with cloud storage. I could go on.
[0] https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/
The use case in mainly deduplication of a high volume data stream (though it's got a few other uses). The write volume is fairly stable so it's sized in such away that we'll never emit dupes even when the upstream source crashes and needs to be rebuilt from backups (for this case that means > a billion cache entries). Something like the opposite of a bloom filter (https://www.somethingsimilar.com/2012/05/21/the-opposite-of-a-bloom-filter/) didn't work because we don't want false negatives either. Since the cache is fed by a Kafka log HA is achieved simply by having multiple consumers individually populating their own cache instance. The persistence mechanisms are to allow for code deployments that don't blow away the cache, not HA.
We actually experimented with grid caches (ignite in particular since it offers off-heap in memory storage as well), but the performance simply isn't there. At the volume we're writing even millisecond latency is a non-starter. We did explore both memcached and redis, but we need strict FIFO and both of those solutions provide nondeterministic LRU.
Guys, this is real. http://www.somethingsimilar.com/2012/05/21/the-opposite-of-a-bloom-filter/
Yes, it's a hash table without conflict resolution. No, that's not the joke.
From the article:
The simplest data structure that fits the description of a “opposite of a Bloom filter” is a hash map. With a large corpus like the one we were dealing with, the memory consumed by a hash map is far too large and its growth is bound only by corpus size. However, from this simple idea a more fruitful one comes out.
The design I pitched uses an array, a hash function, and the ability to swap ids in and out of the array concurrently. The hash function and the array are used in the same way they are inside of a hash map implementation. Unlike a typical hash map implementation, the array size is bounded and hash collisions are not avoided.
Only one person in this comment thread mentioned CAP theorem, which is the (highly informal*) rationale for NoSql's existence and also its deliberate aversion to join dependencies (which more or less violate the core principles of eventual consistency modeled distributed data stores.)
Useful links for erudition:
https://arxiv.org/pdf/1302.0309.pdf
https://codahale.com/you-cant-sacrifice-partition-tolerance/
https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html
https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/
https://www.microsoft.com/en-us/research/publication/replicated-data-consistency-explained-through-baseball/?from=http%3A%2F%2Fresearch.microsoft.com%2Fpubs%2F157411%2Fconsistencyandbaseballreport.pdf
When Jeff Hodges gave a presentation of his "Notes on Distributed Systems for Youngbloods"[1] at Lookout Mobile Security back in like 2014 or 2015, he did this really interesting aside at the end that changed my perception of my job, and it was basically this. You don't get to avoid "politics" in software, because building is collaborative, and all collaboration is political. You'll only hurt yourself by avoiding leveling up in soft skills.
No matter how correct or elegant your code is or how good your idea is, if you haven't built the relationships or put consideration into the broader social dynamic, you're much less likely to succeed.
[1] https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/
As starter read Mikito Takada (mixu) book "Distributed Systems For Fun and Profit" http://book.mixu.net/distsys/single-page.html
Notes on Distributed Systems for Young Bloods (http://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/)
A Note on Distributed Computing (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.7628)
http://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
You can read additional pointer from Henry blog :http://the-paper-trail.org/blog/distributed-systems-theory-for-the-distributed-systems-engineer/#more-617