Henry Robinson

"That could work completely offline and sync transaction data periodically with the central server."

From what I've gathered you want the app to sync data when possible but still work as it should when offline? There are several solutions in the space of this.

A simple solution would be to split or abstract the logic with handling the data versus the functionality of the mobile app itself. In which case you should not be tied down to specific backends.

Anything that works for mongodb or other databases you can usually find something that converts said backend to something working for postgres or whatever for server side.

As for mobile your data backend should be something lightweight and it may be beneficial to use something that is agnostic to mobile like html5. Key players for web data usually are Indexedb and somewhat basket.js : https://addyosmani.github.io/basket.js/ . As the web is usually the way of using what mobile apps are using for storage gathered from a look at angular.js in retrospect.

Some of the other stuff you mentioned:

A more accurate terminology for what you may be looking for in the future for communicating with several distributed systems would be a CRDT key-value(for example; https://github.com/dominictarr/crdt ) if you want it to be purely distributed over many devices completely and provide the same updates to many devices. There is still a lot of on-going research in the area with Riak and Redis having somewhat successful uses in the box at…

It's a huge field, basically distributed computing, burdened here with the glorious purpose of durable data storage. Any introductory text long enough becomes essentially a university-level computer science course.

RADOS is the underlying storage protocol used by Ceph (https://ceph.com/). Ceph is a distributed POSIX-compliant (very few exceptions) filesystem project that along the way implemented simpler things such as block devices for virtual machines and S3-compatible object storage. Clients send read/write/arbitrary-operation commands to OSDs (the storage servers), which deal internally with consistency, replication, recovery from data loss, and so on. Replication is usually leader and two followers. A write is only acknowledged after the OSD can guarantee that all later reads -- including ones sent to replicas -- will see the write. You can implement a filesystem or network block device on top of that, run a database on it, and not suffer data loss. But every write needs to be communicated to replicas, replica crashes need to be resolved quickly to be able to continue accepting writes (to maintain the strong consistency requirement), and so on.

On the other end of the spectrum, we have Cassandra. Cassandra is roughly a key-value store where the value consists of named cells, think SQL table columns. Concurrent writes to the same cell are resolved by Last Write Wins (LWW) (by timestamp, ties resolved by comparing values). Writes going to different servers act as…

You mean in the asynchronous systems model, in the presence of failures, it is impossible to write a reliable failure detector:
http://the-paper-trail.org/blog/a-brief-tour-of-flp-impossibility/
However, you can write a failure detector for a practical system with the help of omega (weakest failure detector):
http://www.cs.utexas.edu/~lorenzo/corsi/cs380d/papers/p685-chandra.pdf

In practice, failure detectors mean timeouts. What I am arguing is that you can have systems that adapt their timeouts for failure (such as our one), based on the current level of network congestion or load or whatever. The setting/changing of the timeout is averaged over a period of time, to ensure it is high enough to not give false positives. You could do the same approach at the per message level, but for all possible messages, it may be prohibitive. If you have a small number of messages, it may work. The problem with messages, compared to failure detection, however, is the following: how do you figure out you have had a false positive? For failure detection it's easy: the timeout expired, butre i'm still alive and my heartbeat arrives late, so we increase the timeout for that node. Vector clocks (fencing tokens) make it easier to reliably find out if messages arrive very late (identify false positives).

That depends on what level of book, and what you want to learn from it. On "how real distributed systems are built", there isn't a really good book available in my opinion. I liked The Datacenter as a Computer, but it's scope is fairly narrow and is more focused on global infrastructure issues than software design issues. On the math behind distributed systems, Nancy Lynch's book Distributed Algorithms (http://www.amazon.com/Distributed-Algorithms-Kaufmann-Management-Systems/dp/1558603484) is my favorite, but it's getting to be quite out-of-date.

Absent a book, there are a lot of great resources online. High Scalability(http://highscalability.com/) has a good amount of quality links to distributed systems content, and some editorial content of varying quality. In general, it's a good site to follow if you're interested in how distributed systems are being used in industry. I also like to follow Aphyr's blog(http://aphyr.com/), Daniel Abadi's blog(http://dbmsmusings.blogspot.com/), and Henry Robinson's blog(http://the-paper-trail.org/blog/) and Peter Bailis's blog(http://www.bailis.org/).

I have many recommendations of different kinds:

## Blogs:

- http://muratbuffalo.blogspot.com/

- https://bartoszsypytkowski.com/

- https://decentralizedthoughts.github.io/

- https://www.the-paper-trail.org/

- https://blog.acolyer.org/

- https://pathelland.substack.com/

## Other web resources

- https://aws.amazon.com/builders-library/ - set of resources from Amazon about building distributed systems

- https://www.youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiB - lecture series from Cambridge

## Books

- https://www.cl.cam.ac.uk/teaching/1213/PrincComm/mfcn.pdf - A great book on the maths of networking (probability, queuing theory etc...)

Start with understanding "Consensus problem". Paxos is one of the famous solution. There is blog called "The Paper Trail" which has lots of articles on this topic[1]. You should start with the oldest post IMO (from Three General Proble,, FLP impossibility, 2/3 phase locking, to Paxos). Once you have an idea, you can readup the "Paxos made simple"[2].

In distributed systems, each node needs to agree on facts (which could be value of a key, ordering of operations on an object etc...), and Paxos helps you with that.

[1] https://www.the-paper-trail.org/tags/consensus/
[2] https://lamport.azurewebsites.net/pubs/paxos-simple.pdf