Project Atomic

By default, Docker will use the AUFS storage backend if available, and then fall back to devicemapper on loopback.

RHEL, CentOS, and Fedora do not ship the AUFS kernel module because it is not part of the mainline Linux kernel and is unlikely to be included in future, and these distros have an "upstream first, no out-of-tree bits" policy. Instead, they recommend using devicemapper on LVM [1][2].

The same advice is provided in the official Docker documentation [3]:

> Docker hosts running the devicemapper storage driver default to a configuration mode known as loop-lvm... The mode is designed to work out-of-the-box with no additional configuration. However, production deployments should not run under loop-lvm mode... The preferred configuration for production deployments is direct lvm.

You might consider using CentOS Atomic Host, which comes preconfigured with LVM thin pools.

OverlayFS is also an alternative, but it can be problematic. It only implements a subset of the POSIX standard [4], which can cause some programs to fail.

[1] http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/
[2] https://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/version-7/getting-started-with-containers/#managing_storage_with_docker_formatted_containers
[3] https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/
[4] https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/

I want a distributed (not cloud) file system that handles many files and WAN latency (i.e. not HDFS). It might be a cross between Git and BitTorrent.

It feels like deb repos, PyPI, NPM, CPAN, CRAN, etc. should be put in there, with the addition of binaries for popular architectures. And probably Docker-like images, although I think if they are not opaque blobs, it would be better for rsync-like differential compression (which Git implicitly provides).

There will be some small files and some big files. I want it to be like Git so I can clone locally, not just go to the cloud. The way that Git is trivial to set up and clone through SSH is nice too.

It probably has to have a notion of "user", like a local file system. (This makes the problem a lot harder; git doesn't really have permissions.)

BTW Julia's package manager just used git, but I watched a talk that said this ended up being a really bad idea, especially on Windows.

As far as I understand IPFS has some of these properties. Has anybody used it? Could I use it for the package repository use case?

I don't know much about it, but Project Atomic sounds similar too: https://www.projectatomic.io/

Any other projects that seem like a close fit?

BTW I think this would also be useful in the data center, as some companies like Twitter apparently use BitTorrent in data centers to start large jobs quickly (i.e replicate the same 500 MB binary to 1000 machines).

Given that you work for SUSE and I kinda like you guys, I really hope that it's different from SLES licensing model.

I work at a german govt. institution which uses SLES and we've recently decided to move away from SLES for a simple reason. If you want to use something like immutable infrastructure SLES is painful to use, because you need to do everything through the SMT admin.

You won't even know which packages are available in SLES until you signup for it. Sure you can use openSUSE, but it's not the same packages, it's not the same package versions, and the open repo, while nice, is imho not as comfortable to use as ubuntu packages. IMHO it's not very user friendly.

And yes yast2 is great, but I'm not talking about graphical user interfaces when I say user friendly.

By the way, there are a few more "ContainerOSs" than just CoreOS nowadays.

- There's project atomic [1] which became atomic host [2]

- There's RancherOS, which I find interesting because it seems to have first class zfs support [3]

[1]: http://www.projectatomic.io/docs/kubernetes/

[2]: https://www.redhat.com/en/resources/enterprise-linux-atomic-host-datasheet

[3]: http://rancher.com/docs/os/v1.0/en/storage/using-zfs/

This sounds similar to Project Atomic[1]. The base OS is CentOS or Fedora, system files are managed by OSTree[2], and applications are run as Docker containers. You could probably also use something like Flatpak[3] for desktop apps, which is the approach used by Endless OS[4].

What I find interesting is these two projects are using a similar approach but targeting opposite ends of the spectrum. Atomic is intended as a cutting edge server OS for enterprises that want an integrated stack for running containerized infrastructure (they also started publicizing Atomic Workstation as of the last release, although I found it painful to use due to a lack of docs). Endless is targeting emerging markets, composed primarily of first-time computer users. OSTree+Flatpak allows the desktop to run similarly to a mobile OS, and there’s minimal chance of users breaking the system itself.

[1] https://www.projectatomic.io
[2] https://ostree.readthedocs.io
[3] https://flatpak.org
[4] https://endlessos.com/for-developers

Docker is still pretty embedded in a lot of workflows, thanks in part to its use by-default in many Kubernetes distributions, and the popularity of Docker Hub - not to mention various tutorials and scripts which refer to docker tooling.

But yep, I'd agree with the general premise here - with the emergence of tools like cri-o[0], podman and buildah (which let you build and ship container images without the need to run a background daemon like docker at all, avoiding the associated operational/security/system overheads) - docker may need to evolve or it'll quickly become less favourable.

Project Atomic[1] runs a good PPA with many of these packages for anyone interested and using Ubuntu.

[0] https://cri-o.io/

[1] http://www.projectatomic.io/

In Fedora, there is the Fedora Tools container [1], in RHEL, there is the RHEL Tools container. I don't envision either of those going away because they allow you to do all kinds of fancy stuff including things that require kernel compatibility, like SystemTap, core dumps, etc.

I am a product manager for containers/rhel/CoreOS at Red Hat, and I very much foresee a similar container for Red Hat CoreOS which will provide similar functionality for troubleshooting kernel, and user space (aka other containers) problems.

[1]: https://www.projectatomic.io/blog/2015/09/introducing-the-fedora-tools-image-for-fedora-atomic-host/
[2]: https://developers.redhat.com/blog/2015/03/11/introducing-the-rhel-container-for-rhel-atomic-host/