Kubernetes Blog

I don't think there's a direct quote of k8s devs saying "this is designed for stateless applications" (although there very well may be), but it's pretty evident if you're following.

Blog post on the official site, about 9 months old, whose title expresses surprise that k8s may be reasonable for stateful applications. The title is "Stateful Applications in Containers!? Kubernetes 1.3 Says “Yes!”" [0]

k8s's "StatefulSets", which emerged in 1.5 (4 months ago, Dec 2016) and were transitioned from the previous feature "PetSets", are still marked a "beta" feature in the newest release (1.6, made a couple of weeks ago). [1]

In Nov. 2016, CoreOS issued a blog post with the subheading "Stateless is Easy, Stateful is Hard" while introducing a new pattern called "operators". Operators are intended to help Kubernetes better handle stateful applications (since the habit of k8s admins is to terminate pods casually). These are complicated to implement and haven't been widely adopted; basically, you register a third-party resource type, tell your operator to listen in for pod deletions, and hijacks these commands to ensure that everything is shut down in the correct order and with ample opportunity to save. [2]

-----

This is all just the k8s level -- let's not forget that the container engine underlying k8s will delete all changes made to the image by default, and has similar weird issues with stateful applications. Accidentally stop the wrong container on the docker cli? You have…

I've been a big fan of ipvlan since it was added to the kernel[1] mainly for the reasons you've outlined and the overall reduction in complexity compared to bridges and overlay networks. It should also be noted that ipvlan offers a bit of an improvement over macvlan because ipvlan also works in places where L2 is not an option (i.e cloud providers).

In the post you mentioned one of the drawbacks to adopting ipvlan was the lack of tooling to manage it. To address this issue, and a few others[2], Kubernetes has recently adopted the Container Network Interface(CNI)[3] standard as the solution for managing network plugins in Kubernetes. CNI ships with a few plugins including ipvlan and, maybe more importantly, the ability to allocate and manage IP addresses[4]. Over time CNI should give Kubernetes the flexibility to work well with multiple container runtimes including Docker and rkt, while allowing third party solutions to fill in the gaps around networking.

[1] https://www.kernel.org/doc/Documentation/networking/ipvlan.txt

[2] http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-libnetwork.html

[3] https://github.com/appc/cni/blob/master/SPEC.md

[4] https://github.com/appc/cni/blob/master/SPEC.md#ip-allocation

DC/OS and k8s have some overlaps, but I wouldn't characterize the two as direct competitors.

Majority of DC/OS users are running distributed data services like Spark, Kafka, Elastic, Cassandra, etc. And they're doing this for at least three reasons 1) Pooling these distributed systems instead of running them in separate clusters, 2) The automation for scaling and high availability that Mesos frameworks enable, and 3) Being able to run these data services in any cloud they choose - Google, Azure, AWS, or their own. The 2016 Mesos Survey (n=500) has more details.

If you are squarely focused on container orchestration, 62% of DC/OS and Mesos community runs containers in production. Average size of their environments is 200+ nodes. A similar survey of the K8s community showed rough average K8s environment is around 20 nodes.

Disclosure: I work at Mesosphere

https://mesosphere.com/blog/2016/11/02/apache-mesos-survey-2016/

http://blog.kubernetes.io/2017/01/kubernetes-ux-survey-infographic.html

It's great that hftguy thinks Google Container Engine is stable (I work on it) but I'm sorry to say it's very easy to prove that is, in fact, running Docker on the nodes.

You can just SSH into one and see for yourself.

Kubernetes was built from the ground up to orchestrate Docker. CoreOS did a lot of work to make it possible to trade rkt in for Docker's engine, and the cri (Container Runtime Interface) is now generalising that so that there is a clear abstraction between the kubelet and the engines it orchestrates. Read about it here: http://blog.kubernetes.io/2016/12/container-runtime-interface-cri-in-kubernetes.html

If you want to do things that are different to what we provide support for on GKE as a Managed Service (tm), you're able to run your own Kubernetes clusters on GCE. (We do let you run a Kubernetes alpha version, but only on non-supported clusters that self-destruct after 30 days.

As the proverb goes, "Choose, or someone will choose for you."

rkt is getting into Kubernetes as an alternative runtime engine:

http://thenewstack.io/kubernetes-chief-we-back-docker-although-appc-might-have-merit/ (2015)

http://blog.kubernetes.io/2016/07/rktnetes-brings-rkt-container-engine-to-Kubernetes.html (2016)

Back then it was, "It's sad to see so much drama; Docker engine has much more traction." Remember all of that?

The story is now, "hey, rkt lets us test out our framework for supporting alternative runtime engine".

There's a shifting attitude there. Goodwill from the community is eroding. I don't know how long this window will remain open, but it won't remain open indefinitely.

>Docker should have just licensed and supported the Docker engine to the mass market (their one moat)

I give them a lot of credit for noticing that nobody else was capitalizing on the existing container tech. But it did exist already. They didn't invent something new. They just put a nice cli, daemon, and image repository around it. I think they know the "license the engine" bit is a non-starter, as the higher level container orchestration platforms are already abstracting away from it.[1]

[1]http://blog.kubernetes.io/2016/12/container-runtime-interface-cri-in-kubernetes.html