Buildkite, a CI/CD company, blogs about pipelines, Bazel, SRE practices, plan changes, and product features.
HN commenters recommend Buildkite warmly, praising its UX and the model where tests run on your own infrastructure, with several using it for years.
I love buildkite and I wish they had better pricing model -- we would definitely love to pay for more.
Pros of buildkite:
- Very powerful. Plugin system is amazing and easy to extend. We wrote our own buildkite-docker executor, and I merged a lot of parts from it to the docker plugin upstream
- UI is a lot prettier and responsive than Gitlab's
- Dynamic pipelines !
- Hooks are extensive and powerful. Precheckout, Checkout, PostCheckout, precommand, command, postcommand. You got it all.
- The GraphQL API is amazing, and probably one of the best API I've used in my life.
- Pipelines are composable. You can make a yaml include another yaml. This was our biggest gripe with Gitlab since we use a monorepo and it got pretty big
- I know for a lot of people it doesn't mean a lot, but I like emojis in the output
- Agents are not stored like Gitlab. In Gitlab, once an agent connects, it gets stored in the database and never disappear until you delete it. It's problematic for our use case, because we use pre-emptibles node to save cost. It's 30% of the real price, and it dies every 24 hours. after 3 months, our Gitlab got super slow because it retained all the infos on all runners that existed. I had to make a script that deleted it from the database.
- Job retries!
- Docs are pretty dope
- Annotate is very cool (https://buildkite.com/docs/agent/v3/cli-annotate)
- Block step. Useful for confirming before deploy or something.
- Pretty cheap (unlimited builds, you pay per…
Sure! Here goes:
The App itself is boring old rails, with postgres (get me started on how much I like postgres and I will literally bore you to tears.) The original views were all haml and sass delivered via the asset pipeline. I sprinkled in angular here or there as the application grew.
Today, I'm slowly tearing away the angular. I've moved to webpack for JavaScript, and React for all new UIs. I've been really enjoying both the speed of development, and how solid everything feels.
Over time, I'm writing less and less ActiveRecord, and more and more sequel… still learning, but getting better.
The app is run on Heroku, and its been A++ would recommend to everyone.
The Calibre agent is node and websockets talking to Chrome®, theres other bits of ruby, and even some golang now.
I use terraform and ansible (although, the ansible may as well have been bash scripts) through a buildkite (https://buildkite.com — the best fucking CI service on the planet) pipeline. AMIs are built, shipped to different regions, then based on some config are rolled out.
Papertrail for logs - omg, logging! Essential. I run thousands of tests per day (I don't know this number off the top of my head, but its a lot)… and the nature of the internet is THERE IS ALWAYS FAILURE. Having logging at my fingertips has always made my job building Calibre easier. I think I pay $20 per month right now, and its the most important thing I pay for.
--
Ok. Whew. Could probably keep going, but I'm not sure how…
The primary difference is that in Nix it's fairly easy to wrap arbitrary build systems, so you can inject third-party dependencies into your codebase without vendoring and adapting their build system (plus, through nixpkgs[0], the majority of all relevant software has already been wrapped).
In Bazel you have to do a lot of that work yourself. Google has essentially ~unlimited manpower to do this for the things in their third_party tree, but for any other size of organisation this is not the case and people resort to all sorts of ugly hacks.
Depending on your needs these different types of projects can also coexist with each other in Nix. To use an example from my work, we have a Nix-native build system for Go[1] and code using this[2] co-exists with code that just uses the standard Go build system[3]. Both methods end up being addressable on the same abstraction level, meaning that they both turn into equivalent build targets (you can see both in a full build[4] of our repo).
And for what it's worth, some of the things Bazel gets extremely right (such as having a straightforward mapping from code location to build target, universal build command) are pretty easy to do in Nix (see readTree[5], magrathea[6]).
[0]: https://github.com/NixOS/nixpkgs
[1]: https://code.tvl.fyi/about/nix/buildGo
[2]: https://cs.tvl.fyi/depot/-/blob/ops/besadii/default.nix
[3]: https://cs.tvl.fyi/depot/-/blob/third_party/gerrit-queue/default.nix
[4]:…
I know this is off topic, but that homepage is a piece of work: https://buildkite.com
I get it's quirky, but I'm at a low energy state and just wanted to know what it does...
Right before I churned out, I happened to click "[E] Exit to classic Buildkite" and get sent to their original homepage:
https://buildkite.com/platform/
It just tells you what it Buildkite does! Sure it looks default B2B SaaS, but more importantly it's clear. "The fastest CI platform" instead of some LinkedIn-slop manifesto.
If I want to know why it's fast, I scroll down and learn it scales to lots of build agents and has unlimited parallelism!
And if I wonder if it plays nice with my stack, I scroll and there's logos for a bunch of well known testing frameworks!
And if I want to know if this isn't v0.0001 pre-alpha software by a pre-seed company spending runway on science-fair home pages, this one has social proof that isn't buried in a pseudo-intellectual rant!
-
I went down the rabbit hole of what lead to this and it's... interesting to say the least.
https://medium.com/design-bootcamp/nothing-works-until-you-make-it-work-d48a2148bf60
https://www.reddit.com/r/branding/comments/1pi6b8g/nothing_works_until_you_make_it_work/
https://www.reddit.com/r/devops/comments/1petsis/comment/nsm3c59/
A few issues I have with this blog post:
1. It doesn't show off the unique capabilities of firecracker very well.
2. The comparison not very fair.
2a. The docker-build step (which dominates the runtime) is run without any caching, just by adding 2 lines to your build-push-action, "cache-from: type=gha, cache-to: type=gha,mode=max" you can make it a lot faster.
2b. ~1m20s of the time is just "VM start". GitHub Actions has had a rough time recently, but you should never wait that long to get your CI running in day-to-day operation.
2c. The tests are unrealistically short at 20s which allows the author to get to their 10x faster number.
Let's say the GitHub Action starts in 5 seconds, the GitHub Actions cache reduces the build time to 2 minutes and the tests take 10 minutes to run. Now Firecracker is 20% faster ...
You can also get comparable performance out of https://buildkite.com/ which lets you self-host runners on AWS meaning you're almost guaranteed to get a hot docker cache (running against locally attached SSDs). You can now start running your tests (almost) as fast with much more mature tooling.
>Lack of local development. It's a known thing that there is no way of running GitHub Actions locally.
This is one thing I really love about Buildkite[0] -- being able to run the agent locally. (Full disclosure: I also work for Buildkite.) The Buildkite agent runs as a normal process too (rather than as a Docker container), which makes the process of workflow development way simpler, IMO. I also keep a handful of agents running locally on my laptop for personal projects, which is nice. (Why run those processes on someone else's infra if I don't have to?)
>Reusability and YAML
This is another one that I believe is unique to Buildkite and that I also find super useful. You can write your workflows in YAML of course -- but you can also write them in your language of choice, and then serialize to YAML or JSON when you're ready to start the run (or even add onto the run as it's running if you need to). This lets you encapsulate and reuse (and test, etc.) workflow logic as you need. We have many large customers that do some amazing things with this capability.
[0]: https://buildkite.com/docs/agent/v3