Buf Blog

ConnectRPC works over the web and I've built several web apps (SPAs) with it. It works with JSON by just setting the Content-Type header to `application/json`. You can add compression easily on the backend with a single option. Generating typed stub code is one line (`buf generate`). Connect-ES is awesome - it's a Typescript-first protobuf library for the web.

Interacting with the API is very simple, for example:

const resp = await gameEventClient.getRecentAnnotatedGames({ limit: annotatedPageSize, offset: recentAnnotatedGamesOffset, }); setRecentAnnotatedGames(resp.games); These function calls are typed and a lot of the code is auto-generated from the .proto spec (i.e. above the gameEventClient and getRecentAnnotatedGames are auto-generated code). On the server side the code is also obviously auto-generated. It all works seamlessly. Even the documentation for the API can be auto-generated. See for example:

https://buf.build/domino14/liwords

The above documentation was auto-generated from my protobuf files. This project uses ConnectRPC on the backend and on the front-end SPA. To me it seems so much simpler and better than the way I've seen people use OpenAPI - where many people seem to create the code _before_ creating the spec. I actually haven't found a good Go generator of stub code from an OpenAPI spec. With ConnectRPC it just works, it's simple, easy, fast, etc. It's easy to add interceptors to do things like parse the…

This is an absolutely wonderful article. I helped build a system very similar to the one quoted below:

> A deploy pipeline that queries your orchestrator for running image tags, checks your migration history against the schema registry, diffs your GraphQL schema against collected client operations, and runs Buf’s compatibility checks: this is buildable today, with off-the-shelf parts.

that was largely successful at eliminating interservice compatibility errors, but it felt like we were scrambling around in a dark and dusty corner of software engineering that not many people really cared about. Good to see that there's a body of academic work that I was not entirely aware about that is looking into this.

The article successfully identifies that, yes, there are ways, with sufficient effort, to statically verify that a schema change is (mostly) safe or unsafe before deployment. But even with that determination, a lot of IDLs make it still very hard to evolve types! The compatibility checker will successfully identify that strengthening a type from `optional` to `required` isn't backwards compatible: _now what_. There isn't a safe pathway for schema evolution, and you need to reach for something like Cambria (https://www.inkandswitch.com/cambria/, cited in the article) to handle the evolution.

I've only seen one IDL try to do this properly by baking evolution semantics directly into its type model, typical (https://github.com/stepchowfun/typical) with it's `asymmetric` type…

> Open-sourcing protobuf was most likely not a big cost, and it had benefits for Google (otherwise it would not have been rational to open source it).

Nah honestly it was more like I said "Hey I want to open source this!" and other engineers said "Yeah that sounds awesome!" so I did. That was Google in ~2007, you could basically do whatever you wanted and as long as it was vaguely plausibly useful management said "sure".

I'm honestly not sure the company had any idea what they should do with OSS protobuf, at least for many years. Management never really gave me any credit for releasing it and gently pressured me to find a different project to work on that more directly aligned with company goals, which I eventually did.

Still not clear if they even know today. They seem more serious about it today but at the same time it's still clear that they mostly don't care about external users, so much so that startups like https://buf.build have to come in and build the actual products around Protobuf that Google never delivered publicly (but has internally). Of course, companies are not hive minds -- there are clearly people at Google who want to make Protobuf a great product and are trying to do so, but I don't think management sees it as strategically interesting, so they face headwinds.

> If protobuf was not open source, something like Cap'n Proto may take its place, and that's not good for Google.

Honestly I don't know if Google cares. Cap'n Proto hasn't caught on much…

I'm not too surprised. Most people for whom TypeScript is an option won't have been exposed to generations of RPCs that fail to deliver simplicity. So it isn't going to be on their radar.

I've been using gRPC and REST'ish+JSON APIs for years now and what I find puzzling is that REST+JSON tends to mean a lot more work, less pleasing code than gRPC, and yet people prefer it. Not because it leads to simpler, better, faster less error prone code (it doesn't. Quite the opposite), but, I think, because people feel they can understand it.

The people over at https://buf.build have been doing a great job trying to tame gRPC btw. The protoc toolchain from Google has been uniquely horrible to work with, but the 'buf' tool from said company has been a real life-saver. Not to mention their Buf Schema Registry, which admittedly I have only used on a few projects so far, but should migrate more projects to.

Though in general, I feel that RPC mechanisms that are too closely tied to a given language are a waste of time. But that's me. tRPC isn't something I'd get into even if I was doing server side TS. It just doesn't seem like a good long term choice.

Buf CLI itself is licensed under a permissive Apache 2.0 License [0]. Since Buf is a compiler, its output cannot be copyrighted (similar to proprietary or GPL licensed compilers). DISCLAIMER: I am not a lawyer.

Buf distinguishes a few types of plugins: the most important being local and remote. Local plugins are executables installed on your own machine, and Buf places no restrictions on use of those. Remote plugins are hosted on BSR (Buf Schema Registry) servers [1], which are rate limited. All remote plugins are also available as local plugins if you install them.

It's worth to mention that the only time I've personally hit the rate limits of remote plugins is when I misconfigured makefile dependencies to run buf on every change of my code, instead of every change of proto definitions. So, for most development purposes, even remote plugins should be fine.

Additionally, BSR also offers hosting of user proto schemas and plugins, and this is where pricing comes in [2].

[0] https://github.com/bufbuild/buf/blob/main/LICENSE

[1] https://buf.build/blog/remote-plugin-execution

[2] https://buf.build/pricing

I've been having fun with connectrpc https://connectrpc.com/

It fixes a lot of the problematic stuff with grpc and I'm excited for webtransport to finally be accepted by safari so connectrpc can develop better streaming.

I initially thought https://buf.build was overkill, but the killer feature was being able to import 3rd party proto files without having to download them individually:

deps: - buf.build/landeed/protopatch - buf.build/googleapis/googleapis The automatic SDK creation is also huge. I was going to grab a screenshot praising it auto-generating SDKs for ~9 languages, but it looks like they updated in the past day or two and now I count 16 languages, plus OpenAPI and some other new stuff.

Edit: I too was swayed by false promises of gRPC streaming. This document exactly mirrored my experiences https://connectrpc.com/docs/go/streaming/