Airbyte Blog

Hi there Michael! We continue to love nomad.

SQL is the C ABI of querying for sure. BI tools will never adapt to use Convex directly, and nor should they.

So... yes, Convex actually had a prototype SQL adapter for the read side of things back in the early few months when we were figuring things out. Convex's read semantics are very compatible with SQL.

We've kept this adapter on ice in part because of point #3 in the article -- we don't want to overpromise things which are a bad idea.

Meaning, if we exposed SQL on the thing as-is, this would presumably be for more analytical type queries involving patterns normal Convex queries can't express. Right now that would be a Bad Idea because your website would slow down just like every other database system allows you to.

So the current recommended practice is use our Airbyte Egress connector (https://airbyte.com/connectors/convex) and get yourself into an offline Clickhouse/MySQL/Snowflake whatever and jam SQL over there to your heart's content. That's basically what we do.

We may one day abstract this away by embedding some sort of mirrored column store sql thing (maybe DuckDB based? who knows) so you can do your analytical work without impact on your OLTP or integrating 3 more systems. But the team hasn't invested in that yet and probably won't for some time.

I am one of Airbyte's founder. The initial version of was actually fully based on Singer and this is when we realize it wouldn't be possible for us to depend on it.

Amongst the main reasons: Singer seems to have been abandoned by StitchData (after they got acquired by Talend), the quality of the connectors is too unpredictable, Singer connectors are not usable outside the box.

We would have preferred to use an existing standard if one already existed. It was a tough decision for us to create something from scratch but now we are very satisfied with the decision. It is way easier for the community and for us to build connectors that meet quality standards and we can make it MIT so the community can have control on the evolution of the protocol.

We actually wrote a few articles about it:

https://docs.airbyte.io/faq/differences-with.../singer-vs-airbyte

https://airbyte.io/articles/data-engineering-thoughts/airbyte-vs-singer-why-airbyte-is-not-built-on-top-of-singer/

Checkout Airbyte and singer.io for data movement. They're tailored for moving offsite data local, but also have support for moving data between different database vendors. I'd recommend starting with Airbyte.

For transformations I'm a sucker for dbt-core, which executes transformations as SQL against your data warehouse vs. a separate process. This keeps everything simple and (generally) snappy. It uses the Python Jinja2 template library to compose SQL templates, and the process that parses templates to SQL also implicitly creates an execution DAG so you could have hundreds of transformations running in the exact order they need to.

There are also many commercial offerings (Fivetran, Dataform) that don't tick self-hosted, but could be easier to deploy if getting internal hardware support is an issue.

https://github.com/airbytehq/airbyte

https://airbyte.com/connectors

https://www.singer.io/#taps

https://github.com/dbt-labs/dbt-core

Sure! Those are actually great questions!

Regarding monetization, you can see more details here: https://airbyte.io/pricing
We consider 2 monetization approaches:
Open core (connectors staying open-source forever) with premium features as: hosting & management (cloud-based control panel without access to your data plan), and enterprise features (privacy compliance, SSO, user access management, etc)
What we call “Powered by Airbyte” where we enable you to offer integrations to your own customers using our API

Regarding our runway, with the team as is, mid 2025. We intend to grow the team though, given the adoption growth we have. We’ve already been approached for a Series-A, but will consider it in mid 2022.

Regarding self-sustainability, do you mean financially? Possibly at the end of 2023 or 2024.

How does that sound to you? Genuinely curious.

Airbyte | Engineering & Product | Remote | Full-Time | https://airbyte.io/

Airbyte is the new open-source data integration platform that runs in the safety of your cloud and syncs data from applications, APIs & databases to data warehouses, lakes and other destinations. In addition to offering a quickly increasing list of maintenance-free connectors, it makes building your own connectors trivial. Airbyte’s ambition is to make data integrations a commodity.

Airbyte has multiple engineering and product openings across Platform & Connectors. We keep a very transparent handbook on culture, our roadmap, and much more! https://handbook.airbyte.io/

Jobs: https://jobs.lever.co/airbyte

I saw this project a few months ago and I'm really surprised how fast they've been adding new connectors [1]. I was initially a little doubtful about them implementing their own protocol [2] instead of using Singer [3], already OOS, but it looks like it panned out for them. In fact I've been using Stitch Data for a couple of years and I've find myself "fighting" with their support team about bugs I think we're Singer related, mainly with DDL commands.

[1] https://docs.airbyte.io/changelog/connectors
[2] https://airbyte.io/articles/data-engineering-thoughts/why-you-should-not-build-your-data-pipeline-on-top-of-singer/
[3] https://www.singer.io/