Company blog for Meltano, an open source ELT and data pipeline platform that spun out of GitLab.
HN commenters mention Meltano as an open source ETL tool and DataOps platform, often in job posts and data tooling discussions.
Meltano | All positions are full time and remote | https://meltano.com (follow the code blocks on our website for a demo!)
We're a seed stage startup that recently spun out of GitLab. A bit about us:
Meltano is an open source DataOps platform that brings together best-in-class open source tools and technologies for the data lifecycle, including the Singer standard for data integration, dbt for transformation, Airflow for orchestration, and soon Superset for visualization. It simplifies configuration, deployment, and monitoring, and lets data teams benefit from DevOps best practices such as version control, code review, and CI/CD.
Stack: Meltano and the Meltano SDK are written in Python. Vue.js (as part of VuePress) and Bulma CSS are used on the front end.
Open Roles:
- Backend Engineer
- Senior Backend Engineer
- Senior Frontend Engineer
- Head of Partnerships
- Content Marketing Manager
- DataOps Evangelist
- Senior Support Engineer
- Technical Marketing Manager
- Senior UI/UX Designer
Apply online at https://meltano.com/jobs. If you have questions please reach out to us on Slack: https://meltano.com/slack
https://meltano.com/ right now says:
"ELT for the DataOps era
Meltano is open source, self-hosted, CLI-first, debuggable, and extensible."
That doesn't tell me what Meltano is.
I happen to recognise "ELT" as "Extract, load, transform" but I imagine a lot of people would have to look up the acronym.
I hadn't seen the term DataOps before, despite working in this area. DevOps for Data? But the definition of DevOps itself is still pretty vague!
I know how hard it is to explain what a product like this does - I'm definitely not there with my own https://datasette.io/ project yet - but what you've got right now could definitely do with some improvement.
Here's my attempt for Meltano, based on a very loose understanding of what it is:
"Meltano is an open-source tool for building automated pipelines for extracting and converting data. Pipelines are defined using version-controlled code, and the framework can be extended using plugins written in Python."
This really isn't very good, but it does fill in the questions that I had when I first visited https://meltano.com/
Meltano | Full-Time | Remote | https://meltano.com
Meltano's all-remote team and community of thousands are on a mission to enable everyone to realize the full potential of their data. To this end, we are bringing software engineering best practices to data teams in the form of an open-source DataOps platform that we envision becoming the foundation of every team's ideal data stack.
Our public company handbook (https://handbook.meltano.com/) has all the details on our values, all-remote work, where we hire, compensation, benefits, and what it's like to work with us in general.
Current Open Roles:
Senior Software Engineer - Core Products
Senior Software Engineer - Full Stack
Stack: Python, React, Next.js, AWS
Apply online at https://meltano.com/jobs. If you have questions, please reach out to us on Slack: https://meltano.com/slack
I gave a positive review of dbt's testing capabilities in this other comment: https://news.ycombinator.com/item?id=36720013
This is what the actual generic tests look like. There's also support for one-off SQL-based tests: https://docs.getdbt.com/reference/resource-properties/tests#out-of-the-box-tests
This article covers a lot of ground as to how to actually use dbt tests in practice. I'm particularly glad they mentioned the dbt-utils and dbt-expectations libraries: https://www.datafold.com/blog/7-dbt-testing-best-practices
The dbt ecosystem is also a strength. This article has good examples of the type of tooling and libraries in the ecosystem: https://meltano.com/blog/top-5-of-dbt-packages-and-tools-in-2022/
I have no affiliation with any of the above linked companies, just a happy dbt user!
There are some great tools in this space, definitely worth exploring.
Our vision is to glue the steps from ETL to dashboard together in an end-to-end solution. We pick whatever we consider best in class and integrate it. So far, we've got Singer, DBT, Jupyter Notebooks and Apache Airflow and we're using VueJS for both the product UI and our website.
We're also working on a blog post exploration what other acquisition might happen in this space. We're adding suggestions to the spreadsheet as we hear them on Twitter, HN, etc https://meltano.com/blog/2019/06/10/first-looker-and-tableau-which-data-visualization-companies-are-likely-to-be-acquired-next/
This is great.
Things to keep on your radar:
Meltano spun out of Gitlab is working in this domain, I think they are making progress.
https://meltano.com/
dbt are building the transformation engine that Meltano is using (I think), and are worth keeping an eye on
https://www.getdbt.com/
In my experience the issue with this domain isn't the "one-off" analysis, but rather orchestrating the BI function across the business, maintaining the single source of truth, testing and deploying across the ETL/ELT layers. I can't speak to how well Ananas is managing these but Meltano and dbt are giving this area a lot of thought.