Holistics is a business intelligence software company whose blog covers SQL data modeling, OLAP cubes, and building analytics stacks.
HN commenters mention it as a BI tool alternative to Chartio and link its OLAP-versus-OLAP-cube piece as a worthwhile deeper read.
It really depends on your budget, data volume and nature. So each person can only offer advice.
I did this a while back at my previous employer/startup (it got acquired), we used AWS, PostgreSQL, Hadoop/Hive, using SQL + custom Ruby script for ETL + processing. You can read some of that writing here
https://engineering.viki.com/blog/2014/data-warehouse-and-analytics-infrastructure-at-viki/
If I can distill them into bullet points, there probably are:
- SQL is great, stick with it.
- We started with PostgreSQL and scales a long way with it (using techniques like table partitioning, unlogged table, etc), then slowly split into Redshift, and later Hadoop/Hive.
- Handling events/behavioural data (JSON/semi-structure at large volume) are very different from handling transactional data (structured, lower volume).
- Take a more lean/incremental approach to it: get a basic DW setup, load data that you can immediately act on, act on it (build reports, run analysis, show to management), then repeat.
You can also check out my startup: https://www.holistics.io , where I turned those experiences above into a data platform that automating DW + BI.
We allow you to do "lean ETL" on top of customers' DW infrastructure (be it Redshift, PostgreSQL, BigQuery or others). We work with small startup to unicorn tech companies.
Well said!
We need more of this problem space exposed to engineers and not just for “analysts”.
I’ll share a couple other articles from a company that does a nice job explaining the technical problems in what is traditionally “business analytics”.
The space is OLAP and you may have scoffed at the idea of “OLAP cubes”, but man were they useful. In the way that excel powers a ton of business processes, cubes powered a lot of analytics. Underlying tech is cool but they are showing their age:
https://www.holistics.io/blog/the-rise-and-fall-of-the-olap-cube/
Another write up of this idea of a semantic layer above raw sql statement: https://www.holistics.io/blog/holistics-data-modeling-explained/
So this “semantic layer” leverages the latest tech to deliver the same business insights faster, better, more flexibly. Ie once you define this semantic layer over your data (ie how all your sql tables are connected), the semantic engine knows how to query up and down your data model, writing the SQL queries for you, on the fly. You can ask and answer new questions without writing new queries. And with modern columnar query engines (eg big query, spark, presto, etc), perf is usually pretty good.
For anyone who is not familiar with Looker, I wrote a write-up and review about it here https://www.holistics.io/blog/looker-review-and-breakdown/. Here are the key points:
- Its use of LookML provides a steep learning curve, yet provides a maintainable and reusable data modeling
- Looker's drill-down ability is decently powerful and easy to use once you are familiar with LookML.
- Looker does not have its own storage layer but instead relies on customer's data warehouses
- Looker, in essence, is a SQL query builder engine that converts business users' drag-and-drop inputs into SQL queries.
- Looker provides highly flexible and sophisticated access control and permission management, sacrificing simplicity for power.
- Looker has limited data preparation capabilities compared to other tools, delegating this task to its partners to provide these capabilities.
Hello Tensor! Will you like to review what our team has built at Holistics (www.holistics.io)? We are listed in the Gartner FrontRunner Quadrant for Business Intelligence Oct 2018.
Here's a customer case study we've just published recently:
https://www.holistics.io/customers/floship-ecommerce-fulfillment-data-automation/
Love to get your feedback on how you find us relative to the tools you are currently evaluating! Send me a note at v(@@)holistics.io or our team at hello(@@)holistics.io]!
Have you looked at Quel? We have multiple languages to run our code on generic CPUs (C, Python, Java, Ada, Go, Rust, Lisp, etc.) Why must we have only one database language that doesn't do a very good job at Codd's relational calculus?
So many people have drank the kool-aid that SQL is the answer. Maybe it's time to change this.
The success of ORM's could be thought as the market voting against SQL.
https://www.holistics.io/blog/quel-vs-sql/
I think it's missing the resources to one of the hardest sections: Data modelling, like Kimball and Data Vault. That, and maybe a section to modern data infrastructure. I'd put a link to [1] and [2] for a quick overview and probably [3] for more detail.
[1] https://www.holistics.io/books/setup-analytics/
[2] https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/
[3] https://awesomedataengineering.com/