Stitch Data Blog

While this is a very open question both in terms of your analytical resources (dedicated data engineers, how tech savy are analysts, can they easily model their own data) as well as constraints from operating in an enterprise setup with regards to DevOps (easy to open ports, can data viz solutions be run online or only on prem), I'll try to mention a bunch of tools assuming they can be used. Notably, one thing you did not mention is which data visualization tool is used and how much that allows general business users to get insights.

Database:

In terms of enterprise ready Data Warehousing databases I would take a good look at Exasol (on prem & cloud) and Snowflake (cloud only). Exasol will likely be the pricier of the two and is much closer to the Java based Oracle world than Snowflake. Personally, I'm not a big fan of Redshift, as Snowflake takes all the pain that one might experience in Redshift away (manual tuning of sort & dist keys, tuning of coupled storage & compute clusters). Again, if you have good data engineering resources readily available it might make sense to look into a comparative cost analysis of the three in more detail. Notably, Snowflake has Snowpipe, a native API allowing you to stream data into the DB.

Data modeling:

You mentioned you use Kimball. In my experience while this is the definite choice for analytical querying and modeling, in an enterprise setup it might make sense to draw a Data Vault model layer between your raw sources and your…

some of the many links in that article

https://apnews.com/article/google-chrome-privacy-lawsuit-settlement-203cc5063f1a1d4013de1900d9376814 https://apnews.com/article/private-incognito-browsing-explainer-google-chrome-f8b3dd9ae41c5d9da027454e5c0c92c6 https://carnegieendowment.org/2023/09/07/facebook-telegram-and-ongoing-struggle-against-online-hate-speech-pub-90468 https://epic.org/issues/consumer-privacy/data-brokers/ https://lithub.com/surveillance-capitalism-and-the-internet-of-things/ https://mitsloan.mit.edu/ideas-made-to-matter/machine-learning-explained https://news.harvard.edu/gazette/story/2019/03/harvard-professor-says-surveillance-capitalism-is-undermining-democracy/ https://proton.me/blog/privacy-washing-2023 https://theconversation.com/explainer-what-is-surveillance-capitalism-and-how-does-it-shape-our-economy-119158

https://www.abc.net.au/news/2018-06-25/healthengine-sharing-patients-information-with-lawyers/9894114 https://www.apa.org/monitor/2024/04/addressing-equity-ethics-artificial-intelligence https://www.axios.com/2017/12/15/sean-parker-unloads-on-facebook-god-only-knows-what-its-doing-to-our-childrens-brains-1513306792 https://www.brennancenter.org/our-work/research-reports/closing-data-broker-loophole https://www.buzzfeednews.com/article/meghara/period-tracker-apps-facebook-maya-mia-fem https://www.commonsense.org/education/articles/a-majority-of-apps-are-about-to-come-clean-and-say-theyve-been-selling-your-data-all-along…

We use Stitch[1] for this. You can selectively replicate some tables from any "source" to any selected destination[2]. I believe 1 destination is there on the free plan. You can point to any self hosted(or cloud) Postgres DBs. There are a bunch of supported destinations eg. Postgres, Snowflake etc. The sources also can be a varied like Google sheets.

We use this to send relevant tables to analytics teams outside of engineering. There's a bit of tweaking initially(i.e. having a timestamp column like 'last_updated` helps to replicate only those rows)

Having a separate readonly DB user is also pretty common. We use that internally inside engineering(where we trust them slightly more). Important bit is to always add a statement_timeout so people cannot not run random slow queries on your production database that are hung for days.

I've also had my eye on https://www.singer.io/ , which seems like an open source ETL tool similar to Stitch. But i've not experimented with it locally yet. It uses "taps" and "targets" nomenclature. And there's a healthy ecosystem of opensource extensions of creating your own taps/targets.

[1] https://www.stitchdata.com/

[2] https://www.stitchdata.com/integrations/destinations/

Stitch | Philadelphia, PA | https://www.stitchdata.com/jobs/

Stitch is an ETL service built for developers. Stitch connects to today’s most popular business tools – Salesforce, Facebook Ads, and many more – and replicates the raw data to a data warehouse. With Stitch, decision-makers are free to interact with their company’s data using the tools of their choice, whether that’s BI platforms like Mode and Looker, data science languages like R and Python, or a simple SQL client.

We are hiring for the following roles, which are all full-time and onsite in Philadelphia:

- Lead QA Engineer
- Product Manager
- Sales Development Representative (x2)

Send your resume to sam@stitchdata.com or apply at the link above!

Typically these platforms will query on a data warehouse that you provide: redshift, bigquery, a big postgres instance, or the like.

In terms of getting data into that warehouse, you can write ETL (extract, transform, load) scripts yourself. There are also a number of ETL-as-a-service providers like Stitch or Fivetran that cover common things you might want to sync into your data warehouse and query like salesforce, google analytics, quickbooks, etc: https://www.stitchdata.com/integrations/sources/