Skip to content
HomeVideosBooksBlogs
AllTop postsTopics
Podcasts
HistoryListsWatch laterLiked videosFollowing

The videos, books, blogs and podcasts Hacker News keeps recommending, all in one place.

VideosBooksBlogsPodcastsAboutPrivacySettings

Data from Hacker News, HackerTube, HackerBooks, HackerBlogs and Open Library. Credits · Not affiliated with Y Combinator.

HomeVideosBooksBlogsPodcasts
Ctrl K
    I

    Insight Data Engineering Blog

    insightdataengineering.com · Company blog · on HN since 2014

    Insight Data Engineering's blog comes from a fellowship program, with posts on Spark, Flink, Elasticsearch, Cassandra and Kafka projects.

    Visit blogHN submissions
    DatabasesDistributed SystemsCareer & Work
    What HN says

    HN commenters discuss the fellowship's admissions, such as who can apply, and one advises moving into data engineering before machine learning.

    Stats for nerds
    21HN people shared it
    33linked in comments
    706story points
    Oct 2016last shared
    2014people per year2016

    Top posts on HN

    1. 1
      Building a Forex Trading Platform Using Kafka, Storm and Cassandra
      1 person · 139 points · first shared 2014 · 39 comments on HN
    2. 2
      Cassandra: Daughter of Dynamo and BigTable
      1 person · 81 points · first shared 2015 · 47 comments on HN
    3. 3
      Anatomy of an Elasticsearch Cluster: Part I
      1 person · 67 points · first shared 2016 · 9 comments on HN
    4. 4
      Streaming Search on Tweets: Storm, Elasticsearch, and Redis
      1 person · 59 points · first shared 2016 · 7 comments on HN
    5. 5
      Insight Data Engineering Fellows Program
      3 people · 53 points · first shared 2014 · 20 comments on HN
    6. 6
      The New Data Engineering Ecosystem: Trends and Rising Stars
      1 person · 50 points · first shared 2015 · 8 comments on HN
    7. 7
      The Data Engineering Ecosystem: An Interactive Map
      1 person · 47 points · first shared 2015 · 8 comments on HN
    8. 8
      Spinning Up a Spark Cluster on Spot Instances: Step by Step
      1 person · 37 points · first shared 2015 · 10 comments on HN
    9. 9
      Working with Apache Spark: Highlights from projects built in three weeks
      2 people · 26 points · first shared 2015 · 6 comments on HN
    10. 10
      Computing shortest distances incrementally with Spark
      1 person · 26 points · first shared 2015 · 10 comments on HN

    What HN people wrote when they linked it

    • vishalzone2002Sep 2014

      Consider doing a soft move rather than a hard move. I will explain what I mean by that.
      Instead of spending your time learning ML I would suggest first make a move to data engineering where you use more of the big data technologies like hadoop, spark, python, pandas, etc. You can also look at some kind of a bootcamp or at the least the topics they cover is obviously helpful: http://insightdataengineering.com/
      Some other tips :
      - Follow data engineering and data science topics on quora.
      - Once you feel confident try to rewrite your linkedin profile from data warehousing developer role to a data engineer role. Add big data technologies to your skill set.
      - Try to get into an auction on hired.com
      - It is false that MS does not use Hadoop. I have interviewed with teams that do but they are all in seattle. Try to move internally to teams like Azure, HDInsight.
      - ETL, data modeling, SQL, designing data warehouses and pipelines are actually still the most useful skill to have. So hold on to that.
      - One thing that I haven't done yet but really mean to is to get on Kaggle. You should too at some point.

    • syntaxingOct 2016

      I actually pretty interested in joining this program since I am near one of the locations! Do you guys accept other type of engineer (in my particular case Mechanical Engineering) with a background in programming?

      Also side note, I think you guys probably just migrated to new domains but there seems to be a couple 404 links/pages on the site? The ones I was looking in particular are these two http://insightdataengineering.com/insightdataengineering.com/data-engineering-notification-list http://insightdataengineering.com/insightdataengineering.com/data-engineering-notification-list

      Also, there's an cert error that pops up in Firefox for http://insightdata.ai

    • javadi82Jun 2015

      I know this beats the very premise of your product - "personalized" guidance but thought I would ask anyway.

      How open are you to sharing general guidelines (that are insightful and not already available everywhere) ?

      It would also help if you could ask the people who are successfully placed to write their learnings along the process. Similar to how the Insight data engineering fellow and others at coding schools share what they learnt/built.

      Eg: http://insightdataengineering.com/blog/mapmycab.html

    • ddrum001Jul 2016

      We don't have a fixed syllabus since we don't operate like a school per se. Instead, the Fellows choose a project and have to make engineering tradeoffs about which tools to use. With that said, most of the past participants focused on distributed systems like Hadoop, Spark, Flink, Kafka, and NoSQL DBs. You can check out past projects on the blog (http://insightdataengineering.com/blog/) and the past Fellows are here (http://www.insightdataengineering.com/fellows.html)

    • jakekMar 2014

      The reason for this is that the Insight Data Science Fellows Program is designed for (and only accept applications from) PhDs/postdocs. The new stand-alone Data Engineering program (http://insightdataengineering.com) is open to all applicants, as long as they have good engineering/CS fundamentals, regardless of level of education or discipline.

    • ronaknnathaniOct 2016

      Sorry about the broken links on the Data Engineering page. Here is the correct one to subscribe to the notification list. http://insightdataengineering.com/data-engineering-notification-list

    Related

    Insight Data Science Blogblog.insightdatascience.com85 people

    Generating custom photo-realistic faces using AI

    Insight Data Science Blog
    Company · last shared 2022
    Uber Engineering Blogeng.uber.com525 people

    Why Uber Engineering Switched from Postgres to MySQL

    Uber Engineering Blog
    Company · last shared 2025
    Databricks Blogdatabricks.com298 people

    DBRX: A new open LLM

    Databricks Blog
    Company · last shared 2026
    InfluxData Bloginfluxdata.com118 people

    Open source takes continued investment, and that must be subsidized somehow

    InfluxData Blog
    Company · last shared 2026
    Aggregate Knowledge Blogblog.aggregateknowledge.com15 people

    AWS Redshift: How Amazon Changed The Game

    Aggregate Knowledge Blog
    Company · last shared 2014
    Treasure Data Blogblog.treasuredata.com17 people

    Distributed Logging Architecture in the Container Era

    Treasure Data Blog
    Company · last shared 2020
    PayPal Tech Blogmedium.com/paypal-tech16 people

    Scaling Kubernetes to Over 4k Nodes and 200k Pods

    PayPal Tech Blog
    Company · last shared 2024
    Airbyte Blogairbyte.com64 people

    The deck we used to raise our seed funding

    Airbyte Blog
    Company · last shared 2026