Big Data

What Hacker News says

1. squeaky-cleanSep 2023
   I haven't heard the term ScalepUp CTO. There's a company named ScaleUp, but I'm assuming you're not their new CTO ;p but a CTO promoted during the "scale up" phase of a startup. I'm a lead engineer but was responsible for scaling a service to 500M monthly users. So sample size=1. And my suggestions will be from the "T" part of CTO and nit really the "C" part.

   Firstly, I think youtube is a great primary resource here that you shouldn't neglect. There's a lot of conference talks by companies that scaled very quickly which go into great technical detail.

   Also blog posts, there's a lot of great blog posts from the scale up phase of companies that don't hide anything.

   The reason these fairly transparent conference sessions and blog posts exist is because the company is trying to interest top-level applicants and to look good for a future sale (I know because I experienced these requests) but it's great for just learning from.

   For actual books, my suggestions will be very related to the system I had to scale. So my main suggestion is to first find some means to learn the optimal ways for your own system to scale before you go deep into any specifics.

   "Designing Data Intensive Applications" by Martin Kleppmann is as the title describes. If "data" storage or manipulation is central to your systems (and what modern app isn't), I'd say it's essential to at least skim through. It's more of a breadth than depth book, but it goes deeper than you'd expect.

   "Big Data" by Nathan Marz was foundational to the system I ended up designing, usefulness may vary. It describes a specific architecture for a flexible and high throughput data-based application. He has a new framework(?) named Rama that's open source, has some good blogs, and was recently featured on HN. And it appears to be based on improvements to the ideas in Big Data. So that's free and may be a better first look, though the available details aren't as in depth as the Big Data book.

   "Event Streams in Action" by Alexander Dean & Valentin Crettaz is another one that may be specific to the area I needed. But if that's what you need, it has what you need.

   "The Data Warehouse Toolkit" is another somewhat specific usecase book. It's also more developer-level than CTO level in a lot of areas. But it includes good, very specific examples such as several chapters dedicated to case studies of specific industries like banking, accounting, ecommerce, healthcare.

   "Working Effectively with Legacy Code" By Michael Feathers is a personal favorite. Maybe a bit opinionated. You might think "We have older/early startup code, but we don't have legacy code", but Michael introduces a unique (and IMO very effective) idea of 'Legacy Code', anything that doesn't have tests is legacy code. It's a great resource on refactoring/maintaining/modifying and even rewriting important code without comprehensive coverage, and not written to make test coverage simple. The way I now look at it, any services written before the "scaleup" phase are almost always legacy code.

   "Scrum: The Art of Doing Twice the Work in Half the Time" By Jeff Sutherland. If you're doing Scrum abd haven't read the original, I'd give it an extreme recommendation. The most important part of Scrum (in my opinion) is the flexibility and self-modification. The team I lead chose to trial Scrum and it was fantastic. The whole company decided to implement it. When the CEO learned we had abandoned literal "standups" for daily slack threads he made us revert to standups. The whole company had to follow the initial template laid out by Scrum with no deviation. It failed horribly and everyone hated it, it became the dreaded "waterfall with sprints". The main issue is he didn't listen to us about how it's meant to be modified, but I also think a big issue is he ordered 50 copies of some 30-page cliff-notes version of Scrum for the whole office, which completely omits the "kaizen" and other details about how malleable it can be.

   "Don't Make Me Think" by Steve Krugg. Not so much about scaling but about user testing. So maybe not technically CTO territory but please advocate for this :P I'm paraphrasing some phrase in the book but 2 hours of proper user testing from random people off Craigslist/etc will provide you with 2 months of dev-work. You will not believe the features users misunderstand or various opportunities for additional sales users completely miss until you try with real people off the street.

   "High Output Management" by Andrew Grove is about where management can be effective or harmful in developer productivity. It's an older book, sir, but it checks out. Some of it is fairly commonplace information nowadays, but still a good intro to tech management, and probably some useful information for very high-level management since Grove was the Director of Engineering at Intel when he wrote it.

   Show morethread
2. robin_philipFeb 2018
   We had a similar problem building an ad network. We ended up logging events into a HDFS using Kafka. Backend for serving was built using Java (Go Seems fine too). As for serving reports in real time, I don’t think that would be doable. We need up refreshing the reports every 30 minutes. Building these analytics views can be accomplished using Cassandra, Hive or ElephantDB or any other. People have used apache storm also. You can look up the book Big Data principles by Nathan Marz for discussions on scalable architecture.

   Disclosure: I did not code this. I only participated in architecture reviews.

   Show morethread
3. BrandonBradleyOct 2016
   For big data, 'Big Data' by Nathan Marz was an excellent read. The conceptual chapters are top notch, and the implementation chapters give you a good look into the tools used for the field at the time of publishing.
   thread
4. mindcrashFeb 2016
   You probably might want to read this (for free): http://book.mixu.net/distsys/single-page.html

   And pay a little to read this book: http://www.amazon.com/Designing-Data-Intensive-Applications-...

   And this one: http://www.amazon.com/Big-Data-Principles-practices-scalable...

   Nathan Marz brought Apache Storm to the world, and Martin Kleppmann is pretty well known for his work on Kafka.

   Both are very good books on building scalable data processing systems.

   Show morethread
5. pyotrgaloisDec 2015
   Two great books:

   - "Functional Programming in Scala" by Paul Chiusano and Rúnar Bjarnason

   - "Big Data: Principles and best practices of scalable realtime data systems" by Nathan Marz and James Warren

   thread
6. ludwigvanNov 2014
   Here are a few suggestions by me, tried to pick items that will hopefully stand the test of time, one per major publisher:

   - Nathan Marz' (of Backtype/Twitter/Storm fame) Big Data (Manning): Don't let the name Big Data make you feel this is only for people with big data needs, in fact Nathan Marz tries to rethink how we store data. Immutable, append-only master data sets and views derived from them. http://manning.com/marz/

   - Martin Kleppmann's Designing Data-Intensive Applications (O'Reilly): http://dataintensive.net/ in beta. "This book will help you navigate the diverse and fast-changing landscape of technologies for storing and processing data. We compare a broad variety of tools and approaches." Martin has been researching for the past year for the book and aim to have a timeless book on the subject.

   - Colin Jones' Mastering Clojure Macros: Write Cleaner, Faster, Smarter Code (Pragmatic Bookshelf): This short, but dense book makes macros understandable. Recommended if you want to learn what makes Lisp so powerful. https://pragprog.com/book/cjclojure/mastering-clojure-macros

   Show morethread