Todd W. Schneider

I recently attended a talk about the growth in volume of rides in NYC that Uber has been providing. It was contrasted against a mostly flat (if slightly declining) volume for yellow cabs (remember, they're not making any more medallions). He blogged about it here: http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-taxi-and-uber-trips-with-a-vengeance/ - chart here: http://toddwschneider.com/data/taxi/uber_vs_taxi_pickups_brooklyn.png

He deplores those poor cabbies? I deplore all those poor riders who couldn't hail cabs and had to choose between worse and worst options because of the cabbies' government imposed monopoly. (What do you do when you can't hail a cab? Wait for hours? Walk? Ride for hours on public transit that goes everywhere but your destination? Give up on going where you need to go?)

Just like (apparently) most others here, I seem to use Facebook mostly to keep in touch with my family and (close?) friends. I unfollow pretty much everyone but my mom and wife. Facebook turned me on to adblock when they kept showing me ads for a degree when I had already earned that same degree from a better local school (and put it in my info on Facebook - so they were basically shamelessly ripping of the advertiser.) Those things make Facebook about a weekly experience now for me. I keep in touch professionally on LinkedIn.

For better or worse, I get my news on Hacker News moreso than any other place. I don't really even bother with the New York Times or Wall Street…

I'll just add to this thread that a large % of Citibike members in NYC are also owners of personal bicycles (like myself - unfortunately can't find source atm). My personal use case for Citibike are short/moderate length trips(1-4 mile) in the city core (Manhattan) where public transportation is unwieldy. Depending on your origin and destination, what is a 10/20 mins ride on a Citibike is easily a 30/40 mins ride on public transit (walking to stations, if you need to change trains to get to the destination. As for taxis and ridehailing, unless you're taking a ride in the middle of the night, traffic will also make your trip longer than biking. A bike can often reach the top speed of traffic in the city and you have the added ability to pass long lines of cars. This doesn't even take into account that Citibike costs 10$ a month, that's likely the cost of one trip if I were to take a car. And something no one has mentioned, buses, are notoriously slow in Manhattan and don't come nearly frequently enough to be of use in most cases.

There aren't even any Citibike docks near where I live but the utility in the city core is almost unmatched. Obviously there are situations where bike(+ bike shares) are less than ideal (ridership goes down in winter/rain/hot days/ect., disabled users) but they are a great boon to the mobility of people living in dense cities.

Some interesting reads:…

This is a good list, but as others have said, is not the level of detail I'm talking about.

Take the NBA for example. Let's look at this: http://toddwschneider.com/posts/ballr-interactive-nba-shot-charts-with-r-and-shiny/ - this is able to give you super detailed analysis thanks to the NBA's stats API.

The equivalent from Opta is thousands a year per competition. I was fortunate enough to get to play with detailed Opta data and ChyronHego data as part of a Man City hack day a couple of years ago. The latter data simply isn't commercially available.

For cricket, you can do something interesting with ball by ball data, but ideally you want ball tracking data. You want to know speed of release, length, speed and movement after the ball has pitched, and speed after interaction with the batsman along with angles, etc. - and that's just to get started. Ideally you want positional data on fielders, etc. too.

Don't get me wrong, this is a great starting set to get people interested, but there's a way to go for high-quality data being accessible to the hobbyist or academic researcher (although I believe Opta gives academics discounts to help make them "the" standard for clubs, etc.)

This is kind of crazy but really interesting as a sort of extreme manifestation of data-centric applications. I say it's crazy, but startups like Supabase are essentially building the professional version of your idea.

The share of discourse on HN about PostgreSQL, SQL, and Databases is growing noticeably. [0] I, for one, hope we are seeing the pendulum swing back from code-centric architectures and NoSQL to a new golden age of data-centric architectures.

Next you need push and pub/sub semantics in PostgreSQL so you can build something like this [1]

[0] https://toddwschneider.com/dashboards/hacker-news-trends/?q=postgresql%2Csql%2Cdatabase&f=title&s=text&m=frac_items&t=year
[1] https://tonsky.me/blog/the-web-after-tomorrow/

If you will use S3 as a storage for cold data it can be become just S3(or other object storage) cost. And ClickHouse compression rates for Twitter-like data can be 10x or better as you do not store raw data and data in stored in columns.

Concerning ingestion costs, I think it took me ~10 hours of 4vCPU vm last time, when I loaded 1.5 TB data to ClickHouse from this dataset: http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-taxi-and-uber-trips-with-a-vengeance/ 1.1 billion rides became 3.5 billion already.

https://toddwschneider.com/dashboards/hacker-news-trends/?q=depression%2Csuicide&f=title&s=text&m=frac_items&t=year

... changing that to quarterly sure is telling about when those topics are discussed most: https://toddwschneider.com/dashboards/hacker-news-trends/?q=depression%2Csuicide&f=title&s=text&m=frac_items&t=quarter

I wonder where that spike of "suicide" in Jan'13 was caused by? And the Jan'18 spike in depression? Maybe by the bitcoin burst?

I'm not sure if I'm amazed or simply terrified. Fascinating!