Teads' engineers write about cloud costs, Kafka and BigQuery analytics, Spark tuning, and scaling infrastructure.
Commenters link it for AWS cost optimization advice and Spark performance tuning tips.
There's also the risk of bad randomness sources and/or bugs.
One popular UUID library got a bug report stating: "We are generating about 1M UUID4 a day, and we are getting several hundred collisions a day". And so they were; turned out to be a bug/weird interaction between the OpenSSL library they were using for randomness and forking. (Details here, although it was all fixed years ago of course: https://github.com/ramsey/uuid/issues/80)
On paper, you should never, ever, ever see a collision when generating a mere million v4 UUIDs a day, much less hundreds of collisions. But that doesn't mean it can't happen!
This is also an interesting bit of analysis; comes from a company that processed a lot of UUIDs generated in browsers, checked, and discovered about 5 collisions per million UUIDs. Again, not what you'd naively expect! (Turned out to be mostly driven by misbehaving crawlers.) https://medium.com/teads-engineering/generating-uuids-at-scale-on-the-web-2877f529d2a2
Blog article about real-life AWS cost optimization strategy :
https://medium.com/teads-engineering/real-life-aws-cost-optimization-strategy-at-teads-135268b0860f
TL;DR:
- Monitor your daily costs
- Cloudify your workloads
- Use reservations, Spot instances, saving plans
- Look around for unused infrastructure
- Optimize S3 consumption
Karthik, I'm no Spark expert but almost all advice I read is to avoid UDFs if at all possible. Examples below:
- https://medium.com/teads-engineering/spark-performance-tuning-from-the-trenches-7cbde521cf60
- https://www.inovex.de/blog/efficient-udafs-with-pyspark/