Target's engineering team writes about its infrastructure, open source projects, store systems, and security tooling like skimmer detection.
HN commenters cite it for Target's developer-productivity turnaround, store deployment pipeline, the EasySweep skimmer checker, and the company's open source fund.
It works well enough for Netflix - https://medium.com/netflix-techblog/edda-learn-the-stories-of-your-cloud-deployments-ade4f4edd9cd - and Target ( https://tech.target.com/2017/04/07/how-and-why-we-moved-to-spinnaker.html ) :). I've only ever used Edda as part of a Spinnaker setup, but there's no reason it couldn't be used with something else, or the same idea in general reused.
You don't have to cache it for very long, but you get two benefits here: Being able to query the data and search through it ways you simply can't do when making an API call to AWS, and having primarily only one system munching through your API throughput limits. If you've got a large account with a lot of resources and a lot of people or systems that are querying the API frequently, you'll probably have more consistently available data, rather than having systems sitting around doing retries and getting throttled there, on repeat.
Sorry -- I misremembered the story. But as another commenter notes, it's routine for women to not know they are pregnant [0]. So my claim is not "absurd", and I don't feel any need to "reflect" on my actions. The patronization is not necessary.
The main intent of my anecdote is to show how retail companies are already analyzing data. And I think it's foolish to assume that we (i.e. people working directly in tech) know better than the people who are working in tech + retail. (Many retail stores are on the technical bleeding edge - Target spins up K8 clusters in each store [1]!)
[0] https://www.webmd.com/baby/features/can-you-be-pregnant-not-know-it#1
[1] https://tech.target.com/infrastructure/2018/06/20/enter-unimatrix.html
This blog post[1] from the developer helped me to understand this better. It's linked on the github page, but I wanted to provide the link for more direct clarity.
Astonishingly, we can process 20 years’ worth of data, sampled every five minutes, in less than 20 seconds.
That sounds quite promising.
[1] https://tech.target.com/2018/12/11/matrix-profile.html
Have you heard of BackStage [1]? A few companies have been adopting it to help make teams and departments to help devs. Target is a notable example of going from a really unproductive environment to much more supportive of devs [2]. Including handling building out infrastructure stacks you mentioned.
* 1. https://backstage.io/
* 2. https://tech.target.com/
>The greatest challenge teams faced with developing software for the stores was figuring out how to effectively build a continuous delivery pipeline to 1,800 deployment targets that was safe and expedient.[1]
[1] https://tech.target.com/infrastructure/2018/06/20/enter-unimatrix.html
This story about a physical card reader checker to detect skimmer devices was interesting and posted here a couple years back.
https://tech.target.com/blog/cybersecurity-easysweep
https://news.ycombinator.com/item?id=36788831