Tom Forbes writes about Python, Django, security research, and cloud tooling, including large-scale scans of public packages.
HN commenters link his S3 deletion visualization, his PyPI scan that found live AWS keys, and his XPath injection work.
Ha! Well, there's a lot.
On major strand is more work like this to make it easy for the community to collaborate. I expect we'll make a lot of iterative improvements to the database over the next few months, aimed at making it easier to contribute to, maintain and use. We need to improve our APIs for this data, for example (currently only available via GraphQL).
Another big one that we're starting to think about is the security vulnerability disclosure process. Our goal there is to support maintainers as much as possible, and there's more we can do. Recent articles on loguru, beg bounties, and the way log4j initially reached public attention all point to problems GitHub can and should help with. In the next 12 months we'd like to give maintainers the option to receive vulnerability disclosures privately on GitHub, and for us to be able to support them through that process. (GitHub already does a bit here - through maintainer security advisories we issued about 30% of the CVEs in the JavaScript ecosystem last year, for example. But we can and will do more.)
Loguru CVE article: https://tomforb.es/cve-2022-0329-and-the-problems-with-automated-vulnerability-management/
Beg bounties: https://www.troyhunt.com/beg-bounties/
Log4j PR: https://github.com/apache/logging-log4j2/pull/608#issuecomment-989762094
Ignore the replies about inventories. $0.005 Per 1k list requests. A list request returns 1k items. 1 million items for $0.005, or 1 billion items for 5 dollars.
Still use lifecycle policies (I visualized how it ages out 1 billion items here[1]), but list request prices are not a factor worth mentioning.
1. https://tomforb.es/visualizing-how-s3-deletes-1-billion-objects-with-athena-and-rust/
This is certainly an interesting idea. A quick Google search gave me this toy example: [1][2]
Of course making it work for all cases would be very hard due to the excessive dynamism of Python, but for simple arithmetic operations it could be made to work fairly well.
[1] http://tomforb.es/automatically-inline-python-function-calls
[2] https://github.com/orf/inliner
Thanks for the fantastic work!
So you found 57 live AWS keys[1]. That's out of 1631 according to your stats (If I didn't misread something). I wonder how many of the top two in the list are live :D
Google API Key 4,015
OpenAI API Key 3,531
[1] https://tomforb.es/i-scanned-every-package-on-pypi-and-found-57-live-aws-keys/
I thought XPath was terrible beyond 1.0?
E.g. https://tomforb.es/xcat-1.0-released-or-xpath-injection-issues-are-severely-underrated/
But it does not seem to be mentioned as an argument.
I wrote my own to use SQLAlchemy models you define:
http://tomforb.es/using-a-custom-sqlalchemy-users-model-with-django
Its really not that bad or hard to do at all.