Earlier quoted context omitted.
I see ChatGPT's bots pull down all of my Python wheels every couple of weeks. Wheels that haven't changed in years, with a "Last-Modified" and "ETag" that haven't changed. The only thing that makes sense to me is it's cheaper them to re-pull and re-analyze the data than to develop a cache.
Or, could it be, just possibly be (gasp), that some of the devs at these "hotshot" AI companies are just ignorant or lazy or pressured enough, so as to not do such normal checks? Wouldn't be surprised if so.
For what it's worth, mj12bot.com is even worse. They pull down every wheel every two or three days, even though something like chemfp-3.4-cp35-cp35m-manylinux1_x86_64.whl hasn't changed in years - it's for Python 3.5, after all.