I was telling a friend recently about how there was this "golden age" when you could access all sorts of free APIs, and how I still long for this time. I remember the public Netflix API, Twitter APIs and Flickr API with particular fondness. My personal site was a big mashup of all of my data. I also abused the hell out of Yahoo Pipes - I would run RSS feeds through like 15 different languages with Babelfish before ba…
I remember when you could just stream in every tweet and reddit comment ever posted with a few lines of code. There used to be dedicated API methods for doing so. Now, there's all sorts of namespacing, rate limiting, pagination, and upper bounds on data access that make this impossible, or at least infeasible.
Researchers are passing around lists of tweet IDs ("dehydrated", they call them) that can be "rehydrated" (that is, turned back into full tweets) if you have the right permission from twitter to do so.
The whole setup is really shameful.
It would be de-facto illegal to build a "Google for Twitter" today. I settled on doing it for ActivityPub/Mastodon because it's less likely I'll get sued into oblivion for creating a search engine that way.