If you do non-registration rate limiting on an api, then you still need some identifier to calculate api usage; its not hard to find an identifier for a single machine (ie IP address, which isn't actually but close enough),
But without registration, theres no way I can think of to associate multiple machines with a single person.
And presumably if you're rate-limiting on individual users of the api (and not the total usage across all users), then you're trying to maximize the number of users accessing the api simultaneously (ie you don't want 1 user maxing out resources, denying all other users).
But with things like cloud computing, its trivial for a single user to have an absurd number of machines (legally). AWS charges on compute-hour, not number of machines. Running 1 instance for 50 hours and 50 instances for 1 hour has the same cost.
But for the server, that difference is significant; you apply rate limiting because you don't want to be hammered for a short duration followed by nothing, at least not from one user.
Registration can also be bypassed somewhat trivially (temporary emails and aliases), but its a good deal more effort than bypassing non-registration rate limiting. And presumably most people who want to scrape a dataset large enough to bother bypassing the limiter are nautrally, by their job, aware of what I've described. Bypassing registration takes a good deal more work, and more specific tooling
So tl;dr
Rate limiting by ip only affects 1 machine, but not necessarily 1 user
Rate limiting by registration affects n machines, with 1 user
Before cloud vps, 1 machine basically correlated to 1 user. Now its trivial for n machines to correlate to 1 user.