Crawler works from a single IP. User-Agent is fixed to the robot's UA. Cookies are totally ignored. The search-engine works with just 2 servers. Crawler/Indexer and Webserver/queries. Crawler is a root-server with 1gbit/s connection hosted in a datacenter. Webserver sits here at home with 200mbit downstream and 20mbit upstream.
I use the Alexa top-1-million sites as seed-list for the crawler. The errors that do appear during the crawl are either sites that have an outage or more likely simply dead-links. Oh, and URLs that turn out to be blocked by robots.txt. There are a lot of sites out there which block anything but Google and Bing from crawling them.
Cloudflare is not an option for me. It would let Cloudflare know what my users are searching for. VERY big no-no. :)
I can filter out 99% of automated queries. Luckily they are still pretty dumb at the moment and give me enough fixed clues to identify them.
I like your idea of keeping the API free with a very low request-rate. That could work. I would have to find a way that they can't just generate many API-keys though. Using captchas for API-key requests won't stop them from doing that.
I posted a "Show HN" about a year ago. Brought in about 1500 extra visitor that day. Got up to 9th place on the HN homepage that day.
New webdesign is already done. I have a German site too. https://deusu.de which actually gets 90% of traffic. That site already has the new design.