Earlier quoted context omitted.
Google wouldn't know that someone is curling it which their script said everything was ok. While the website was basically down because Chrome and Firefox will both block a site based on Google's safe browsing list. They could use Google's safe browsing api to check if they're on that list as well as curl.
Frankly, Google's Safe Browsing list is one of those things which should be broken out of Google and ran by an independent entity, much like the Let's Encrypt model. I don't know how to help make this happen.
1. Mozilla and the Chrome clones (Edge, Brave, etc.) partner to make an (open as possible) standard for blocking and reviewing, and start maintaining their own list upstream from Google.
2. When Google adds to their blocklist, independently check it according to the consortium's own standards.
3. Cut a deal with Bing or Yandex to scan for malware as part of their crawls, to get technology independence.
4. Put pressure on Google to get onboard.
Step 4 is the hardest.