Earlier quoted context omitted.
Yes :( Do you have an idea of how to remove company websites in an automated way? I didn't want to manually review all 7k indexed websites. This is the GPT prompt I used for filtering domains to add, but it gives false positives: You are an API. Return a JSON array of booleans indicating whether each provided domain is someone's personal website. Use common sense. Make sure to return false for company websites.
For the 404s (assuming the status code isn't a 4xx), use a URL that you strongly suspect won't exist, then you can do a comparison (levenshtein distance, bag of words, etc.) to see if it's very similar to one of about, ideas, etc. pages.
I think openai embeddings API could be useful here. Perhaps one of the neurons responds to corporate speak.