Show HN: AboutIdeasNow – search /about, /ideas, /now pages of 7k+ personal sites

Post by phgn » Mon, Feb 26, 2024, 6:00 PM UTC

layer8 wrote: post #39

phgn wrote: post #37

Earlier quoted context omitted.

Yes :( Do you have an idea of how to remove company websites in an automated way? I didn't want to manually review all 7k indexed websites. This is the GPT prompt I used for filtering domains to add, but it gives false positives: You are an API. Return a JSON array of booleans indicating whether each provided domain is someone's personal website. Use common sense. Make sure to return false for company websites.

Maybe change the API so that GPT can express uncertainty (make it a ternary value or even a confidence percentage), and then check the “uncertain” cases manually.

Post by netghost » Mon, Feb 26, 2024, 6:15 PM UTC

phgn wrote: post #37

KomoD wrote: post #34

Last I checked bleepingcomputer, ifixit, vibilagare.se, psyche.co and libreboot.org aren't personal sites. None of those actually have an /about page, yet your site says they do... Another funny thing: just search 404 or not found and you'll get a lot of 404 pages

Yes :( Do you have an idea of how to remove company websites in an automated way? I didn't want to manually review all 7k indexed websites. This is the GPT prompt I used for filtering domains to add, but it gives false positives: You are an API. Return a JSON array of booleans indicating whether each provided domain is someone's personal website. Use common sense. Make sure to return false for company websites.

Post by KomoD » Mon, Feb 26, 2024, 6:19 PM UTC

netghost wrote: post #43

phgn wrote: post #37

Earlier quoted context omitted.

Yes :( Do you have an idea of how to remove company websites in an automated way? I didn't want to manually review all 7k indexed websites. This is the GPT prompt I used for filtering domains to add, but it gives false positives: You are an API. Return a JSON array of booleans indicating whether each provided domain is someone's personal website. Use common sense. Make sure to return false for company websites.

For the 404s (assuming the status code isn't a 4xx), use a URL that you strongly suspect won't exist, then you can do a comparison (levenshtein distance, bag of words, etc.) to see if it's very similar to one of about, ideas, etc. pages.

Post by phgn » Mon, Feb 26, 2024, 6:40 PM UTC

KomoD wrote: post #45

netghost wrote: post #43

Earlier quoted context omitted.

For the 404s (assuming the status code isn't a 4xx), use a URL that you strongly suspect won't exist, then you can do a comparison (levenshtein distance, bag of words, etc.) to see if it's very similar to one of about, ideas, etc. pages.

> For the 404s (assuming the status code isn't a 4xx) Most are a 4xx code, I checked myself, some may be 301/302 redirect to 4xx not being handled properly by their crawler

Post by digging » Mon, Feb 26, 2024, 6:53 PM UTC