I'm working on a chrome extension for use in a headless browser in marginalia search to capture information about network traffic, ads, and popovers when visiting a website, to better identify nuisance websites. A bit of a janky setup, but I've mostly gotten it to do what I want it to do after some head scratching.
Holy shit, so you are uh building a search engine from scratch. Do you crawl yourself? What is your infrastructure? What is your goal for search.marginalia ?
Yup
> Do you crawl yourself?
Yup
> What is your infrastructure?
All custom built in Java, sitting on a rack server in a basement in Sweden.
> What is your goal for search.marginalia ?
I'm basically building what I feel is lacking in internet search and discovery, which is tools for finding stuff based on something other than a popularity metric, as those tend to feed into themselves to make the web seem so small.