Live data from Hacker News

We analyzed 425k favicons

iconmap.io

31–40 of 131 posts

Re: We analyzed 425k favicons

#31

Also, we turned up 2,000 domains that redirect to a very shady site called happyfamilymedstore[dot]com. Stuff like avanafill[dot]com, pfzviagra[dot]com, prednisoloneotc[dot]com. These domains made it into the Tranco 100k somehow. Full list here - https://gist.github.com/gurgeous/bcb3e851087763efe4b2f4b992f...

Does anyone know the story behind these? How do seemingly obscure sites consistently get massive amount of obscure content placed highly in results.

Re: We analyzed 425k favicons

#32
post #25
post #17

> In fact, I recommend that browsers ignore these hints because they are wrong much of the time. I don't agree. That's the kind of coddling that encourages incompetence. Instead of compensating for others' mistakes, just let their stuff break. I wonder if Safai on iOS ignores the hints. When I tested, I was surprised to see that pressing the share icon, which holds the option for `Add to Home Screen`, would cause a d…

People make mistakes all the time. Breaking because somebody made a mistake that you can correct for just leads to unnecessarily fragile code. What's the point of failing and breaking stuff if someone tells you their image is 144x144 but it's really 145x145? Who does that benefit?

The opposite is the case. Overall, being too lenient in what code accepts and applying heuristics will lead to way worse problems down the line. For example, you want your compiler to fail hard instead of saying: "Oh, this isn't a pointer, but I'm sure you meant well, I'm just going to treat it as a pointer!"

In this particular case, it seems to me that the hints serve no purpose and should be abolished, and in the meantime fully ignored, altogether. All necessary metadata is contained in the image file, and browsers should also be (relatively) strict in what image files with what metadata they accept, for security reasons alone.

And if they also went so far as limiting file size, the perpetrators that clog up bandwidth by putting up multi-MB favicons would catch on much earlier (or at all), too.

So what actually is the point of those hints, if browsers have to fallback anyway?

Re: We analyzed 425k favicons

#33
post #26

I got mine down to 160 bytes with some pixel tweaking and converting it to a 16-color indexed PNG. It's not a lot of work or very difficult (I'm an idiot at graphics editing), but you do need to spend the (small amount of) effort. I embed it as a data URI and it's just four lines of (col-80 wrapped) base64 text, which seems reasonable to me. Haven't managed to get my headshot down to less than 10k without looking hor…

“ I got that 280k Discord favicon down to just 24K simply by opening it in GIMP and saving it again. “

You made me laugh out loud.

I agree that stuff like YouTube.com saying 144x but really 145x seems like it should be embarrassing.

Re: We analyzed 425k favicons

#34
post #26

I got mine down to 160 bytes with some pixel tweaking and converting it to a 16-color indexed PNG. It's not a lot of work or very difficult (I'm an idiot at graphics editing), but you do need to spend the (small amount of) effort. I embed it as a data URI and it's just four lines of (col-80 wrapped) base64 text, which seems reasonable to me. Haven't managed to get my headshot down to less than 10k without looking hor…

Oh hey, Discord must have seen this article -- their favicon is down to 14k now.

Re: We analyzed 425k favicons

#35
post #26

I got mine down to 160 bytes with some pixel tweaking and converting it to a 16-color indexed PNG. It's not a lot of work or very difficult (I'm an idiot at graphics editing), but you do need to spend the (small amount of) effort. I embed it as a data URI and it's just four lines of (col-80 wrapped) base64 text, which seems reasonable to me. Haven't managed to get my headshot down to less than 10k without looking hor…

there are png optimizer programs, e.g. optipng

Re: We analyzed 425k favicons

#36
I have always wanted to do this _exact_ analysis - so awesome! Every time I am building some kind of semi-intelligent parser to fetch an arbitrary visual icon for a URL I think to myself there has gotta be a better way do do this.

Re: We analyzed 425k favicons

#37
post #21
post #17

> In fact, I recommend that browsers ignore these hints because they are wrong much of the time. I don't agree. That's the kind of coddling that encourages incompetence. Instead of compensating for others' mistakes, just let their stuff break. I wonder if Safai on iOS ignores the hints. When I tested, I was surprised to see that pressing the share icon, which holds the option for `Add to Home Screen`, would cause a d…

Browsers ignore the hints because they aren't needed. The image file itself has everything you need for rendering it.

The point for the hints is probably that the browser doesn't need to fetch the 2000×2000 favicon if it only needs something in 16×16 to render in the tab bar.

Re: We analyzed 425k favicons

#38
post #16

Not really relevant, but using Go to fetch the data, and then Ruby to process the data is the best. I used this exact set up for a project and it was amazing. Really the sweet spot of use cases for both languages.

Can you please explain why they are the best languages for these jobs?

Go's got an awesome feature set built in to the language for building small networked services. I implemented a client to a cryptocurrency network to extract information about its status and clients. I can't really express why it's so good, it just feels right.

Same for Ruby, the syntax is perfectly suited for transforming, digging through and acting upon data. I didn't even add a Gemfile, only used standard library functions, transforming the data the Go program mined into usable information serialized in JSON which was subsequently used as a static database for a webpage.

You can find the source here: https://github.com/tinco/stellar-core-go, the Go is in cmd and the Ruby is in tools.

The site it powers is now defunct, apparently they changed some stuff in the past 3 years and the crawler no longer functions.

Re: We analyzed 425k favicons

#39

Also, we turned up 2,000 domains that redirect to a very shady site called happyfamilymedstore[dot]com. Stuff like avanafill[dot]com, pfzviagra[dot]com, prednisoloneotc[dot]com. These domains made it into the Tranco 100k somehow. Full list here - https://gist.github.com/gurgeous/bcb3e851087763efe4b2f4b992f...

Does anyone know the story behind these? How do seemingly obscure sites consistently get massive amount of obscure content placed highly in results.

It happens through search engine optimization, SEO, and a mix of planting reviews and other tactics. Think of it like this - what would you do to get people talking about your site? You'd somehow put links, conversations, reviews, quotes, etc. in front of them.

Re: We analyzed 425k favicons

#40

Also, we turned up 2,000 domains that redirect to a very shady site called happyfamilymedstore[dot]com. Stuff like avanafill[dot]com, pfzviagra[dot]com, prednisoloneotc[dot]com. These domains made it into the Tranco 100k somehow. Full list here - https://gist.github.com/gurgeous/bcb3e851087763efe4b2f4b992f...

Does anyone know the story behind these? How do seemingly obscure sites consistently get massive amount of obscure content placed highly in results.

What most of them do is they will use Wordpress exploits to get into random wordpress website ran by people who know nothing about managing a website and are running on a $3/mo shared hosting account.

After they get into these random wordpress sites, then then embed links back to their sketchy site in obscure places on the wordpress site that they hacked, so that owners of the site don't notice, but search bots do. They usually leave the wordpress site alone, but will create a user account to get back into it again later if Wordpress patches an exploit. All of this exploit and link adding is automated, so it is just done by crawlers and bots.

This is done tens of thousands or even millions of times over. All of these sketchy backlinks eventually add up, even if they are low quality, and provide higher ranking for the site they all point to.

Think of websites like mommy blogs, diet diaries, family sites, personal blogs, and random service companies (plumbers, pest control, restaurants, etc) that had their nephew throw up a wordpress site instead of hiring a professional.

I don't mean to pick on wordpress, but it really is the most common culprit of these attacks. Because so many Wordpress sites exist that are operated by people who aren't informed about basic security. Plus, wordpress is open source, so exploits get discovered by looking at source code and attackers will sell those exploits instead of reporting them. So Wordpress is in an infinite cycle of chasing exploits and patching them.

Post reply on HN