Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On Extensions, Userscripts, and Archivers

content.4chan.org

Post reply on HN
Print view View original

21–30 of 41 posts

Re: On Extensions, Userscripts, and Archivers

#21

Post by soup10 »

unimpressive wrote: post #11
soup10 wrote: post #10

Meh, I'm always super unimpressed when simple text based websites have trouble scaling. Everything that's highly requested should be available in memory, and it should be trivial to spit it out instantly. I'm not a scaling wizard, but I'd guess 99/100 times the reason CRUD apps have problems scaling is because they are over-engineered, and there is a tendency to solve scaling issues by adding another layer of complex…

4chan is an image board. None of the content is there long enough to be considered "Highly requested.". And like half the posts have jpeg's and png's attached. Besides, how does caching solve the "My bandwidth bills are killing my wallet!" issue?

Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work.

I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to each box, add it to a table/indexes in memory and write the post to disk and backups for recovery purposes. Then either update and cache every page the new post affects, or mark the pages dirty and update and cache them the next time they are requested.

Done, all pages served out of memory super fast, what am I missing?

As far as bandwidth bills, most browsers observe cache settings and won't re-download what it has already downloaded. His complaint is about getting hit too hard serving the html/text not the images.

Re: On Extensions, Userscripts, and Archivers

#22

Post by phpnode »

ramblerman wrote: post #13
rdl wrote: post #4

The main problem seems to be revenue. With 4chan, you could probably do a unique form of advertising -- companies pay ~$100/cpm to NOT have their products or brand featured on any 4chan pages. :)

Lol, In all seriousness though I'm pretty sure they could still find regular customers without too much trouble. Not to perpetuate any stereotypes but fleshlight / masculine enlargement products and so forth would be happy to advertise there.

I'd imagine that kind of adult product would perform pretty badly on 4chan. The audience is definitely not in buying mode when they're there, and it's not like most people are going there explicitly for porn, so they don't have the kind of "motivation" that drives that kind of purchase. Probably things like cool / novelty items would do better there, things that appeal to teenagers with not too much money.

Re: On Extensions, Userscripts, and Archivers

#23

Post by rdl »

phpnode wrote: post #22
ramblerman wrote: post #13

Earlier quoted context omitted.

Lol, In all seriousness though I'm pretty sure they could still find regular customers without too much trouble. Not to perpetuate any stereotypes but fleshlight / masculine enlargement products and so forth would be happy to advertise there.

I'd imagine that kind of adult product would perform pretty badly on 4chan. The audience is definitely not in buying mode when they're there, and it's not like most people are going there explicitly for porn, so they don't have the kind of "motivation" that drives that kind of purchase. Probably things like cool / novelty items would do better there, things that appeal to teenagers with not too much money.

I think a lot of people just use ad blockers.

Re: On Extensions, Userscripts, and Archivers

#24

Post by Paul_S »

rdl wrote: post #23
phpnode wrote: post #22

Earlier quoted context omitted.

I'd imagine that kind of adult product would perform pretty badly on 4chan. The audience is definitely not in buying mode when they're there, and it's not like most people are going there explicitly for porn, so they don't have the kind of "motivation" that drives that kind of purchase. Probably things like cool / novelty items would do better there, things that appeal to teenagers with not too much money.

I think a lot of people just use ad blockers.

I've not seen an ad on the Internet since 2006. Apparently I'm evil and I'm bringing down the economy. If that means fewer jobs in advertising I count that as a win.

Re: On Extensions, Userscripts, and Archivers

#25

Post by true_religion »

soup10 wrote: post #21
unimpressive wrote: post #11

Earlier quoted context omitted.

4chan is an image board. None of the content is there long enough to be considered "Highly requested.". And like half the posts have jpeg's and png's attached. Besides, how does caching solve the "My bandwidth bills are killing my wallet!" issue?

Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work. I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to eac…

They already use CloudFlare.

> In the past 30 days, CloudFlare has proxied 1,331,004,996 page views from 4chan

Re: On Extensions, Userscripts, and Archivers

#26

Post by [deleted] »

soup10 wrote: post #21
unimpressive wrote: post #11

Earlier quoted context omitted.

4chan is an image board. None of the content is there long enough to be considered "Highly requested.". And like half the posts have jpeg's and png's attached. Besides, how does caching solve the "My bandwidth bills are killing my wallet!" issue?

Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work. I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to eac…

[deleted]

Re: On Extensions, Userscripts, and Archivers

#27

Post by pyre »

soup10 wrote: post #21
unimpressive wrote: post #11

Earlier quoted context omitted.

4chan is an image board. None of the content is there long enough to be considered "Highly requested.". And like half the posts have jpeg's and png's attached. Besides, how does caching solve the "My bandwidth bills are killing my wallet!" issue?

Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work. I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to eac…

  > His complaint is about getting hit too hard serving
  > the html/text not the images.
Then why is he complaining about things like image pre-loading and 'expand-all' extensions/userscripts?

Re: On Extensions, Userscripts, and Archivers

#28

Post by TazeTSchnitzel »

gcb wrote: post #20
MichaelGG wrote: post #17

"despite having a bunch of fast drives in RAID 10 and 64 GB of RAM, our image server simply cannot keep up. " Wait, so he only has a single image server?

It's not like a site that forgets it's contents in 2min or less needs more than that.

It's not really 2 minutes, it's far more than that (a thread I watched lasted >1 day), but a LOT of content drops off the site every second.

Re: On Extensions, Userscripts, and Archivers

#29

Post by charliesome »

soup10 wrote: post #21
unimpressive wrote: post #11

Earlier quoted context omitted.

4chan is an image board. None of the content is there long enough to be considered "Highly requested.". And like half the posts have jpeg's and png's attached. Besides, how does caching solve the "My bandwidth bills are killing my wallet!" issue?

Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work. I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to eac…

> All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest).

Congratulations, you just massively blew out your bandwidth bill. The cheapest option is to host your static content yourself, especially if you're serving over a petabyte of it per month.

Re: On Extensions, Userscripts, and Archivers

#30

Post by bsphil »

soup10 wrote: post #21
unimpressive wrote: post #11

Earlier quoted context omitted.

4chan is an image board. None of the content is there long enough to be considered "Highly requested.". And like half the posts have jpeg's and png's attached. Besides, how does caching solve the "My bandwidth bills are killing my wallet!" issue?

Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work. I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to eac…

500 requests per second on average (far higher during peak hours) on a single box?
Post reply on HN