On Extensions, Userscripts, and Archivers • Hacker News Board

Re: On Extensions, Userscripts, and Archivers

#31

Post by soup10 » Fri, Jul 06, 2012, 4:11 PM UTC

charliesome wrote: post #29

soup10 wrote: post #21

Earlier quoted context omitted.

Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work. I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to eac…

> All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). Congratulations, you just massively blew out your bandwidth bill. The cheapest option is to host your static content yourself, especially if you're serving over a petabyte of it per month.

It really doesn't matter where you host the static/image files, as long as it's completely separate from the application server and doesn't eat into it's resources.

Re: On Extensions, Userscripts, and Archivers

#32

Post by soup10 » Fri, Jul 06, 2012, 4:15 PM UTC

true_religion wrote: post #25

soup10 wrote: post #21

Earlier quoted context omitted.

Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work. I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to eac…

They already use CloudFlare. > In the past 30 days, CloudFlare has proxied 1,331,004,996 page views from 4chan

> Notice that I say proxied and not cached. CloudFlare does not cache HTML—every connection/request for HTML is passed on to our servers, and our server must send a response.

CloudFlare is not caching his html, thus the performance problems, because his backend is probably dog-slow.

Re: On Extensions, Userscripts, and Archivers

#33

Post by soup10 » Fri, Jul 06, 2012, 4:18 PM UTC

bsphil wrote: post #30

soup10 wrote: post #21

Earlier quoted context omitted.

Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work. I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to eac…

500 requests per second on average (far higher during peak hours) on a single box?

All you are doing is spitting out strings from memory. With a beefy box it could probably be higher than that.

Re: On Extensions, Userscripts, and Archivers

#34

Post by soup10 » Fri, Jul 06, 2012, 4:20 PM UTC

pyre wrote: post #27

soup10 wrote: post #21

Earlier quoted context omitted.

Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work. I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to eac…

> His complaint is about getting hit too hard serving > the html/text not the images. Then why is he complaining about things like image pre-loading and 'expand-all' extensions/userscripts?

Yea, I see that now. He's complaining about both. Still, asking 4chan add-on developers to be courteous seems pretty silly. Gotta detect/ban/rate-limit them on the back-end.

Re: On Extensions, Userscripts, and Archivers

#35

Post by lincolnwebs » Fri, Jul 06, 2012, 4:51 PM UTC

soup10 wrote: post #34

pyre wrote: post #27

Earlier quoted context omitted.

> His complaint is about getting hit too hard serving > the html/text not the images. Then why is he complaining about things like image pre-loading and 'expand-all' extensions/userscripts?

Yea, I see that now. He's complaining about both. Still, asking 4chan add-on developers to be courteous seems pretty silly. Gotta detect/ban/rate-limit them on the back-end.

Are you new to the Internet? Detect/ban... against 4chan users. facepalm

Re: On Extensions, Userscripts, and Archivers

#36

Post by dm8 » Fri, Jul 06, 2012, 4:59 PM UTC

Thats interesting. And 4chan had only one guy (moot) managing all of this makes it even more impressive.

By the way, some of the posters commenting on 4chan degrading morality. This is an anon forum and its just a reflection of our society! The Guardian summarized 4chan as "lunatic, juvenile... brilliant, ridiculous and alarming". Perhaps the best description for 4chan.

And remember 4chan gave us LolCats, Memes, Rage faces etc.

P.S. By the way, why did mods kill the other thread?

Re: On Extensions, Userscripts, and Archivers

#37

Post by ketralnis » Fri, Jul 06, 2012, 5:39 PM UTC

soup10 wrote: post #10

Meh, I'm always super unimpressed when simple text based websites have trouble scaling. Everything that's highly requested should be available in memory, and it should be trivial to spit it out instantly. I'm not a scaling wizard, but I'd guess 99/100 times the reason CRUD apps have problems scaling is because they are over-engineered, and there is a tendency to solve scaling issues by adding another layer of complex…

Apparently your "advice" is so naïve as to have spawned an entire other post http://news.ycombinator.com/item?id=4208134

Re: On Extensions, Userscripts, and Archivers

#38

Post by sneak » Fri, Jul 06, 2012, 6:26 PM UTC

dm8 wrote: post #36

Thats interesting. And 4chan had only one guy (moot) managing all of this makes it even more impressive. By the way, some of the posters commenting on 4chan degrading morality. This is an anon forum and its just a reflection of our society! The Guardian summarized 4chan as "lunatic, juvenile... brilliant, ridiculous and alarming". Perhaps the best description for 4chan. And remember 4chan gave us LolCats, Memes, Rage…

> And remember 4chan gave us LolCats, Memes, Rage faces etc.

4chan gave us a voice. We are now legion.

Shared culture, sure - but it showed us there was an "us" to share that culture among. Its real effects are much, much larger.

Re: On Extensions, Userscripts, and Archivers

#39

Post by trentmb » Fri, Jul 06, 2012, 8:26 PM UTC

Paul_S wrote: post #24

rdl wrote: post #23

Earlier quoted context omitted.

I think a lot of people just use ad blockers.

I've not seen an ad on the Internet since 2006. Apparently I'm evil and I'm bringing down the economy. If that means fewer jobs in advertising I count that as a win.

Adblock Plus for FireFox has an option to whitelist non-intrusive ads.

Re: On Extensions, Userscripts, and Archivers

#40

Post by kalleboo » Fri, Jul 06, 2012, 10:10 PM UTC

soup10 wrote: post #31

charliesome wrote: post #29

Earlier quoted context omitted.

> All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). Congratulations, you just massively blew out your bandwidth bill. The cheapest option is to host your static content yourself, especially if you're serving over a petabyte of it per month.

It really doesn't matter where you host the static/image files, as long as it's completely separate from the application server and doesn't eat into it's resources.

> We've also seen our image server's RAID array go from being relatively idle to getting absolutely slammed

Sounds like it is?