Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preventing server overload: limit requests being processed

evanjones.ca

Post reply on HN
Print view View original

51–60 of 69 posts

Re: Preventing server overload: limit requests being processed

#51

Post by Animats »

thunderbong wrote: post #50
Animats wrote: post #35

I've used fair queuing to solve this problem. I have a site and API which rates other sites, checking them for ad links and trying to match them to real world business records. Inspecting a site takes 10 seconds to 2 minutes. The API, which is usually used from browser add-ons which tag web search results, lets anyone request the rating info for a site. Most requests are already cached and return immediately. Request…

That's really interesting. Can you tell more about how it is implemented technology wise?

Python and MySQL/MariaDB. The queue is an in-memory table. Fair queuing is implemented as a big SQL SELECT statement.

The time scale is slow enough that polling it once a second for new work is sufficient. If it had to go fast, it would require something like a standard fair queuing implementation, where you hash the IP address and put the request onto 1 of N queues, serviced round-robin.

Re: Preventing server overload: limit requests being processed

#52

Post by tedunangst »

wutbrodo wrote: post #37

> Over Quota This application is temporarily over its serving quota. Please try again later. I'm actually not super sure is this is a single-serving joke site like isthemissiononfire.com, or if this is just a hilarious coincidence.

What host is that? GCE? How much quota do you need for a tech blog (as opposed to kardashian watch or whatever)?

Re: Preventing server overload: limit requests being processed

#53

Post by javajosh »

thunderbong wrote: post #50
Animats wrote: post #35

I've used fair queuing to solve this problem. I have a site and API which rates other sites, checking them for ad links and trying to match them to real world business records. Inspecting a site takes 10 seconds to 2 minutes. The API, which is usually used from browser add-ons which tag web search results, lets anyone request the rating info for a site. Most requests are already cached and return immediately. Request…

That's really interesting. Can you tell more about how it is implemented technology wise?

Doesn't matter all that much. For ingestion, you might have one VM running one Java process and one DB process as a microservice (I like Dropwizard and PostgreSQL for this). The worker itself would be another Java process with a 100 thread max thread pool that either periodically checks for new work, or uses the new NOTIFY semantics new to Postgres 9.6. In this case the "secret sauce" is that the worker has some extra smarts to reorder the queue to avoid resource hogs, which is the cool part.

Re: Preventing server overload: limit requests being processed

#54

Post by HappyDeveloper »

wutbrodo wrote: post #37

> Over Quota This application is temporarily over its serving quota. Please try again later. I'm actually not super sure is this is a single-serving joke site like isthemissiononfire.com, or if this is just a hilarious coincidence.

I thought that was pretty funny lol! They probably weren't expecting all that traffic from hacker news

Re: Preventing server overload: limit requests being processed

#55

Post by lhuser123 »

HappyDeveloper wrote: post #54
wutbrodo wrote: post #37

> Over Quota This application is temporarily over its serving quota. Please try again later. I'm actually not super sure is this is a single-serving joke site like isthemissiononfire.com, or if this is just a hilarious coincidence.

I thought that was pretty funny lol! They probably weren't expecting all that traffic from hacker news

Indeed, very funny :)

Re: Preventing server overload: limit requests being processed

#56

Post by robbles »

jonhohle wrote: post #44
kyrra wrote: post #5

[I'm a Google employee, opinions are my own] The GCP team did a blog post similar to this about using Load Shedding to survive a spike[0]. It's definitely a great way to survive a sudden spike in traffic, but not optimal, as serving errors is also bad (unless your clients have some sort of retry mechanism in-place). But if your clients do have a retry mechanism in place (especially one that has a backoff time on retr…

Retries aren't the only option if you control (or can at least influence) the client. Fallback sources and sane (or customer preferential) defaults are some additional options when the primary source of truth is unavailable. Both of these bypass thundering heard situations where recovery can be difficult even if the application returns to full health. As a concrete example, when I worked at a large online retailer, w…

Why wouldn't you start with the cache?

Isn't the consistent database only useful if you require it to be used, and don't fall back to a less consistent alternative?

Re: Preventing server overload: limit requests being processed

#57

Post by boulos »

For those asking, this seems to be hosted on App Engine and he's gone over his quota. App Engine's throttling is both a blessing and a curse (presumably he's using the free tier and never noticed this). You'll note that it's explicitly saying he's over quota not that it can't handle it ;).

Disclosure: I work on Google Cloud.

Re: Preventing server overload: limit requests being processed

#58

Post by sitkack »

kyrra wrote: post #5

[I'm a Google employee, opinions are my own] The GCP team did a blog post similar to this about using Load Shedding to survive a spike[0]. It's definitely a great way to survive a sudden spike in traffic, but not optimal, as serving errors is also bad (unless your clients have some sort of retry mechanism in-place). But if your clients do have a retry mechanism in place (especially one that has a backoff time on retr…

Isn't the other option serving errors to all clients? What are the options for

    * currently serving 1k clients
    * can serve 10k more
    * see a spike in 50k clients
I'd rather serve 11k clients and tell 39k of them to try again later (with enough capacity to tell them to try again).

Re: Preventing server overload: limit requests being processed

#59

Post by sitkack »

evanj wrote: post #29
kailuowang wrote: post #9

I developed a library to guard against server overload. It automatically gauges the capacity and rejects requests exceeding it. http://iheartradio.github.io/kanaloa/docs/theories.html

Awesome this is exactly the sort of system I had in mind when I wrote that article, I'll take a look, thanks!

Logged in users should take priority, depending on the number of page requests, all users will see an error at some point breaking the site for everyone.
Post reply on HN