Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

highscalability.com

Post reply on HN
Print view View original

11–20 of 45 posts

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#11

Post by colanderman »

> context switches passing tcp-packets back and forth from the operating system were taking up 35%

You could get rid of this (and in doing so, double your TPS) by switching to a memory-polling-based network driver like PF_RING [1] (and obviously, keeping the kernel on its own core like you are doing).

> Lookup data in memory (this is fast enough to happen in-thread)

> I knew I had it right when I watched the output of the “top” command,

Does all your data live in cache? If not, have you tried using perf [2] to measure load stalls? They are typically the bottleneck once you get past context switches. Hyperthreading should help at least somewhat here (do you have it enabled?).

[1] http://www.ntop.org/products/pf_ring/

[2] https://perf.wiki.kernel.org/

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#12

Post by colanderman »

sillysaurus wrote: post #4

Is there a domain where this is applicable outside of high performance trading? I'm trying to think of another use case that would legitimately generate 1M queries/sec.

Intrusion protection systems must legitimately handle upwards of 15M queries/sec when under attack (on a 10 Gbps link).

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#14

Post by recuter »

derefr wrote: post #9
sillysaurus wrote: post #4

Is there a domain where this is applicable outside of high performance trading? I'm trying to think of another use case that would legitimately generate 1M queries/sec.

Some of the larger combats (5000+ ships) in EVE Online might do that. (I single EVE out because other MMOs generally shard by user-cohort, so having that number of people on one shard is impossible. EVE, meanwhile, shards by location within the virtual world (each star system is a shard), so the entire player-base can "gather" on a single shard for a confrontation.)

> In February 2013, EVE Online reached over 500,000 subscribers.

So what, the entire game userbase needs to be awake and in the same spot? :0)

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#15

Post by pwelch »

derefr wrote: post #9
sillysaurus wrote: post #4

Is there a domain where this is applicable outside of high performance trading? I'm trying to think of another use case that would legitimately generate 1M queries/sec.

Some of the larger combats (5000+ ships) in EVE Online might do that. (I single EVE out because other MMOs generally shard by user-cohort, so having that number of people on one shard is impossible. EVE, meanwhile, shards by location within the virtual world (each star system is a shard), so the entire player-base can "gather" on a single shard for a confrontation.)

I've never played EVE Online but from what I have read about it the engineering behind it seems interesting. Have any suggests for Dev Blog posts from their team?

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#16

Post by jonahx »

colanderman wrote: post #12
sillysaurus wrote: post #4

Is there a domain where this is applicable outside of high performance trading? I'm trying to think of another use case that would legitimately generate 1M queries/sec.

Intrusion protection systems must legitimately handle upwards of 15M queries/sec when under attack (on a 10 Gbps link).

I'm curious, can you elaborate on this? What would they be querying? What kind of intrusion and what kind of attack?

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#17

Post by voltagex_ »

recuter wrote: post #14
derefr wrote: post #9

Earlier quoted context omitted.

Some of the larger combats (5000+ ships) in EVE Online might do that. (I single EVE out because other MMOs generally shard by user-cohort, so having that number of people on one shard is impossible. EVE, meanwhile, shards by location within the virtual world (each star system is a shard), so the entire player-base can "gather" on a single shard for a confrontation.)

> In February 2013, EVE Online reached over 500,000 subscribers. So what, the entire game userbase needs to be awake and in the same spot? :0)

It's getting there: http://wiki.eveonline.com/en/wiki/Major_Fleet_Engagements

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#18

Post by [deleted] »

recuter wrote: post #14
derefr wrote: post #9

Earlier quoted context omitted.

Some of the larger combats (5000+ ships) in EVE Online might do that. (I single EVE out because other MMOs generally shard by user-cohort, so having that number of people on one shard is impossible. EVE, meanwhile, shards by location within the virtual world (each star system is a shard), so the entire player-base can "gather" on a single shard for a confrontation.)

> In February 2013, EVE Online reached over 500,000 subscribers. So what, the entire game userbase needs to be awake and in the same spot? :0)

[deleted]
Post reply on HN