Live data from Hacker News

Ask HN: How much traffic do you serve and with which database engine?

news.ycombinator.com

11–20 of 39 posts

Re: Ask HN: How much traffic do you serve and with which database engine?

#11

I'll bite, just so you get a real answer instead of the very correct but annoying "don't worry about it right now" answers everyone else is going to provide! We have a rails monolith that sends our master database instance between 2,000 and 10,000 queries per second depending on the time of year. We have a seasonal bike business with more traffic in the summer. 5% of queries are insert/update/delete, the rest read. m…

wonderful, thank you. Some translations to AWS RDS...

"512gb of ram and dual EPYC 74F3 24 core processors, backed by a 4 drive raid10 nvme linux software raid volume on top of micron 9300 drives"

roughly translates to about an db.r8g.16xlarge (64 vCPUs, 512gb ram) $4,949 / month on-demand for compute

I'm not familiar enough with hardware to determine IOPS for the raid config but I believe it is greater than the maximum for io2 block express storage on aws (256k IOPS):

$0.10 per provisioned IOPS-month = 256000$.10 = $25,600 / month IOPS -- which feels high so I might be way off on the raid setup's IOPS

$0.125 per GB-month storage = 500gb $0.125 = $62.50

That's about $31,930 / month without any reserved discounts for an estimated capacity of 5,000 rps, sound about right? Would you say your total hardware cost is less than one or two months of comparable compute on AWS if the above is true?

Re: Ask HN: How much traffic do you serve and with which database engine?

#12
TiDB, 200k inserts per second; 200b per row on average. Bursty insert pattern, e.g., can have 200k inserts for an hour, then almost 0 for days. 8k reads per second on average, mostly reads by primary key. 20 hosts; 16 threads x 128GB RAM, 8TB NVME RAID 10. 60TiB of useful storage with replication factor of 3. Keyset pagination is the key. Also using rocksb for inserts batching. Costs around 20k on ovh

Re: Ask HN: How much traffic do you serve and with which database engine?

#13
I’m not going to pile on with “these are bad questions”.

We serve a few hundred concurrently online users of our web dashboard which visualises data from our ~100k online IoT device fleet.

With respect to our time series systems:

We store around 1M fields (kv pairs) per second, around 10k rows per second i.e. almost a trillion rows per day and almost a hundred trillion new data points per day.

We compress this down to around 1TB of data per day using Clickhouse and store and make available the last 30 days of data for “online analysis”.

We also store all the data we’ve ever ingested in BigQuery for M/L purposes. I shudder to think of how much this costs.

Our queries are all pretty simple and fast, just scanning and aggregating with filtering, all under 100ms.

This costs around 30k USD per month on AWS, not including networking which is too hard for me to separate from our non time series workloads.

Our infrastructure is all python and microservices, using Kafka, and under 1000 cores for the whole pipeline excluding databases.

Our databases are I believe low tens of the largest graviton instances that were available the last time we upgraded.

This is maintained by a backend team of around 15 people, and a total engineering team including hardware, data and software of around 45 people.

Re: Ask HN: How much traffic do you serve and with which database engine?

#14
post #11

I'll bite, just so you get a real answer instead of the very correct but annoying "don't worry about it right now" answers everyone else is going to provide! We have a rails monolith that sends our master database instance between 2,000 and 10,000 queries per second depending on the time of year. We have a seasonal bike business with more traffic in the summer. 5% of queries are insert/update/delete, the rest read. m…

wonderful, thank you. Some translations to AWS RDS... "512gb of ram and dual EPYC 74F3 24 core processors, backed by a 4 drive raid10 nvme linux software raid volume on top of micron 9300 drives" roughly translates to about an db.r8g.16xlarge (64 vCPUs, 512gb ram) $4,949 / month on-demand for compute I'm not familiar enough with hardware to determine IOPS for the raid config but I believe it is greater than the maxim…

Yup, last time I priced this in RDS I got to maybe $20k a month for two reserved instances across AZs.

I pay for our rack outright every 3-4 months from what I can tell. Still takes the same number of infra/ops/sre people as well. We staff 2, but really just have 1.25 worth of FTE work, you just need more for redundancy.

Pretty nuts! This is also why I am so dismissive of performance optimization. Yeah, I'll just buy a new set of three machines with 2tb of ram each in a few years and call it good, still come out ahead.

Re: Ask HN: How much traffic do you serve and with which database engine?

#15
post #7
post #3

Asking about customers is the wrong question. 1. It's often information that cannot be casually shared for legal reasons (MNPI) 2. A single customer might generate many queries. There have been times where a single one of my employer's customers generates more traffic than most companies will ever reach at peak.

Fair. Please interpret as queries rather than customers.

This is also the wrong question because the load of a query can vary drastically based on the query's structure/complexity, the size and types of the data, and which indexes and tables are used.

The right question is - discuss a problem you'd like to solve using PostgreSQL and find people who have solved similar problems.

Re: Ask HN: How much traffic do you serve and with which database engine?

#16

You’re asking the wrong questions. Query complexity and patterns matter. Nobody can answer this for you. You have to do the analysis based on your workload.

Agreed and thank you for pointing that out. I regret including my own app details as they were a distraction from the main question, I was mainly looking for other peoples' real world experiences and numbers, not help with my own app.

While reviewing IoT databases I saw a lot of discussion here and elsewhere about database performance and wanted to see some numbers. Of course, applications with different workloads will have different database needs, but it is still helpful to hear that 5000 rps were served with ABC database with XYZ hardware assuming that the majority of queries on the majority of apps are simple lookups.

Re: Ask HN: How much traffic do you serve and with which database engine?

#17
post #15
post #7

Earlier quoted context omitted.

Fair. Please interpret as queries rather than customers.

This is also the wrong question because the load of a query can vary drastically based on the query's structure/complexity, the size and types of the data, and which indexes and tables are used. The right question is - discuss a problem you'd like to solve using PostgreSQL and find people who have solved similar problems.

Thank you for pointing that out, I agree. I'm just looking for peoples' experiences with their applications, which of course all have different workloads.

People did get the spirit of the question and have provided helpful responses, including context about their query structure/complexity.

Re: Ask HN: How much traffic do you serve and with which database engine?

#18
post #2

I think you might be asking the wrong questions. They key questions are Queries per second and the median response size of the query. For example at reddit (15 years ago) we had 10x more vote traffic than comment traffic, but we only needed two databases to handle votes (technically only one the other was just for redundancy). But we needed nine comments databases. Mainly because the median query response was so much…

Thank you very much! Did you start with separate databases or wait until you needed to scale workloads separately before breaking things up?

Re: Ask HN: How much traffic do you serve and with which database engine?

#19
post #11

Earlier quoted context omitted.

wonderful, thank you. Some translations to AWS RDS... "512gb of ram and dual EPYC 74F3 24 core processors, backed by a 4 drive raid10 nvme linux software raid volume on top of micron 9300 drives" roughly translates to about an db.r8g.16xlarge (64 vCPUs, 512gb ram) $4,949 / month on-demand for compute I'm not familiar enough with hardware to determine IOPS for the raid config but I believe it is greater than the maxim…

Yup, last time I priced this in RDS I got to maybe $20k a month for two reserved instances across AZs. I pay for our rack outright every 3-4 months from what I can tell. Still takes the same number of infra/ops/sre people as well. We staff 2, but really just have 1.25 worth of FTE work, you just need more for redundancy. Pretty nuts! This is also why I am so dismissive of performance optimization. Yeah, I'll just buy…

Much appreciated, thank you and congratulations on the bike business

Re: Ask HN: How much traffic do you serve and with which database engine?

#20
Let's do some math.

First, the IOT devices reporting daily. In the absence of further context, I’m going to assume that it doesn’t matter when the devices report, so they can be configured to spread out their load. I’m also going to assume 1kb of data per device, but with an HTTPS API there’s roughly 7kb of overhead that we need to account for when calculating bandwidth. (Source: http://netsekure.org/2010/03/tls-overhead/ . TLS session resumption gets that down to ~300 bytes, but adds implementation complications.)

    $ units -1v '1M req/day' 'req/sec'
        1M req/day = 11.574074 req/sec
    $ units -1v '1M req/day * 8kbyte/req' 'kbyte/sec' # Incoming bandwidth
        1M req/day * 8kbyte/req = 92.592593 kbyte/sec
    $ units -1v '1M req/day * 1kbyte/req' 'years/TB' # Storage
        reciprocal conversion
        1 / (1M req/day * 1kbyte/req) = 2.7379093 years/TB
It looks like our load here is a whopping 12 RPS, and we could handle the traffic on an ISDN line from the 1990s. Data storage is a little trickier; if we can’t compress or delete old data we may have to stop by Best Buy for a new hard drive every half decade or so.

Users can’t be configured to load balance themselves, so we’ll be pessemistic and assume that every single one of them logs in to check their device over their morning coffee. We’ll also assume that every time they do that they want all of the data from the last 3 months, though in practice this could probably be summarized before we send it to them.

    $ units -1v '5000 req/1 hour' 'req/sec'
        5000 req/1 hour = 1.3888889 req/sec
    $ units -1v '5000 req/1 hour * 90 kbyte/req' 'MB/sec' # Outgoing bandwidth
        5000 req/1 hour * 90 kbyte/req = 0.125 MB/sec
For this, we have just under 2 RPS, but our responses are quite a lot bigger so the bandwidth is higher—we probably want to move into the early 2000s and upgrade to a DSL connection. Oh, and we also want to make sure our disk can handle the read load—but conveniently, since we’re just pulling raw data and not asking for any complicated joins, these numbers are actually the same. 2 RPS gives us 500ms per request; since a spinning rust drive pessimistically takes 10ms/seek we only have a ~50-seek budget per request so we probably want to make sure to cluster or partition the table by device to improve data locality. (Doing that increases the write load significantly, though, so maybe we want to upgrade to an SSD or think about a database that’s smart enough to do some kind of periodic bulk rebalancing.)

Oh, I almost forgot, we'll also want to make sure we have disk space to keep track of all of those idle logins who aren’t doing anything:

    $ units -1v '1M records * 1kbyte/rec' 'GB'
        1M records * 1kbyte/rec = 1 GB
Modern computers—for very relative definitions of modern—are fast, if they don’t get bogged down. Based on my numbers, you could probably run your system from a Macbook in your desk drawer; but it would be trivial to add a requirement which would multiply any of these numbers by several orders of magnitude. The problem with trying to compare architectures is that you have to really understand what your needs are, and not only are they going to be difficult to compare to someone else’s setup (small differences can add up fast when multiplied across requests) but it’s also hard to tell how much effort the other system put into optimizing (or, for that matter, how much effort you want to put into optimizing).
Post reply on HN