Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lies, Damn Lies and Database Benchmarks

questdb.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Lies, Damn Lies and Database Benchmarks

#21

Post by hilariously »

fragmede wrote: post #18
hilariously wrote: post #16

Earlier quoted context omitted.

Yeah, the tl;dr is that benchmarking is freaking hard because what you actually care about is "does my workload today and in the future run better or worse given current setup?" but identifying what your workload actually is, what systems you are going to be allowed to run it on, what tweaks would even be possible if you know the interiors of a system and how it aligns with your hardware, and it all comes with the pr…

Yeah, but also, I want to know the p50 warm performance, not just the p99. Run the same query twice in a row after cold start. And then another 10 times. Then do another different set of queries and at the end of the day, or a week, still have no real idea how the system will perform in prod for your particular use case. Benchmarking is hard , no argument from me!

Yep, I actually want to know the system has some sort of baseline performance that only hockey sticks under conditions I can monitor and control... but also the business wants to try new feature X and vendor is promising new performance for feature Y, and new patches are coming in affecting ???.

Re: Lies, Damn Lies and Database Benchmarks

#22

Post by SOLAR_FIELDS »

bitlad wrote: post #2

Reminds me of the recent Terminal Bench controversy [1][2][3] If theres a benchmark, people will cheat, lie and optimize for that benchmark. Honest depends on the compliance enforced on teams. But if, compliance itself is weak, it is going to be taken advantage of. Like growing up india, you would optimize for the exam and not what you learn from it. [1] https://news.ycombinator.com/item?id=47920787 [2] https://www.t…

There's just too many nuances to take any measurement less than an order of magnitude of difference seriously without further investigation. Even a 2x can be a simple configuration change for these things. Usually the differences of >1 order of magnitude are enough that you cannot hand wave away the difference without a grossly obvious oversight in configuration

Re: Lies, Damn Lies and Database Benchmarks

#23

Post by gandreani »

hansvm wrote: post #19
jaapz wrote: post #8

Anyone here using QuestDB in production? What is your use case? What is your experience? We want to migrate away from InfluxDB eventually (because of their 180 on OSS, and their tendency to reinvent the product every major release), and QuestDB seems like an interesting option.

I used it for awhile for "prod" for a toy project. I was scraping nearly every sale across every Target location for awhile. It was fast and easy to use. At the time many (read) queries had bugs of various kinds, requiring strange workarounds to get certain joins and whatnot to work (and not just throwing errors either -- often reporting zero data when there was data there, things like that) and not really being comp…

Man I have the strangest Deja Vu with this comment. I swear it's like the third time I read this??

Re: Lies, Damn Lies and Database Benchmarks

#24

Post by simplesocieties »

jaapz wrote: post #8

Anyone here using QuestDB in production? What is your use case? What is your experience? We want to migrate away from InfluxDB eventually (because of their 180 on OSS, and their tendency to reinvent the product every major release), and QuestDB seems like an interesting option.

Been using it for half a year now in prod to collect sensor data from IoT devices.

My only complaints are:

1) Memory usage is a bit high. We went with the AWS instance they recommended in the docs and even that went over our provisioned memory. It's not much but I think it could be improved

2) You need to buy their enterprise plan if what you're storing is remotely sensitive like health data, PII, etc. Any row level security or credential features are locked behind that license. Our use case isn't that sensitive so we can get away with putting it in a VPN and password protecting it, but if you need DB-level security the FOSS license is severely behind Postgres in terms of features.

Other than that, it's never gone down, it's very, very fast and comes with it's own webui for querying your data. We migrated from AWS Timestream and couldn't be happier with the switch.

Re: Lies, Damn Lies and Database Benchmarks

#26

Post by hansvm »

gandreani wrote: post #23
hansvm wrote: post #19

Earlier quoted context omitted.

I used it for awhile for "prod" for a toy project. I was scraping nearly every sale across every Target location for awhile. It was fast and easy to use. At the time many (read) queries had bugs of various kinds, requiring strange workarounds to get certain joins and whatnot to work (and not just throwing errors either -- often reporting zero data when there was data there, things like that) and not really being comp…

Man I have the strangest Deja Vu with this comment. I swear it's like the third time I read this??

At most the second from me, and last time I didn't talk about the project (haven't talked about the project online at all IIRC).

Re: Lies, Damn Lies and Database Benchmarks

#27

Post by fragmede »

hilariously wrote: post #21
fragmede wrote: post #18

Earlier quoted context omitted.

Yeah, but also, I want to know the p50 warm performance, not just the p99. Run the same query twice in a row after cold start. And then another 10 times. Then do another different set of queries and at the end of the day, or a week, still have no real idea how the system will perform in prod for your particular use case. Benchmarking is hard , no argument from me!

Yep, I actually want to know the system has some sort of baseline performance that only hockey sticks under conditions I can monitor and control... but also the business wants to try new feature X and vendor is promising new performance for feature Y, and new patches are coming in affecting ???.

And oh, you just have to set this one setting that you need to pay a consultant to know all the settings in order to tweak and tune the database for your use case. Still the TPC competitions such that oracle won because they managed to hand two assembly from five instructions down to four as an impressive bit of computer engineering nerditry.

Re: Lies, Damn Lies and Database Benchmarks

#28

Post by gandreani »

hansvm wrote: post #26
gandreani wrote: post #23

Earlier quoted context omitted.

Man I have the strangest Deja Vu with this comment. I swear it's like the third time I read this??

At most the second from me, and last time I didn't talk about the project (haven't talked about the project online at all IIRC).

I believe you! I wasn't implying anything about your project. I just can't quite shake the feeling.

Re: Lies, Damn Lies and Database Benchmarks

#29

Post by Devont »

simplesocieties wrote: post #25

Having Deja Vu reading this. Remember just recently when Spacetime DB fudged their benchmark numbers for their 2.0 release. https://www.youtube.com/watch?v=C7gJ_UxVnSk

SpacetimeDBs “benchmarks” might be the most egregious, blatantly misleading benchmark I’ve ever seen. Lost a lot of respect for the team behind it

Re: Lies, Damn Lies and Database Benchmarks

#30

Post by hansvm »

gandreani wrote: post #28
hansvm wrote: post #26

Earlier quoted context omitted.

At most the second from me, and last time I didn't talk about the project (haven't talked about the project online at all IIRC).

I believe you! I wasn't implying anything about your project. I just can't quite shake the feeling.

I'm probably not the only culprit then :) I'd be curious to know who else has done things like that.
Post reply on HN