Live data from Hacker News

TimescaleDB vs. Amazon Timestream

blog.timescale.com

161–170 of 203 posts

Re: TimescaleDB vs. Amazon Timestream

#161
post #136

Earlier quoted context omitted.

On our managed Timescale Forge, it's multiple AZ. (On some benchmarking equipment on Digital Ocean, it's advertised as "multiple racks", but managed to reduce blast radius.)

This contradicts what is in the blog post. From the machine configuration section: > 1 remote client machine, 1 database server, both in the same cloud datacenter > Disk Size: 4.8TB of disk in a raid0 configuration (EXT4 filesystem) Both those statements lead me to believe it's a single server with locally attached SSDs in a RAID0. Which is it? I know benchmarking is hard, and it's difficult to test certain aspects o…

It's raided network-attached block storage. We'll update the blog post to make that clear.

Re: TimescaleDB vs. Amazon Timestream

#162
post #151

Earlier quoted context omitted.

Sorry, the previous reply suggested it was on row_id, time I thought, anyway. I think the question is why should row_id be unique? What looks it up by row_id? And can they look it up by user_id/time instead? It may be impossible. On the other hand, if row_id is really what you're searching by, not by time, then just use our partitioning on row_id, assuming it's a bigint or something like that, you can just partition…

No problem, my fault I misread your comment. `row_id` is a unique identifier for the row, for example my API will need it when the user wants to delete the specific row. Since it's possible for multiple rows to have the same `time`, even for the same `user_id`, I cannot assume uniqueness there. Partitioning on the row_id by making it an ordinal instead of a UUID could work, however I feel I would be missing out on Ti…

I mean, I'm not sure this is the place to get into detailed schema discussions, but I think I wouldn't use a row_id here at all, certainly, there's no need for this to be unique across all of the rows, you could make a unique constraint on user_id, time, row_id and then provide all 3 of them for the deletes, if the only reason is for the deletes, then that should be fine. If there are other reasons, then you probably have a different unique constraint that has some real world meaning (and that likely involves time).

Re: TimescaleDB vs. Amazon Timestream

#163
post #82

Earlier quoted context omitted.

That's surprising to me since Postgres does less efficient writes compared to MySQL in order to optimize for read queries[1] - a sensible tradeoff since most of the time reads are more common than writes in OLTP workloads. TimescaleDB essentially solves this problem for the specific case of time-series inserts. [In Postgres] if we have a table with a dozen indexes defined on it, an update to a field that is only cove…

The parent post didn't have enough detail, but I'd assume the dump would be using COPY for inserts. AFAIK when using COPY, the indexes are only built at the end of the command, not on each insert.

(Just noticed this was about dump/restore, during restores you can in fact write the data and then build the index in a separate operation and that can be faster, sorry for missing context)

Re: TimescaleDB vs. Amazon Timestream

#164

Earlier quoted context omitted.

I know the first thing that shocked me was how they could get so close to something that was purpose built for time series. Even being within spitting distance is really great in my opinion for an off the shelf, general tool. > However none of the databases tested above is anywhere close to the efficiency of a column store database that can do vectorized execution over batches of rows. ClickHouse is a good example of…

I'm out of my depth here (not a db guy but curious), so forgive the interruption, but have either of you messed with either SciDB or VoltDB? I have found them to be quite interesting especially given Stonebraker himself being involved in the design. Druid and timescale have also been on my radar too, and I think these comments have finally given me the push to play around with them in my sandbox.

I've never even heard of SciDB, thanks for the pointer. I have heard of VoltDB and the only thing I remembered was that it was in-memory...

I'm still not 100% sure what SciDB is for but it looks like none of the databases we're mentioning are directly comparable:

- SciDB for scientific computing (??)

- VoltDB for in memory + SQL

- Druid for OLAP queries

- TimescaleDB for row-based timeseries storage

Seems like all apples to oranges to me. I personally like TimescaleDB because it runs on Postgres and I can usually find a way to do all those other things in postgres relatively efficiently (memory requires some squinting while using UNLOGGED tables).

Re: TimescaleDB vs. Amazon Timestream

#165
post #57

If you take one thing away from this it's do not use AWS timestream, and if you are using it, get rid of it quick. It's unfit for purpose. That first graph tells you everything you need to know. Time to insert a billion events: TimescaleDB: 5 min AWS Timestream: ~2 weeks! I think the team at AWS that built it should be reassigned and contractors or an A team brought in to try and salvage it. Because they've built the…

> I think the team at AWS that built it should be reassigned and contractors or an A team brought in to try and salvage it That's a knee-jerk assessment. You don't know the first thing about Timestream's development and we are only a few years into the development process of what I believe is a novel architecture for a timeseries database. Give it another couple of years and then we can take a look at how Timestream…

Yeah, it's harsh. But we're talking 6000x worse ingest performance for a database whose whole raison d'etre is based on ingesting high volumes of data - otherwise just use an RDBMS.

That's like making a car with square wheels. The only logical thing for management to do to a team who delivered that is disband them - because something is horribly wrong.

Now to be completely fair they do get good ingest performance if you open thousands of connections to send the data over. So they can probably fix it. I still wouldn't touch it with a stick though, that's not the only issue they have at present.

Re: TimescaleDB vs. Amazon Timestream

#166
post #57

If you take one thing away from this it's do not use AWS timestream, and if you are using it, get rid of it quick. It's unfit for purpose. That first graph tells you everything you need to know. Time to insert a billion events: TimescaleDB: 5 min AWS Timestream: ~2 weeks! I think the team at AWS that built it should be reassigned and contractors or an A team brought in to try and salvage it. Because they've built the…

(Post author from Timescale)

For the record, in reviewing HN conversation tonight I saw this and realized it was an incorrect quote of the article. Totally honest mistake I'm sure, but I wanted to set the record straight.

We spent a little over a week working at Timestream benchmarking. Trying different approaches to batching metrics for ingest, threading differently, running multiple EC2 instance, etc. to improve performance.

Once we felt like we were getting the best we could, we started our final ingest which we let run for nearly 40 hours (~2 days).

The other value is absolutely correct, however. We were able to ingest 1 billion metrics into Timescale in 5 minutes.

Re: TimescaleDB vs. Amazon Timestream

#167

Disclosure: I work at AWS but not on Timestream. Opinions my own. Unless I'm missing something this is not an apples to apples benchmark. TimescaleDB is running as a single node without any replication whereas Amazon Timestream is replicated[0] to three AWS Availability Zones for durability. I've only skimmed the TSBS[1] repo and the start/stop scripts for TimescaleDB. Can someone confirm this? 0 - https://aws.amazon…

Would that really explain the difference between 5 minutes on Timescale and 2 weeks on Timestream? Timeseries databases need the ability to ingest data rapidly.

Please see below for correction... but at some point today a poster said "~2 weeks" for Timestream... but that's incorrect. We ran the data load/ingest for "~2 days" (40 hours).

Just want to make sure the right numbers are being used! Thanks!

(Post author @Timescale)

Re: TimescaleDB vs. Amazon Timestream

#168

Earlier quoted context omitted.

(Disclaimer: post author and Timescale employee) I'm sorry you feel like we were trying to be dishonest in the post. On the contrary, we put a lot of effort (and 7,000+ words) into trying to explain everything that we did - just as we've done with other benchmarks which others have linked to. The TimescaleDB test did not use continuous aggregates for these test, only raw time-series data stored in hypertables. For ea…

Sorry for the harsh comment. I've been reading about your columnar compression pipeline [1], and it sort of makes sense if the comparison is against a regular row-oriented DB. AWS Timestream must really be doing something wrong here, or serving an entirely different use case. 5-175x faster queries and 150x-220x cheaper I do get it. But 6000x higher inserts does not make sense to me. It is insane, and literally unbeli…

We're happy for people to poke at this and helping us to improve. It's obviously hard to work at something for weeks, see the numbers (even knowing you really tried for days to move the needle) and then still publish numbers that seem impossible. And again, if you look at TSBS, this isn't the first time we've run benchmarks on other databases, so we were just as shocked and put extra effort into it.

In the end, if you read the article (and not just the headlines - not saying you are, but it's easy to see 6000x and latch on to it), the comparison is absolutely focused on this one, pretty straight forward use case (although we normally run 5 different scenarios):

From one client, given a specific kind of workload (100 hosts, 10 CPU metrics every 10 seconds for 30 days = ~1 billion metrics) - how fast could we save the data. Most other time series databases at least perform marginally well with the same setup... load data with one client.

But Timestream just doesn't seem setup to work that way. Some of the responses today imply that we need really large clients with thousands of threads to get those speeds. And that might work if we kept going and spent more time and significantly more money. We just haven't ever had to do that before.

If your use case better aligns with what Timestream offers, then it might be a great product for you. Given some of the many other concerns we discovered along the way, it doesn't yet seem like the time to jump in.

All the best!

Re: TimescaleDB vs. Amazon Timestream

#169
post #162

Earlier quoted context omitted.

No problem, my fault I misread your comment. `row_id` is a unique identifier for the row, for example my API will need it when the user wants to delete the specific row. Since it's possible for multiple rows to have the same `time`, even for the same `user_id`, I cannot assume uniqueness there. Partitioning on the row_id by making it an ordinal instead of a UUID could work, however I feel I would be missing out on Ti…

I mean, I'm not sure this is the place to get into detailed schema discussions, but I think I wouldn't use a row_id here at all, certainly, there's no need for this to be unique across all of the rows, you could make a unique constraint on user_id, time, row_id and then provide all 3 of them for the deletes, if the only reason is for the deletes, then that should be fine. If there are other reasons, then you probably…

You make some good points. Thanks for the insights/discussion!

Re: TimescaleDB vs. Amazon Timestream

#170

Earlier quoted context omitted.

Would that really explain the difference between 5 minutes on Timescale and 2 weeks on Timestream? Timeseries databases need the ability to ingest data rapidly.

Please see below for correction... but at some point today a poster said "~2 weeks" for Timestream... but that's incorrect. We ran the data load/ingest for "~2 days" (40 hours). Just want to make sure the right numbers are being used! Thanks! (Post author @Timescale)

STILL! 5 minutes vs 2 DAYS can't be attributed to just being replicated.
Post reply on HN