Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Timestream – Fast, scalable, fully managed time series database

aws.amazon.com

Post reply on HN
Print view View original

81–90 of 131 posts

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#81

Post by lozenge »

have_faith wrote: post #31
bruth wrote: post #17

Earlier quoted context omitted.

Really depends on the use case. Working in healthcare, vital signs can be modeled as time series points, but are lower frequency than, say, metrics from servers. However we want to store every point so a spike is not missed. One could argue an unsustained spike is noise, but in the healthcare domain there may be a correlation with some external event (the purpose is surprised and their heart rate spikes).

The clever thing to do in this scenario would be to keep every spike but delete all the data between similar data points after storing. So you get low granularity for identical/nearly-the-same data points and high granularity when something interesting happens. I don't have any experience with time-series data so maybe this is commonplace.

That would be impossible to run any new analyses on.

What some would do is record in blocks where every point after the earliest is stored as a delta. Then each block is more compressible as it contains a lot of 0s.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#82

Post by StreamBright »

dominotw wrote: post #72
stingraycharles wrote: post #71

Earlier quoted context omitted.

QuasarDB employee chiming in - we’re not actually worried, our clients are typically operating at a scale that would make costs very prohibitive for this AWS product. Having said that, I can definitely see this be an interesting product for people doing less than 10k inserts per second.

> make costs very prohibitive for this AWS product. Competing with AWS on just cost sounds worrying to me.

Actually not really. We have migrated several customers to AWS and our experience with AWS services are kind of all over the place. S3 is super cheap and reliable and there is almost nothing that beats that but for example CloudWatch is extremely expensive for a large scale operation that is easy to beat with custom software (like Prometheus or something similar). I guess even Datadog would beat them (and they have much more advanced features and integrations). This means that there are multiple software vendors can exist in the same space even when Amazon has an offering in that space.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#83

Post by lykr0n »

StreamBright wrote: post #79
lykr0n wrote: post #43

Earlier quoted context omitted.

Have you looked at Clickhouse for timeseries data? It's the one database I've found that can scale and can query in near-realtime. I've loaded a 100 Billion Rows in into a 5 shard database and can do full queries across the whole dataset in under 10 seconds. It also natively consumes multiple kafka topics.

Do you have a schema available publicly? I would like to build a similar system using custom software + S3 + Parquet + Athena for this task and see if it works.

The schema is

    CREATE TABLE points(Timestamp DateTime,Client String,Path String,Value Float32,Tags Nested(Key String,Value String)) ENGINE = MergeTree() ORDER BY (Client, Timestamp, Path) PARTITION BY toStartOfDay(Timestamp)
And this is a like query I was using

    SELECT (intDiv(toUInt32(Timestamp), 15) * 15) * 1000 as t, Path, Value as c FROM points_dist WHERE Path LIKE 'tst_val1' and Tags.Value[indexOf(Tags.Key, 'server')] = 'node' and Timestamp >= toDateTime(1543421708) GROUP BY t, Path, Value ORDER BY t, Path
This table was made on 5 servers via a distributed table partitioned on the timestamp- so distribution was even.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#84

Post by sciurus »

This is not cheap for the "DevOps" use case.

Imagine you have 1000 servers submitting data to 100 timeseries each minute. That's 100,000 writes a minute (unless they support batch writes across series) At $0.50 per million writes that's $72 a day or $26k a year.

Now imagine you want to alert on that data. Say you have 100 monitors that each evaluate 1GB of data once a minute. At $10 per TB of data scanned, that's $1,440 a day or $525k a year!

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#85

Post by abalone »

willlll wrote: post #26

I'm actually impressed at how incredibly expensive they made this. $0.50 per million 1KB writes, which is 20x what aurora charges, since aurora allows 8KB writes. And Aurora is already expensive if you actually read/write to it.

> $0.50 per million 1KB writes, which is 20x what aurora charges, since aurora allows 8KB writes.

That's a weird comparison. 20x is only true if you write 8KB with every entry, and you haven't included the storage and instance savings.

It's not hard to come up with suboptimal scenarios where this is more expensive, but that's missing the point. It's optimized for a specific kind of usage pattern.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#86

Post by Alex3917 »

sciurus wrote: post #84

This is not cheap for the "DevOps" use case. Imagine you have 1000 servers submitting data to 100 timeseries each minute. That's 100,000 writes a minute (unless they support batch writes across series) At $0.50 per million writes that's $72 a day or $26k a year. Now imagine you want to alert on that data. Say you have 100 monitors that each evaluate 1GB of data once a minute. At $10 per TB of data scanned, that's $1,…

Between RDS and these proprietary database products, AWS is now its own biggest competitor. That's potentially fine, but there is an inherent conflict of interest there and that needs to be properly managed, and while that may be happening it didn't come across in the keynote.

The only way I can have trust in Amazon's proprietary products is if RDS continues to get less expensive every year, since that is effectively the BATNA to these new products. It's been a while now since the last RDS cost reductions, and unless we continue to see more of those it's hard to have confidence that Amazon will continue to treat their customers of these new proprietary services fairly over the long term.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#87

Post by ZeroCool2u »

temuze wrote: post #9

Honest question: when dealing with time-series data, do you actually need every data point? Is that level of granularity really necessary? IMO, it makes way more sense to decide the aggregations you want ahead of time (e.g. "SELECT customer, sum(value) FROM purchases GROUP BY customer"). That way, you deal with substantially less data and everything becomes a whole lot simpler.

In finance it can be critical.

Some tasks actually require absolute granularity, up to 6 decimal places of precision and thereafter reliance on atomic order of arrival, for deterministic results on data from high frequency trading.

Without absolute knowledge of the order or if there's aggregation the best you can do is approximate, which often is considered suboptimal when the real solution is available.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#90

Post by [deleted] »

lozenge wrote: post #81
have_faith wrote: post #31

Earlier quoted context omitted.

The clever thing to do in this scenario would be to keep every spike but delete all the data between similar data points after storing. So you get low granularity for identical/nearly-the-same data points and high granularity when something interesting happens. I don't have any experience with time-series data so maybe this is commonplace.

That would be impossible to run any new analyses on. What some would do is record in blocks where every point after the earliest is stored as a delta. Then each block is more compressible as it contains a lot of 0s.

[deleted]
Post reply on HN