Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Timestream – Fast, scalable, fully managed time series database

aws.amazon.com

Post reply on HN
Print view View original

91–100 of 131 posts

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#91

Post by jaxxstorm »

sciurus wrote: post #84

This is not cheap for the "DevOps" use case. Imagine you have 1000 servers submitting data to 100 timeseries each minute. That's 100,000 writes a minute (unless they support batch writes across series) At $0.50 per million writes that's $72 a day or $26k a year. Now imagine you want to alert on that data. Say you have 100 monitors that each evaluate 1GB of data once a minute. At $10 per TB of data scanned, that's $1,…

Well, that depends on what you consider cheap. Hiring someone to manage a time series system like graphite or prometheus is going to cost you a whole lot more than $26k a year

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#92

Post by brootstrap »

Been searching for years to a good alternative to postgres for storing gobs of weather timeseries data. So far we have been running postgres system for many years in production and have hired multiple contractors to implement a 'real timeseries solution'. All of which have been utter shit and complete failures. The AWS services are expensive as all hell. With a little bit of imagination we created a unique schema for timeseries data that doesnt require terabytes of space, and processes billions of data points a day, and has blazing fast queries into said data.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#93

Post by DelaneyM »

brootstrap wrote: post #92

Been searching for years to a good alternative to postgres for storing gobs of weather timeseries data. So far we have been running postgres system for many years in production and have hired multiple contractors to implement a 'real timeseries solution'. All of which have been utter shit and complete failures. The AWS services are expensive as all hell. With a little bit of imagination we created a unique schema for…

I once handled gobs of web tracking data in Cassandra, using Hadoop over Cassandra to build queryable rollups in MySQL and Pig for on the fly analysis

Neither tech works on its own, but together (substituting Cassandra columns for hdfs) was magic for the specific data configuration & use case.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#94

Post by jws »

brootstrap wrote: post #92

Been searching for years to a good alternative to postgres for storing gobs of weather timeseries data. So far we have been running postgres system for many years in production and have hired multiple contractors to implement a 'real timeseries solution'. All of which have been utter shit and complete failures. The AWS services are expensive as all hell. With a little bit of imagination we created a unique schema for…

I moved a decade's worth of weather time series data from well indexed Sqlite to InfluxDB and was nothing but pleased. It ended up taking an order of magnitude less storage and so much faster to query that I didn't even bother to benchmark it. You can probably write a simple query to your Postgres database to cough out the text file to load InfluxDB to see how it works for you. Then it comes down to how easy it is to replace your query and insert functions… So it's all easy except for the hard part.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#95

Post by porker »

brootstrap wrote: post #92

Been searching for years to a good alternative to postgres for storing gobs of weather timeseries data. So far we have been running postgres system for many years in production and have hired multiple contractors to implement a 'real timeseries solution'. All of which have been utter shit and complete failures. The AWS services are expensive as all hell. With a little bit of imagination we created a unique schema for…

Did you try https://www.timescale.com/?

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#97

Post by Jedd »

beginningguava wrote: post #63
sciurus wrote: post #30

It's got to be a rough day for the team at https://www.influxdata.com/ . This could become serious competition for their InfluxCloud hosted offering.

At what point are open source projects going to change their licensing to prevent the major cloud providers from just stealing their products? I highly doubt AWS built this from scratch. Amazon, Google, and Microsoft are going to choke the life out of these projects Redis and MongoDB at least seem to have woken up https://www.geekwire.com/2018/open-source-companies-consider...

> At what point are open source projects going to change their licensing to prevent the major cloud providers from just stealing their products?

There's a lot to unravel in there.

I prefer 'free software' to 'open source' as it has a clearer meaning, especially in this context. Even so, no one can steal free / open source software (or as you say, product -- though that turn strongly implies a commercial offering).

By definition you can't really stop anyone from using your free software, unless perhaps you start naming companies explicitly, but I can't imagine it'd be an easy process, or have a happy outcome, if you started targeting 'major cloud providers' for special conditions.

Note that I am not an apologist for AWS, Google, Microsoft, etc - but it feels like the fundamental problem here is not massive corporations charging other people to access free software.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#98

Post by Tehnix »

Quite excited for this! We have currently been experimenting with using DynamoDB, and managing our own rollups of our incoming data (previously on an RDS, which is not a good choice for this kind of data).

---

I've seen a lot of people complain about pricing, so I thought I'd share a little why we are excited about this:

We have approximately 280 devices out, monitoring production lines, sending aggregated data every 5 seconds, via MQTT to AWS IoT. The average messages published that we see is around ~2 million a day (equipment is often turned off, when not producing). The packet size is very small, and highly compressable, each below 1KB, but let's just make it 1KB.

We then currently funnel this data into Lambda, which processes it, and puts it into DynamoDB and handles rollups. The costs of that whole thing is approximately $20 a day (IoT, DynamoDB, Lambda and X-Ray), with Lambda+DynamoDB making up $17 of that cost.

Finally, our users look at this data, live, on dashboards, usually looking at the last 8 hours of data for a specific device. Let's throw around that there will be 10,000 queries each day, looking at the data of the day (2GB/day / 280devices = 0.007142857 GB/device/day).

---

Now, running the same numbers on the AWS Timestream pricing[0] (daily cost):

- Writes: 2million * $0.5/million = $1

- Memory store: 2 GB * $0.036 = $0.072

- SSD store: (2GB * 7days) * $0.01 (GB/day) * 7days = $0.98

- Magnetic store: (2 GB * 30 days) * $0.03 (GB/month) = $1.8

- Query: 10,0000 queries * 0.007142857GB/device/day --> 71GB = free until day 14, where it'll cost $10, so $20 a month.

Giving us: $1 + $0.072 + $0.98 + $1.8 + ($20/30) = $4.5/day.

From these (very) quick calculations, this means we could lower our cost from ~$20/day to ~$4.5/day. And that's not even taking into account that it removes our need to create/maintain our own custom solution.

I am probably missing some details, but it does look bright!

[0] https://aws.amazon.com/timestream/pricing/

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#99

Post by [deleted] »

jaxxstorm wrote: post #91
sciurus wrote: post #84

This is not cheap for the "DevOps" use case. Imagine you have 1000 servers submitting data to 100 timeseries each minute. That's 100,000 writes a minute (unless they support batch writes across series) At $0.50 per million writes that's $72 a day or $26k a year. Now imagine you want to alert on that data. Say you have 100 monitors that each evaluate 1GB of data once a minute. At $10 per TB of data scanned, that's $1,…

Well, that depends on what you consider cheap. Hiring someone to manage a time series system like graphite or prometheus is going to cost you a whole lot more than $26k a year

[deleted]

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#100

Post by SEJeff »

lykr0n wrote: post #54
haggy wrote: post #49

Earlier quoted context omitted.

> I've loaded a 100 Billion Rows Have you done any load tests that would more closely mirror a production environment such as performing queries while clickhouse is handling a heavy insert load?

I'm working on developing benchmarking tools for internal testing, but both Yandex and CloudFlare use Clickhouse for realtime querying. I'm still in development phase for my product, but I'll make sure to post information & results when we launch here. https://blog.cloudflare.com/http-analytics-for-6m-requests-p... But I've spent a long time looking at the various solutions out there, and while ClickHouse is not perf…

I'd say kdb (from kx systems) is the best database for this problemspace, but it is prohibitively expensive for the http analytics use case. It is also a pain to query, but unbelievable what it can do.
Post reply on HN