This is looking like a managed druid... that would be very nice to have.
Amazon Timestream – Fast, scalable, fully managed time series database
111–120 of 131 posts
Re: Amazon Timestream – Fast, scalable, fully managed time series database
#112It's got to be a rough day for the team at https://www.influxdata.com/ . This could become serious competition for their InfluxCloud hosted offering.
At what point are open source projects going to change their licensing to prevent the major cloud providers from just stealing their products? I highly doubt AWS built this from scratch. Amazon, Google, and Microsoft are going to choke the life out of these projects Redis and MongoDB at least seem to have woken up https://www.geekwire.com/2018/open-source-companies-consider...
> I highly doubt AWS built this from scratch
Unfounded doubt.
Re: Amazon Timestream – Fast, scalable, fully managed time series database
#113This is not cheap for the "DevOps" use case. Imagine you have 1000 servers submitting data to 100 timeseries each minute. That's 100,000 writes a minute (unless they support batch writes across series) At $0.50 per million writes that's $72 a day or $26k a year. Now imagine you want to alert on that data. Say you have 100 monitors that each evaluate 1GB of data once a minute. At $10 per TB of data scanned, that's $1,…
That's convoluted... from you example, there are 100K writes/minute, while you assume 1GB data evaluated per minute per alarm. That is you're assuming 10K/item/timeseries for each alarm , while reality is going to be closer to 10-100 bytes/item/time-series, which cuts down the expense by two or three orders of magnitude.
That's a really good point, I was guesstimating very quickly when I wrote that. Depending on what your doing for the alarm and how they measure reads it may well be a lot less.
Let's say the read is 1MB instead of 1GB, that's now $1.44 a day and $525 a year. Query pricing becomes not so bad.
From my own experience, the 100 metrics per server estimate I was giving above is pretty low, though. Once you factor in different combinations of tags closer to 1000 is more realistic. That potentially brings up the write pricing quite a bit.
Re: Amazon Timestream – Fast, scalable, fully managed time series database
#114Earlier quoted context omitted.
I wouldn't jump the gun on this. I've been working within Amazon cloud for years and every year they make massive claims about new services at re:invent. Not saying this isn't going to be a good product, just saying it will probably take a while to be as useful as you're hoping.
I agree w/ this. They make insane promises but, the promises don't like up to expectations. Kinesis analytics for example, can aggregate data across a time window (sliding window) from a stream (Kinesis). A huge issue that isn't document or stated is that when kinesis analytics restarts due to the process dies (being migrated, binpacked, etc.) the ENTIRE time window has to get re-aggregated. So your count drops to 0.…
Similar examples have been Fargate and Lambda, which have surprisingly long cold start times depending on runtimes and VPC configurations. IMHO it is a big part of AWS expertise to know these things and be able to choose the right services not just according to the marketing brochures, but according to how they really work for each use case. Having said that, I'm glad to now have learned about that Kinesis Analytics restart issue.
Re: Amazon Timestream – Fast, scalable, fully managed time series database
#115Earlier quoted context omitted.
> At what point are open source projects going to change their licensing to prevent the major cloud providers from just stealing their products? There's a lot to unravel in there. I prefer 'free software' to 'open source' as it has a clearer meaning, especially in this context. Even so, no one can steal free / open source software (or as you say, product -- though that turn strongly implies a commercial offering). By…
Free software and open source aren't always the same thing though. Free Software is software that through the license enforces a philosophy. Open source is software that through the license enforces the source code to remain open. I'm not a fan of RMS or his attitudes on most things, but am a strong OSS fan as it is the best way to develop and maintain software.
> Free software and open source aren't always the same thing though.
Entirely agree, hence I drew the distinction. I eschew 'open source' as it's highly ambiguous, and mostly misses the point.
> Free Software is software that through the license enforces a philosophy.
I would disagree. Free software ensures the user has certain freedoms.
> Open source is software that through the license enforces the source code to remain open.
This is a very circular definition -- open source is open.
> I'm not a fan of RMS or his attitudes on most things, but am a strong OSS fan as it is the best way to develop and maintain software.
As it happens, rms is no fan of OSS.
Re: Amazon Timestream – Fast, scalable, fully managed time series database
#116I wonder how this compares to KDB
The core of Amazon's timeseries db likely doesn't fit into a CPU's L1 cache. It does with KDB :)
>The core of Amazon's timeseries db likely doesn't fit into a CPU's L1 cache. It does with KDB :)
Does it?
https://kx.com/discover/in-memory-computing/ seems to indicate that it takes up ~600 Kb (I'm not sure if this is bits or bytes, but even if it's bits, that turns into 75KB)
L1 cache is per core. Skylake Xeons have a 64KB cache per core, 32KB for data and 32KB for instructions. Even with an even split there, you're not fitting 75KB (or 600KB) into the L1 cache.
Bits would be a weird measurement to use when talking about memory utilization, so I'm pretty sure that it's 600 kilobytes. You're not anywhere close to fitting that into the L1 cache. L2 cache, sure. But you get the relatively spacious 1 megabyte for L2.
I'm also not sure that the "core" fitting into the CPU cache is particularly meaningful for performance anyway. it doesn't say anything about how much outside of the core gets used, how big the working set size is for your workload, how much meaningful work is done on that working set of data, etc. If you're frequently using parts of the software that don't fit in the cache, or getting evicted from it for other code, or your working set of data doesn't fit in the cache and you're constantly going to main memory for the data you're working on, the "core" fitting in L1 cache (or L2 cache, which looks more realistic) is going to be basically meaningless.
Re: Amazon Timestream – Fast, scalable, fully managed time series database
#117Earlier quoted context omitted.
Do you have a schema available publicly? I would like to build a similar system using custom software + S3 + Parquet + Athena for this task and see if it works.
The schema is CREATE TABLE points(Timestamp DateTime,Client String,Path String,Value Float32,Tags Nested(Key String,Value String)) ENGINE = MergeTree() ORDER BY (Client, Timestamp, Path) PARTITION BY toStartOfDay(Timestamp) And this is a like query I was using SELECT (intDiv(toUInt32(Timestamp), 15) * 15) * 1000 as t, Path, Value as c FROM points_dist WHERE Path LIKE 'tst_val1' and Tags.Value[indexOf(Tags.Key, 'serve…
Thanks! How big is the 100B rows in your system?
Re: Amazon Timestream – Fast, scalable, fully managed time series database
#118Earlier quoted context omitted.
Have you looked at Clickhouse for timeseries data? It's the one database I've found that can scale and can query in near-realtime. I've loaded a 100 Billion Rows in into a 5 shard database and can do full queries across the whole dataset in under 10 seconds. It also natively consumes multiple kafka topics.
> I've loaded a 100 Billion Rows Have you done any load tests that would more closely mirror a production environment such as performing queries while clickhouse is handling a heavy insert load?
We load millions of rows per second and use a handful (or more?) of materialized views to build appropriate summaries. Various clients make queries to the "raw data" and the views and it all works fine, basically.
Re: Amazon Timestream – Fast, scalable, fully managed time series database
#119Earlier quoted context omitted.
The schema is CREATE TABLE points(Timestamp DateTime,Client String,Path String,Value Float32,Tags Nested(Key String,Value String)) ENGINE = MergeTree() ORDER BY (Client, Timestamp, Path) PARTITION BY toStartOfDay(Timestamp) And this is a like query I was using SELECT (intDiv(toUInt32(Timestamp), 15) * 15) * 1000 as t, Path, Value as c FROM points_dist WHERE Path LIKE 'tst_val1' and Tags.Value[indexOf(Tags.Key, 'serve…
Thanks! How big is the 100B rows in your system?
Takes up 200gb or so across 5 servers (this is according to ClickHouse's query stats). Actual disk might be a bit higher.
Re: Amazon Timestream – Fast, scalable, fully managed time series database
#120Earlier quoted context omitted.
The core of Amazon's timeseries db likely doesn't fit into a CPU's L1 cache. It does with KDB :)
>The core of Amazon's timeseries db likely doesn't fit into a CPU's L1 cache. It does with KDB :) Does it? https://kx.com/discover/in-memory-computing/ seems to indicate that it takes up ~600 Kb (I'm not sure if this is bits or bytes, but even if it's bits, that turns into 75KB) L1 cache is per core. Skylake Xeons have a 64KB cache per core, 32KB for data and 32KB for instructions. Even with an even split there, you'…
Gah, I meant L2 cache, but was being entirely too smug. I remember a presentation a KX rep gave at our office a few jobs ago where this was one of their bullet points. I found it amusing, and a bit odd.