Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Timestream – Fast, scalable, fully managed time series database

aws.amazon.com

Post reply on HN
Print view View original

121–130 of 131 posts

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#121

Post by cthalupa »

SEJeff wrote: post #120
cthalupa wrote: post #116

Earlier quoted context omitted.

>The core of Amazon's timeseries db likely doesn't fit into a CPU's L1 cache. It does with KDB :) Does it? https://kx.com/discover/in-memory-computing/ seems to indicate that it takes up ~600 Kb (I'm not sure if this is bits or bytes, but even if it's bits, that turns into 75KB) L1 cache is per core. Skylake Xeons have a 64KB cache per core, 32KB for data and 32KB for instructions. Even with an even split there, you'…

Gah, I meant L2 cache, but was being entirely too smug. I remember a presentation a KX rep gave at our office a few jobs ago where this was one of their bullet points. I found it amusing, and a bit odd.

Gotcha! Definitely an interesting marketing point. I probably would have had the same reaction :)

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#122

Post by SEJeff »

Jedd wrote: post #115
SEJeff wrote: post #102

Earlier quoted context omitted.

Free software and open source aren't always the same thing though. Free Software is software that through the license enforces a philosophy. Open source is software that through the license enforces the source code to remain open. I'm not a fan of RMS or his attitudes on most things, but am a strong OSS fan as it is the best way to develop and maintain software.

> Free software and open source aren't always the same thing though. Entirely agree, hence I drew the distinction. I eschew 'open source' as it's highly ambiguous, and mostly misses the point. > Free Software is software that through the license enforces a philosophy. I would disagree. Free software ensures the user has certain freedoms. > Open source is software that through the license enforces the source code to r…

RMS has said publically that proprietary software should be illegal, and that free software guarantees user freedoms. This is a philosophy, and one that opines every user wants to become a developer. For things like emacs, and much of GNU, this is true, for most of the rest of the world, this is not.

I eschew free software because I'm not about forcing my views on others (which is literally the mission of GNU). I'm about developing software to be the best it can be, and maybe meeting some friends along the way. Open source, being the best software development model overall, allows me to meet that goal. You could almost say some of RMS's more extreme quirks border on authoritarian (see the example with the abortion joke in the libc manual he FORBADE removal of and demanded be re-added when a dev simply overruled him). He's not acting in a manner that encourages "freedom", but as a simple and obvious dictator of all things GNU or claiming to be GNU. He's frequently tried to shape the path of GNOME (which I am a former foundation member and was on the sysadmin team) in areas he literally has no business weighing in on. Then there are some more gross personality problems, like his sexism, or tendency to actually eat his toenail gunk[1], or to ever refuse to be wrong on anything, even when an entire community disagrees with him.

Dr Stallman has done a great deal of good for the world with Free Software, however like the VAX and PDP-11, his time has passed. Open source won just like Linux won over GNU/Hurd. It is ok that he won, by losing.

[1] https://www.youtube.com/watch?v=I25UeVXrEHQ

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#123

Post by StreamBright »

lykr0n wrote: post #119
StreamBright wrote: post #117

Earlier quoted context omitted.

Thanks! How big is the 100B rows in your system?

Takes up 200gb or so across 5 servers (this is according to ClickHouse's query stats). Actual disk might be a bit higher.

Thanks!

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#124

Post by ric2b »

SEJeff wrote: post #102
Jedd wrote: post #97

Earlier quoted context omitted.

> At what point are open source projects going to change their licensing to prevent the major cloud providers from just stealing their products? There's a lot to unravel in there. I prefer 'free software' to 'open source' as it has a clearer meaning, especially in this context. Even so, no one can steal free / open source software (or as you say, product -- though that turn strongly implies a commercial offering). By…

Free software and open source aren't always the same thing though. Free Software is software that through the license enforces a philosophy. Open source is software that through the license enforces the source code to remain open. I'm not a fan of RMS or his attitudes on most things, but am a strong OSS fan as it is the best way to develop and maintain software.

> Free Software is software that through the license enforces a philosophy.

This is incorrect, licenses like MIT or BSD are also free software license because they afford the 4 freedoms to the user (even though they don't enforce them on derivative works).

Licenses like the Redis one are open-source but not free software because they place limitations on the user (can't sell hosted service, IIRC)

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#125

Post by antirez »

ric2b wrote: post #124
SEJeff wrote: post #102

Earlier quoted context omitted.

Free software and open source aren't always the same thing though. Free Software is software that through the license enforces a philosophy. Open source is software that through the license enforces the source code to remain open. I'm not a fan of RMS or his attitudes on most things, but am a strong OSS fan as it is the best way to develop and maintain software.

> Free Software is software that through the license enforces a philosophy. This is incorrect, licenses like MIT or BSD are also free software license because they afford the 4 freedoms to the user (even though they don't enforce them on derivative works). Licenses like the Redis one are open-source but not free software because they place limitations on the user (can't sell hosted service, IIRC)

Redis is BSD licensed. You mean some Redis module developed independently by Redis Labs that are not part of Redis itself.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#126

Post by Jedd »

SEJeff wrote: post #122
Jedd wrote: post #115

Earlier quoted context omitted.

> Free software and open source aren't always the same thing though. Entirely agree, hence I drew the distinction. I eschew 'open source' as it's highly ambiguous, and mostly misses the point. > Free Software is software that through the license enforces a philosophy. I would disagree. Free software ensures the user has certain freedoms. > Open source is software that through the license enforces the source code to r…

RMS has said publically that proprietary software should be illegal, and that free software guarantees user freedoms. This is a philosophy, and one that opines every user wants to become a developer. For things like emacs, and much of GNU, this is true, for most of the rest of the world, this is not. I eschew free software because I'm not about forcing my views on others (which is literally the mission of GNU). I'm a…

rms's toenails aside, I'd suggest that every licence is about enforcing a set of views on others. There are plenty of licences to choose from, so it's fairly easy to find one compatible with your own views.

In the context of GP's (beginningguava) comment about 'open source projects' needing to change their licensing to prevent corporations making money by SaaSing various tools, my point was twofold - first, by definition you can't have free software with restrictions like that, and second you'd be merely fighting the symptoms (with little chance of success).

Aside - I'm curious what you mean by the 'open source software development model', as I don't think that's actually a thing.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#127

Post by SEJeff »

Jedd wrote: post #126
SEJeff wrote: post #122

Earlier quoted context omitted.

RMS has said publically that proprietary software should be illegal, and that free software guarantees user freedoms. This is a philosophy, and one that opines every user wants to become a developer. For things like emacs, and much of GNU, this is true, for most of the rest of the world, this is not. I eschew free software because I'm not about forcing my views on others (which is literally the mission of GNU). I'm a…

rms's toenails aside, I'd suggest that every licence is about enforcing a set of views on others. There are plenty of licences to choose from, so it's fairly easy to find one compatible with your own views. In the context of GP's (beginningguava) comment about 'open source projects' needing to change their licensing to prevent corporations making money by SaaSing various tools, my point was twofold - first, by defini…

Fair enough, the open source software development model is no different in reality from the free software development model generally speaking.

It goes back to ESR's The Cathedral and the Bazaar and is what he deems "the bazaar model" or "bazaar style" before him, Larry Augustin, and Bruce Perens (if memory serves) went on to coin the phrase "open source". Even if you don't necessarily agree with ESR (I see him in a similar vein as RMS fwiw), his thoughts on software development models have generally speaking, been proven true.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#128

Post by hnmullany »

lykr0n wrote: post #43
citilife wrote: post #20

At my day job, I build a lot of machine learning systems that require data to be fed in a time series manner[1]. Often this means building systems to analyze terabytes of logs [semi]-realtime. All I have to say is - thank god! This is going to make my job a lot easier, and likely empower us to remove our current infrastructure setup. I know at one point we actually considered building our own time series database. In…

Have you looked at Clickhouse for timeseries data? It's the one database I've found that can scale and can query in near-realtime. I've loaded a 100 Billion Rows in into a 5 shard database and can do full queries across the whole dataset in under 10 seconds. It also natively consumes multiple kafka topics.

Specialized tools are specialized: just remember the limitations! - Bad with heterogeneous hardware (Cloudflare experience) - Non-throttled recovery (source replicas flooded with replication load) - No real delete/update support, and no transactions - No secondary keys - Own protocol (no MySQL protocol support) - Limited SQL support, and the joins implementation is different. If you are migrating from MySQL or Spark, you will probably have to re-write all queries with joins. - No window functions

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#129

Post by valyala »

hnmullany wrote: post #128
lykr0n wrote: post #43

Earlier quoted context omitted.

Have you looked at Clickhouse for timeseries data? It's the one database I've found that can scale and can query in near-realtime. I've loaded a 100 Billion Rows in into a 5 shard database and can do full queries across the whole dataset in under 10 seconds. It also natively consumes multiple kafka topics.

Specialized tools are specialized: just remember the limitations! - Bad with heterogeneous hardware (Cloudflare experience) - Non-throttled recovery (source replicas flooded with replication load) - No real delete/update support, and no transactions - No secondary keys - Own protocol (no MySQL protocol support) - Limited SQL support, and the joins implementation is different. If you are migrating from MySQL or Spark,…

ClickHouse isn't general purpose DBMS. It is the best tool for collecting and near-online analyzing huge amount of events with many properties. We were successfully using ClickHouse cluster with 10 shards for collecting up to 3M events per second with 50 properties each (properties translate to columns). Each shard was running on n1-highmem-16 instance in Google Cloud. The cluster was able to scan tens of billions of rows per second for our queries. The scan performance was 100x better than on the previous highly tuned system built on PostgreSQL.

ClickHouse may be used as a timeseries backend, but currently it has a few drawbacks comparing to specialized solutions: - It has no efficient inverted index for fast metrics lookup by a set of label matchers. - It doesn't support delta coding yet - https://github.com/yandex/ClickHouse/issues/838 .

Learn how we created a startup - VictoriaMetrics - that builds on performance ideas from ClickHouse and solves the issues mentioned above - https://medium.com/devopslinks/victoriametrics-creating-the-... . Currently it has the highest performance/cost ratio comparing to competitors.

Re: Amazon Timestream – Fast, scalable, fully managed time series database

#130

Post by haggy »

askbjoernhansen wrote: post #118
haggy wrote: post #49

Earlier quoted context omitted.

> I've loaded a 100 Billion Rows Have you done any load tests that would more closely mirror a production environment such as performing queries while clickhouse is handling a heavy insert load?

We load millions of rows per second and use a handful (or more?) of materialized views to build appropriate summaries. Various clients make queries to the "raw data" and the views and it all works fine, basically.

> it all works fine, basically

The "basically" is what intrigues me :D

Post reply on HN