Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Not to Build a Time-Series Database

outlyer.com

Post reply on HN
Print view View original

111–120 of 128 posts

Re: Why Not to Build a Time-Series Database

#111

Post by ghc »

geocar wrote: post #98
ghc wrote: post #52

Earlier quoted context omitted.

> Distributed relational column-oriented databases are the best at large OLAP data volumes and queries. KDB+ is one of those, even though they call it a TSDB in marketing terminology because of its FinTech customer base. You're mistaken about Kdb's relational features. Kdb was designed as a time series processing engine using arrays (columns). Column storage doesn't have anything to do with whether a database is rela…

> Kdb wasn't originally any more relational than the language Erlang is. To wit: SQL wasn't originally any more relational[1] than the language Erlang is. [1]: Codd, E. F. A Relational Model of Data for Large Shared Data Banks -- Communications of the ACM, Vol. 13, No. 6, June 1970, pp. 377-387 I'm not sure what value such a statement brings the world.

I was drawing a comparison between the Kdb runtime and the Erlang runtime, because the OP seems to be conflating the ability to emulate relational features at the application level with first-class support for relational semantics as in a relational database. Support for relational semantics can be emulated via programming languages with sophisticated runtimes like Erlang, but I wouldn't classify any database supporting a Turing complete language with a sophisticated runtime as a relational database.

Re: Why Not to Build a Time-Series Database

#112

Post by wmfiv »

cmroanirgo wrote: post #91
tuukkah wrote: post #60

TLDR: "Why Not to Build a Time-Series Database? Because we're building one and you should pay us." > Hopefully our story will make you think twice before trying to build your own TSDB in house using open-source solutions, or if you’re really crazy, building a TSDB from scratch. Building and maintaining a TSDB is a full time job, and we have dedicated expert engineers who are constantly improving and maintaing our TSD…

Hmmm. I used to be part of a team that handled market data at crazy rates and we took exactly the opposite approach to these guys. When I see: "You Can Lose a Few Datapoints Here and There" I see that these guys are barking the wrong tree. 1. We used single thread per network card. (Yes, we architected clusters/failovers, etc... but not once was it required because of data rates) 2. The server could handle a fully sa…

Right. For those interested OpenHFT has created a really nice set of open source solutions to do this.

https://github.com/OpenHFT/Chronicle-Queue#design

Re: Why Not to Build a Time-Series Database

#113

Post by dominotw »

manigandham wrote: post #66
dominotw wrote: post #65

Earlier quoted context omitted.

> Time-series is just data with time as a primary component. Doesn't it have a special feature that data comes in a linearly increasing time dimension.

I don't see why that's so special. Other data can have incrementing IDs or some other value. In fact all data can be considered to have a timestamp, at the very minimum being when it was inserted into the database, so it's a rather vague definition overall.

I think, its different because its dimension where usefulness of data goes does as the data ages. TimescaleDB for example, does optimisations ( they call it chunks or something) based on this fact.

Re: Why Not to Build a Time-Series Database

#114

Post by anon12345690 »

dub wrote: post #22
manigandham wrote: post #11

"time-series database" is some of the most overhyped nonsense since noSQL. Time-series is just data with time as a primary component. It comes in all shapes and volume, but if you have a lot of data and are running heavy OLAP queries than we already have an entire class of capable databases. Use any modern distributed relational column-oriented database, set primary key to metric id + timestamp, and you'll be able to…

A modern, distributed, relational, column-oriented database will often stress emphatically in the documentation that using timestamps as primary keys is an anti-pattern that's likely to lead to hot tablets: https://cloud.google.com/spanner/docs/schema-design#choosing...

the first section on that page shows what the op was saying, to use the name column first and timestamp last

Re: Why Not to Build a Time-Series Database

#115

Post by manigandham »

ghc wrote: post #107
manigandham wrote: post #81

Earlier quoted context omitted.

I never said relational is related to column-storage. kdb+ has SQL semantics and relational queries, but it's a combination of the q language integrated into a database so sure, it's a superset of a relational database. Perhaps we disagree on what relational means.

My point was that the current relational features of Kdb didn't exist originally (they were grafted on later) so it's not "marketed" as a TSDB, but it is in fact a TSDB marketed as a relational DB. The definition of relational is very precise, whether you use the domain calculus, relational calculus or relational algebra. Wiki has a good summary of what must be natively supported by a database system to be relational…

Why does it matter what it was originally? We're talking about what the product is today, not 20 years ago.

kdb+ supports a superset of SQL and relational algebra, so it's a relational database. How it's implemented doesn't matter if it provide the functionality, which it can.

Re: Why Not to Build a Time-Series Database

#116

Post by manigandham »

dominotw wrote: post #113
manigandham wrote: post #66

Earlier quoted context omitted.

I don't see why that's so special. Other data can have incrementing IDs or some other value. In fact all data can be considered to have a timestamp, at the very minimum being when it was inserted into the database, so it's a rather vague definition overall.

I think, its different because its dimension where usefulness of data goes does as the data ages. TimescaleDB for example, does optimisations ( they call it chunks or something) based on this fact.

Timescale is just an automatic partitioning extension for Postgres. You can also do it with the native partitioning feature, or pg_partman, or Citus, or other tools.

Partitioning the table is the optimization, so that you skip over data when querying and manage it in smaller parts, but Timescale doesn't do anything about older data and neither do most databases.

Re: Why Not to Build a Time-Series Database

#117

Post by pvitz »

wmfiv wrote: post #112
cmroanirgo wrote: post #91

Earlier quoted context omitted.

Hmmm. I used to be part of a team that handled market data at crazy rates and we took exactly the opposite approach to these guys. When I see: "You Can Lose a Few Datapoints Here and There" I see that these guys are barking the wrong tree. 1. We used single thread per network card. (Yes, we architected clusters/failovers, etc... but not once was it required because of data rates) 2. The server could handle a fully sa…

Right. For those interested OpenHFT has created a really nice set of open source solutions to do this. https://github.com/OpenHFT/Chronicle-Queue#design

Do you know any article or book outlining the architecture of a full HFT system, I.e. from market data consumption to pricing to trading? Thanks in advance!

Re: Why Not to Build a Time-Series Database

#118

Post by dominotw »

viraptor wrote: post #88
Daneel_ wrote: post #85

I’m surprised no one has brought up Splunk in here (that I could see at a cursory glance). They manage to do time-series storage on a pretty large scale (over 5PB/day for their largest customer).

I wouldn't really say that splunk is time-series in the same way they wanted. It's mainly logs indexing, but not the kind of aggregation you'd want from numeric, labelled metrics.

> aggregation you'd want from numeric, labelled metrics.

We use it exactly for this, works great.

Re: Why Not to Build a Time-Series Database

#119

Post by anarazel »

manigandham wrote: post #116
dominotw wrote: post #113

Earlier quoted context omitted.

I think, its different because its dimension where usefulness of data goes does as the data ages. TimescaleDB for example, does optimisations ( they call it chunks or something) based on this fact.

Timescale is just an automatic partitioning extension for Postgres. You can also do it with the native partitioning feature, or pg_partman, or Citus, or other tools. Partitioning the table is the optimization, so that you skip over data when querying and manage it in smaller parts, but Timescale doesn't do anything about older data and neither do most databases.

> Timescale is just an automatic partitioning extension for Postgres.

That, and a good PR operation.

Re: Why Not to Build a Time-Series Database

#120

Post by viraptor »

dominotw wrote: post #118
viraptor wrote: post #88

Earlier quoted context omitted.

I wouldn't really say that splunk is time-series in the same way they wanted. It's mainly logs indexing, but not the kind of aggregation you'd want from numeric, labelled metrics.

> aggregation you'd want from numeric, labelled metrics. We use it exactly for this, works great.

Do you use the new metrics store? I just learned they have one now in the new versions.
Post reply on HN