Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing ArcticDB: A Database for Observability

polarsignals.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Introducing ArcticDB: A Database for Observability

#21

Post by caust1c »

brancz wrote: post #18
caust1c wrote: post #16

Earlier quoted context omitted.

How does it compare to BadgerDB/RocksDB/LevelDB? I see that it's using Arrow and Parquet of course, but the Sparse Index sounds very similar to LSM Tree like storage engines, except using something like a K-Way Merge algorithm and a heap structure to manage that somehow? I'm more of an operator and user of these systems, so as an operator I care more about the usability than what's underneath, but also am reasonably…

It's closer to DuckDB rather than Badger/RocksDB/LevelDB, but similar in the sense that it is an embeddable database, not one that is operated standalone. It's not unlike an LSM tree, but the difference is that the leafs in the tree are not individual keys, but rather describe a range of values that are all read at once if read. So this allows high write throughput _and_ high read throughput, trading off mutability (…

It was my understanding that those databases do have sparse indexes, but admittedly I may have applied that assumption based on my majority experience with Clickhouse where it uses LSM Tree engines and also has a sparse index.

Would it be correct to say this is like an embeddable clickhouse engine, minus the SQL interface and using Arrow and Parquet as the storage format?

Re: Introducing ArcticDB: A Database for Observability

#22

Post by brancz »

caust1c wrote: post #21
brancz wrote: post #18

Earlier quoted context omitted.

It's closer to DuckDB rather than Badger/RocksDB/LevelDB, but similar in the sense that it is an embeddable database, not one that is operated standalone. It's not unlike an LSM tree, but the difference is that the leafs in the tree are not individual keys, but rather describe a range of values that are all read at once if read. So this allows high write throughput _and_ high read throughput, trading off mutability (…

It was my understanding that those databases do have sparse indexes, but admittedly I may have applied that assumption based on my majority experience with Clickhouse where it uses LSM Tree engines and also has a sparse index. Would it be correct to say this is like an embeddable clickhouse engine, minus the SQL interface and using Arrow and Parquet as the storage format?

Yes, that's correct! Plus the dynamic column feature, which we think is crucial for Observability-type workloads (from what we know only InfluxDB IOx supports a similar feature).

Re: Introducing ArcticDB: A Database for Observability

#23

Post by caust1c »

brancz wrote: post #22
caust1c wrote: post #21

Earlier quoted context omitted.

It was my understanding that those databases do have sparse indexes, but admittedly I may have applied that assumption based on my majority experience with Clickhouse where it uses LSM Tree engines and also has a sparse index. Would it be correct to say this is like an embeddable clickhouse engine, minus the SQL interface and using Arrow and Parquet as the storage format?

Yes, that's correct! Plus the dynamic column feature, which we think is crucial for Observability-type workloads (from what we know only InfluxDB IOx supports a similar feature).

Nice! Thanks for the reply! Looking forward to see where this goes! :-)

Re: Introducing ArcticDB: A Database for Observability

#24

Post by bigcat12345678 »

brancz wrote: post #2

Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb

Does ArcticDB support store data in memory, and flush overflown & old data onto disk (or other persistent storage sink)?

Re: Introducing ArcticDB: A Database for Observability

#25

Post by brancz »

bigcat12345678 wrote: post #24
brancz wrote: post #2

Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb

Does ArcticDB support store data in memory, and flush overflown & old data onto disk (or other persistent storage sink)?

Not yet, but we will soon, though we already rotate the in memory state when a certain (configurable) size is reached. The idea is that we’ll put data in parquet format into object storage from where it can be consumed from any parquet compatible tools. That plus additional metadata is what we’ll use for long term storage for profiling data ourselves.

Re: Introducing ArcticDB: A Database for Observability

#26

Post by bigcat12345678 »

brancz wrote: post #25
bigcat12345678 wrote: post #24

Earlier quoted context omitted.

Does ArcticDB support store data in memory, and flush overflown & old data onto disk (or other persistent storage sink)?

Not yet, but we will soon, though we already rotate the in memory state when a certain (configurable) size is reached. The idea is that we’ll put data in parquet format into object storage from where it can be consumed from any parquet compatible tools. That plus additional metadata is what we’ll use for long term storage for profiling data ourselves.

Just to clarify more on the in-memory part, it offers the same Arrow APIs for querying, right? Or only specialized query API tailored to Parca's use?

We'd like it to be Arrow APIs, such that we can use it for other purposes, but still in the Observability space actually.

Read the other comment: https://news.ycombinator.com/item?id=31263825 Already explained that the in-memory part, but missing the serialization part.

Re: Introducing ArcticDB: A Database for Observability

#27

Post by brancz »

bigcat12345678 wrote: post #26
brancz wrote: post #25

Earlier quoted context omitted.

Not yet, but we will soon, though we already rotate the in memory state when a certain (configurable) size is reached. The idea is that we’ll put data in parquet format into object storage from where it can be consumed from any parquet compatible tools. That plus additional metadata is what we’ll use for long term storage for profiling data ourselves.

Just to clarify more on the in-memory part, it offers the same Arrow APIs for querying, right? Or only specialized query API tailored to Parca's use? We'd like it to be Arrow APIs, such that we can use it for other purposes, but still in the Observability space actually. Read the other comment: https://news.ycombinator.com/item?id=31263825 Already explained that the in-memory part, but missing the serialization part.

ArcticDB has a general purpose dataframe-like query builder that Parca uses to build its queries. The query engine scans parquet row groups and those that may contain interesting data are converted to arrow and go through the query plans the query planner creates. The query plans other than the table scan expect arrow frames.

Hope that explains it, but happy to elaborate more!

Re: Introducing ArcticDB: A Database for Observability

#28

Post by bigcat12345678 »

brancz wrote: post #27
bigcat12345678 wrote: post #26

Earlier quoted context omitted.

Just to clarify more on the in-memory part, it offers the same Arrow APIs for querying, right? Or only specialized query API tailored to Parca's use? We'd like it to be Arrow APIs, such that we can use it for other purposes, but still in the Observability space actually. Read the other comment: https://news.ycombinator.com/item?id=31263825 Already explained that the in-memory part, but missing the serialization part.

ArcticDB has a general purpose dataframe-like query builder that Parca uses to build its queries. The query engine scans parquet row groups and those that may contain interesting data are converted to arrow and go through the query plans the query planner creates. The query plans other than the table scan expect arrow frames. Hope that explains it, but happy to elaborate more!

IIUC: The query written by users of arcticdb is in a dataframe-like python-ish language.

The query engine first go through the inmemory parquet rows to get a subset that contains the relevant data. These relevant data is returned as arrow frames.

Then the query planer produces a query plan and the engine execute the query plan on the previously returned arrow frames.

Does the query engine produce the query plan before selecting the arrow frames, and uses the plan for the selecting process as well?

Re: Introducing ArcticDB: A Database for Observability

#30

Post by brancz »

bigcat12345678 wrote: post #28
brancz wrote: post #27

Earlier quoted context omitted.

ArcticDB has a general purpose dataframe-like query builder that Parca uses to build its queries. The query engine scans parquet row groups and those that may contain interesting data are converted to arrow and go through the query plans the query planner creates. The query plans other than the table scan expect arrow frames. Hope that explains it, but happy to elaborate more!

IIUC: The query written by users of arcticdb is in a dataframe-like python-ish language. The query engine first go through the inmemory parquet rows to get a subset that contains the relevant data. These relevant data is returned as arrow frames. Then the query planer produces a query plan and the engine execute the query plan on the previously returned arrow frames. Does the query engine produce the query plan befor…

As it stands, right now the query planner doesn’t look at any of the data (yet) when putting together the plan, but this will inevitably happen.
Post reply on HN