Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing ArcticDB: A Database for Observability

polarsignals.com

Post reply on HN
Print view View original

11–20 of 32 posts

Re: Introducing ArcticDB: A Database for Observability

#11

Post by brancz »

eatonphil wrote: post #9
bfm wrote: post #6

Earlier quoted context omitted.

What was the motivation for building your own instead of using something like DuckDB which support parquet out of the box? What are the differences?

In addition to other reasons they may have, an embedded database for Go being built in Go means they don't need to require CGO, which DuckDB in Go would require.

This was definitely part of the motivation, but even more importantly (and it's possible that we missed it in the duckdb documentation when we explored doing exactly this) we needed the ability to add columns dynamically when we see a new label-name. This is sort of an analogy of wide-columns in Cassandra, but forcing it into the columnar layout to allow it to be searched and aggregated by efficiently. From our research all the open source column databases at best support a map type, through which we loose the columnar layout since the values of the map are all stored together giving us row-based database characteristics. (all databases except InfluxDB IOx, whose developers we talked to extensively and who highly inspired this design)

Re: Introducing ArcticDB: A Database for Observability

#12

Post by brancz »

jtwebman wrote: post #8
brancz wrote: post #2

Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb

You really should change the front page to why I should use it vs why we built it. I don't care why you built it that it is in Go. I do care why I should use it which must be hidden somewhere in why you built it but I don't have that kind of time.

Thank you so much for the feedback! By front page I assume you mean the project readme? I agree, I think we leaned a bit too far into design documentation in the readme rather than explaining why and how to use it. We'll get that fixed!

Re: Introducing ArcticDB: A Database for Observability

#13

Post by brancz »

witcher wrote: post #3

Finally seeing Apache Parquet and Apache Arrow used with Go efficiently and effectively! Great job. Looking forward to exploring this more in the Prometheus and CNCF Ecosystem. The underneath library used ( https://github.com/segmentio/parquet-go ) looks amazing too!

I agree, both Parquet and Arrow are super powerful technologies. This Parquet library is an absolute bliss to work with (and contribute to), the APIs are so well designed!

Re: Introducing ArcticDB: A Database for Observability

#14

Post by bfm »

brancz wrote: post #11
eatonphil wrote: post #9

Earlier quoted context omitted.

In addition to other reasons they may have, an embedded database for Go being built in Go means they don't need to require CGO, which DuckDB in Go would require.

This was definitely part of the motivation, but even more importantly (and it's possible that we missed it in the duckdb documentation when we explored doing exactly this) we needed the ability to add columns dynamically when we see a new label-name. This is sort of an analogy of wide-columns in Cassandra, but forcing it into the columnar layout to allow it to be searched and aggregated by efficiently. From our resea…

It makes sense, DuckDB's documentation has significantly improved, but it is still lacking when it comes to the limitations of using parquet. We have also hit some roadblocks when updating schemas backed by parquet files, so we now only use DuckDB for querying parquet via SQL.

Re: Introducing ArcticDB: A Database for Observability

#15

Post by hintymad »

It's great to see more solutions in this space, and congratulations on launching and open sourcing it. Is it possible to explain why it matters to an end users? The Why We Built It section seems solely focus on implementation and what matters to the team.

> First, we needed something embeddable for Go

I'm not sure why it matters if I just need to store and analyze large quantity of profiling data, but let me assume this matters to the creator of the db.

> The second and more pressing argument was: in order for us to be able to translate the label-based data-model to a table-based layout, we needed the ability to create columns whenever we see a label-name for the first time.

Why does this matter to me, an end user? Does it make ingestion faster? Does it make query faster? Does it support higher throughput compared with M3DB or Netflix Atlas or FB Gorilla? Does it make distributed query more scalable? Does it enable the support of higher cardinality and more dimensions? Does it enable more expressive aggregations or query semantics in general? Does it enable the db to model data beyond multi-dimensional timer series?

Re: Introducing ArcticDB: A Database for Observability

#16

Post by caust1c »

brancz wrote: post #2

Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb

How does it compare to BadgerDB/RocksDB/LevelDB? I see that it's using Arrow and Parquet of course, but the Sparse Index sounds very similar to LSM Tree like storage engines, except using something like a K-Way Merge algorithm and a heap structure to manage that somehow?

I'm more of an operator and user of these systems, so as an operator I care more about the usability than what's underneath, but also am reasonably skeptical of new databases since theres literally hundreds being written every year.

So what benefits does this structure and data format provide over classical LSM-like databases which are currently dominating the high-write-throughput embedded DB space?

Re: Introducing ArcticDB: A Database for Observability

#17

Post by brancz »

hintymad wrote: post #15

It's great to see more solutions in this space, and congratulations on launching and open sourcing it. Is it possible to explain why it matters to an end users? The Why We Built It section seems solely focus on implementation and what matters to the team. > First, we needed something embeddable for Go I'm not sure why it matters if I just need to store and analyze large quantity of profiling data, but let me assume t…

Great point! It’s important to have the data that we want to separately search and aggregate by laid out in their own column and sorted in order to allow fast processing of that data (query latency) but also achieve better compression since repetitive values can be efficiently encoded (saving both memory and disk once we persist data).

We can keep cost of ingestion low because we make trade offs about the mutability of data, as sorting will never change if data is immutable. We globally maintain sorting by requiring writes to be sorted, that way, worst case we look at each in-memory block but in practice at far less when inserting.

One thing to clarify, for now arcticDB is just an embeddable database similar to badger, but it's possible we might make it a distributed database in the future (for now this suffices for what we need it to do).

Let me know if that clarifies it, happy to elaborate further!

Re: Introducing ArcticDB: A Database for Observability

#18

Post by brancz »

caust1c wrote: post #16
brancz wrote: post #2

Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb

How does it compare to BadgerDB/RocksDB/LevelDB? I see that it's using Arrow and Parquet of course, but the Sparse Index sounds very similar to LSM Tree like storage engines, except using something like a K-Way Merge algorithm and a heap structure to manage that somehow? I'm more of an operator and user of these systems, so as an operator I care more about the usability than what's underneath, but also am reasonably…

It's closer to DuckDB rather than Badger/RocksDB/LevelDB, but similar in the sense that it is an embeddable database, not one that is operated standalone. It's not unlike an LSM tree, but the difference is that the leafs in the tree are not individual keys, but rather describe a range of values that are all read at once if read. So this allows high write throughput _and_ high read throughput, trading off mutability (which is ok for the Observability use cases we aim for it to fulfil).

I think it's reasonable to be skeptical about new databases, if it helps, we worked on the Prometheus and Thanos project storage layers before we started the work on this, which now powers hundreds of thousands of monitoring stacks out there.

Re: Introducing ArcticDB: A Database for Observability

#20

Post by brancz »

psanford wrote: post #19

I'm a bit confused by this being an in-memory only database and also using Apache Parquet. Isn't Apache Parquet a file format specification? Whats the point of using it if you aren't serializing data to disk? Is the goal to eventually support durable storage?

You're absolutely right that that's confusing.

Short answer: We just wanted to release this as soon as possible and haven't gotten to it yet.

Slightly longer answer: We currently build parquet buffers in-memory which we are soon going to persist. We still want to finish up some details about how we partition and compact data over time before we do that though. We've learned from previous projects that once we write to disk people start to depend on that :)

Post reply on HN