Introducing ArcticDB: A Database for Observability
polarsignals.com
Introducing ArcticDB: A Database for Observability
1–10 of 32 posts
Re: Introducing ArcticDB: A Database for Observability
#2Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it!
It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb
Re: Introducing ArcticDB: A Database for Observability
#3Finally seeing Apache Parquet and Apache Arrow used with Go efficiently and effectively!
Great job. Looking forward to exploring this more in the Prometheus and CNCF Ecosystem.
The underneath library used (https://github.com/segmentio/parquet-go) looks amazing too!
Re: Introducing ArcticDB: A Database for Observability
#4Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb
It looks like there's not a textual query interface to it at the moment. Do I have that wrong or are you interested in adding a high level query interface in the future?
Re: Introducing ArcticDB: A Database for Observability
#5Nice article !
Super curious about your plan for persistence and compression ?
With TSDB interning labels, do you expect any increase of size for the label part ?
And finally any specific reason for not having this under the Parca repo ? IMHO working across multiple repo in go can be a PITA.
Re: Introducing ArcticDB: A Database for Observability
#6Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb
What was the motivation for building your own instead of using something like DuckDB which support parquet out of the box? What are the differences?
Re: Introducing ArcticDB: A Database for Observability
#7Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb
It looks like there's not a textual query interface to it at the moment. Do I have that wrong or are you interested in adding a high level query interface in the future?
Very good question! While we may add a textual query interface (maybe even SQL) in the future, we very intentionally started with this abstraction, since most Observability projects out there (eg. Prometheus, Grafana Loki, Parca) have specialised query languages that we wanted to have a lower level abstractions that those languages could be implemented on top of, instead of having to transpile to SQL (or whatever proprietary language we might have come up with).
Re: Introducing ArcticDB: A Database for Observability
#8Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb
You really should change the front page to why I should use it vs why we built it. I don't care why you built it that it is in Go. I do care why I should use it which must be hidden somewhere in why you built it but I don't have that kind of time.
Re: Introducing ArcticDB: A Database for Observability
#9Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb
What was the motivation for building your own instead of using something like DuckDB which support parquet out of the box? What are the differences?
In addition to other reasons they may have, an embedded database for Go being built in Go means they don't need to require CGO, which DuckDB in Go would require.
Re: Introducing ArcticDB: A Database for Observability
#10Nice article ! Super curious about your plan for persistence and compression ? With TSDB interning labels, do you expect any increase of size for the label part ? And finally any specific reason for not having this under the Parca repo ? IMHO working across multiple repo in go can be a PITA.
Great questions! We're planning on persisting data in parquet format in object storage, potentially with additional metadata to help the query planner optimize queries. In terms of compression, parquet already supports various modern compression mechanisms (zstd, lz4) that we already support if the schema specifies it. Though we have thought about potentially allowing different compression schemes to be used in different situations, for now it's static and part of the schema.
While the pure storage for strings of labels might increase, since we got rid of the inverted index entirely, the saving of that is greater than what we're spending on potentially duplicate strings.
We intentionally put it on the Polar Signals GitHub org to distance it from the Parca project. While we initially developed it for Parca, we think the applications can be much wider so we wanted to emphasize that. I do agree it can be a pain to have it separate, but for now we think having it separate is worth it.