Will InfluxDB IOx eventually replace InfluxDB v2?
IOx is the data storage layer. It will replace the current TSM data storage system in InfluxDB, but it won't replace InfluxDB as a whole.
IOx: InfluxData’s New Storage Engine
11–20 of 32 posts
Re: IOx: InfluxData’s New Storage Engine
#12Just wanted to also give a shout out to Apache DataFusion[0] that IOx relies on a lot (and contributes to as well!).
It's a framework for writing query engines in Rust that takes care of a lot of heavy lifting around parsing SQL, type casting, constructing and transforming query plans and optimizing them. It's pluggable, making it easy to write custom data sources, optimizer rules, query nodes etc.
It's has very good single-node performance (there's even a way to compile it with SIMD support) and Ballista [1] extends that to build it into a distributed query engine.
Plenty of other projects use it besides IOx, including VegaFusion, ROAPI, Cube.js's preaggregation store. We're heavily using it to build Seafowl [2], an analytical database that's optimized for running SQL queries directly from the user's browser (caching, CDNs, low latency, some WASM support, all that fun stuff).
[0] https://github.com/apache/arrow-datafusion
Re: IOx: InfluxData’s New Storage Engine
#13Isn't InfluxDB rewriting their storage engine for the nth time? It makes me have a little less faith in their project to be honest.
Re: IOx: InfluxData’s New Storage Engine
#14Just wanted to also give a shout out to Apache DataFusion[0] that IOx relies on a lot (and contributes to as well!). It's a framework for writing query engines in Rust that takes care of a lot of heavy lifting around parsing SQL, type casting, constructing and transforming query plans and optimizing them. It's pluggable, making it easy to write custom data sources, optimizer rules, query nodes etc. It's has very good…
DataFusion is great, we're happy to be contributing to it. Also excited to see so many people around the world picking it up and contributing as well. With our development efforts on IOx, it's like a strong tailwind. But we put a ton of effort into helping manage community efforts (thanks, alamb! our developer on IOx that is also on the Arrow PMC).
Re: IOx: InfluxData’s New Storage Engine
#15Isn't InfluxDB rewriting their storage engine for the nth time? It makes me have a little less faith in their project to be honest.
The original TSM engine is still used by InfluxDB v2 OSS.
The InfluxDB Cloud platform uses a variation of TSM that's tailored for a distributed SaaS rather than stand-alone nodes (this was originally intended to be used in InfluxDB v2 OSS as well, but alpha-testing showed that the old engine performed better there so it ultimately was reverted for the beta release).
So IOx is really the first major new storage engine in InfluxDB.
Re: IOx: InfluxData’s New Storage Engine
#16Just wanted to also give a shout out to Apache DataFusion[0] that IOx relies on a lot (and contributes to as well!). It's a framework for writing query engines in Rust that takes care of a lot of heavy lifting around parsing SQL, type casting, constructing and transforming query plans and optimizing them. It's pluggable, making it easy to write custom data sources, optimizer rules, query nodes etc. It's has very good…
> We're heavily using it to build Seafowl, an analytical database that's optimized for running SQL queries directly from the user's browser...
Interesting. Where does seafowl fit in when I compare it with, say, data-stack-in-a-box approach, for ex: meltano + dbt + duckdb + superset [0]? Is my thinking right that seafowl possibly replaces both duckdb (with IOx) and superset (if there's a web front-end)?
Incidentally, dagster had an article up just yesterday making a case for poor-man's datalake with dbt + dagster + duckdb [1]. What does splitgraph replace if I were to use it in a similar setup?
Thanks.
Re: IOx: InfluxData’s New Storage Engine
#17Isn't InfluxDB rewriting their storage engine for the nth time? It makes me have a little less faith in their project to be honest.
I would argue the other way and praise them for the storage engine changes. Each iteration has had drawbacks, but based on the real-world reported usage they've made decisions to better support what customers are asking for and actually running into, as opposed to trying to iterate on the same engine over and over and making assumptions of real-world usage. Sure, there are drawbacks, but at the end of the day they're continuing to make good improvements for their customers.
Re: IOx: InfluxData’s New Storage Engine
#18Just wanted to also give a shout out to Apache DataFusion[0] that IOx relies on a lot (and contributes to as well!). It's a framework for writing query engines in Rust that takes care of a lot of heavy lifting around parsing SQL, type casting, constructing and transforming query plans and optimizing them. It's pluggable, making it easy to write custom data sources, optimizer rules, query nodes etc. It's has very good…
DataFusion is great, we're happy to be contributing to it. Also excited to see so many people around the world picking it up and contributing as well. With our development efforts on IOx, it's like a strong tailwind. But we put a ton of effort into helping manage community efforts (thanks, alamb! our developer on IOx that is also on the Arrow PMC).
Your initial experiments and decision to build on arrow-rs has been great for the project. Thank you and everyone involved.
Re: IOx: InfluxData’s New Storage Engine
#19Will InfluxDB IOx eventually replace InfluxDB v2?
IOx is the data storage layer. It will replace the current TSM data storage system in InfluxDB, but it won't replace InfluxDB as a whole.
Personally, very excited to see this happening. Huge congrats!
Some constructive criticism around naming... You don't have to have Flux in every single damn thing you create!
InfluxDB IOx is not replacing InfluxDB v2 because... It's just a new storage engine.
For querying we have Flux or InfluxQL...
Re: IOx: InfluxData’s New Storage Engine
#20Isn't InfluxDB rewriting their storage engine for the nth time? It makes me have a little less faith in their project to be honest.
Member of the engineering team here - I would break the history into 3 phases:
1) Alpha / Beta phase where we experimented with several off-the-shelf key-value stores (RocksDB, LevelDB, & BoltDB). During this early phase, we learned from observing a wide variety of workloads / use-cases that we needed a custom built engine to achieve our early performance goals. But, using these off-the-shelf key-value stores allowed our (at the time) very small team to focus on developing a useful beta product and gathering user feedback.
2) TSM storage engine for 1.0 - Developed from scratch based on our learnings from phase 1, this was the first production storage engine that shipped with 1.0 in 2016 and carried us through 2.0. It served as the workhorse for 3 - 4 years as both the number of users and size of their workloads skyrocketed, eventually bumping into architectural limits of TSM.
3) IOx - equipped with a larger engineering team and years of experience with a wide variety of workloads and use-cases, IOx was developed to handle rapidly growing time series workloads that users need to handle.