Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Influxdb made the switch from Go to Rust

old.reddit.com

Post reply on HN
Print view View original

71–80 of 162 posts

Re: Influxdb made the switch from Go to Rust

#71

Post by PaulWaldman »

the_duke wrote: post #64
tetha wrote: post #34

Earlier quoted context omitted.

We're migrating off of InfluxDB due to that rollercoaster, honestly. It's hard enough to find time to maintain the monitoring stack at work. Casually dropping "Oh, and now you get to rebuild the entire grafana to change the query language" on that doesn't help. And apparently, version 3 does the same thing, except backwards. Sorry, but at that point, we've decided to rebuild the entire metric visualization once on Ti…

Fair warning, I had serious scaling issues with Timescale. Solutions like Grafana Mimir, Victoria Metrics, Clickhouse, or yes, the new Influx implementation, are much more scalable and will give you much fewer headaches. ClickhouseDB is realy brilliant, btw, it's a powerhouse. Especially with the fairly recent additions that enable hybrid local + S3 option, pushing older metrics to S3 for cheap long-term storage.

Can you elaborate on what scaling issues you had with Timescale?

Re: Influxdb made the switch from Go to Rust

#72

Post by pauldix »

vb-8448 wrote: post #3

Can someone explain what's the InfluxData's market? Or how they make/plan to make money? If we speak about metrics, Prometheus just win.

So it sounds like you're asking about our use cases. We have customers across almost every vertical. But what's most common are application and server monitoring, sensor data (industrial, rockets, satellites, etc.), and network monitoring.

Metrics is certainly one use case that people pay us for. With v3, we expect that real-time analytics and some more data warehousing types of use cases will become interesting. We always envisioned InfluxDB as a store for observational data of all kinds, not just metrics.

On how we make money, we sell our products. We have at this time:

- InfluxDB v1 Enterprise (a self-managed, clustered implementation of InfluxDB)

- InfluxDB v1 Cloud (Enterprise, but as as single-tenant managed service. We still run this for hundreds of customers)

- InfluxDB v2 Cloud (multi-tenant, usage based, we're running this for thousands of customers)

- InfluxDB v3 Serverless (multi-tenant, usage based v3)

- InfluxDB v3 Cloud Dedicated (single-tenant, resource based pricing)

- InfluxDB v3 Clustered (self-managed v3, clustered database)

We'll have single server versions in the future, but we're a bit off from that. Right now our focus is on continuing support for our v1 and v2 customers, and further developing our v3 products for new customers and customers that want to migrate over.

Re: Influxdb made the switch from Go to Rust

#73

Post by d3w4s9 »

This is really surprising for a commercial company. Not about the choice of language itself, but the fact that they prioritize such rewrites over features, similar to the concerns from other commenters. They mention "performance" and "garbage collector" and "error handling" which are almost technical details.

A company can make good money for quite a while as long as the performance is "good enough", and usually focuses on adding features instead of worrying about any rewriting/re-architect until there is a bottleneck or issues start to significantly slow down development. How "successful" such a rewrite still needs to be seen, but this is risky for most companies/products.

Re: Influxdb made the switch from Go to Rust

#74

Post by pauldix »

mardifoufs wrote: post #7

I love influx but damn do they like moving (too?) fast and quickly changing stuff. In a way, it's pretty cool since it means that they don't get stuck with bad decisions for backwards compatibility reasons, but it's a bit of a roller coaster for users. Not sure what's the best solution though. Having a "stable" but fundamentally limited product (I guess influxdb v1) or breaking stuff in hopes of ending up with a way…

We're trying to make the transition from v1 to v3 easier by brining the write and query APIs from that version forward. We wanted to do the same for Flux, but found it was too difficult in the near term. We might be able to do something in the future, but for now we're focused on making core improvements to the v3 engine.

We'll have data migration tools for v1 and v2 into v3 later this year/early next.

Re: Influxdb made the switch from Go to Rust

#75

Post by hu3 »

Is there a comparison of the new InfluxDB against https://clickhouse.com?

I ask because ClickHouse is quite hot at the moment from my experience in consulting and that seems to be reflected in Google Trends [1].

And there are some startups relying on ClickHouse for their log/monitoring products like https://signoz.io and https://hyperdx.io.

[1] https://trends.google.com/trends/explore?date=all&q=ClickHou...

Re: Influxdb made the switch from Go to Rust

#76

Post by efxhoy »

PaulWaldman wrote: post #71
the_duke wrote: post #64

Earlier quoted context omitted.

Fair warning, I had serious scaling issues with Timescale. Solutions like Grafana Mimir, Victoria Metrics, Clickhouse, or yes, the new Influx implementation, are much more scalable and will give you much fewer headaches. ClickhouseDB is realy brilliant, btw, it's a powerhouse. Especially with the fairly recent additions that enable hybrid local + S3 option, pushing older metrics to S3 for cheap long-term storage.

Can you elaborate on what scaling issues you had with Timescale?

I ran into issues with TS too. Main issue I recall now was maintaining a grand total count of events that were already rolled up into daily counts was not fast since it always looked back at all the data. There was no way in the ts patterns to efficiently express it without handrolling something. The issue was that a grand total count can’t be expressed in terms of a hypertable because there’s no time column.

It’s fantastic for workloads that neatly fit in the hypertable pattern though.

Re: Influxdb made the switch from Go to Rust

#77

Post by pauldix »

mardifoufs wrote: post #60
maxprimer wrote: post #41

Earlier quoted context omitted.

Been using both 1.x and 2.x for telemetry (oss & paid both). I am pretty excited with 3.x's interoperability. Archiving to standard data formats makes the data science team's job's easier, and with a more standard ANSI SQL query engine with jdbc support, and high cardinality tags, it will greatly speed up front end development and analysis use cases. As well, I am one of those folks that happens to find the Flux quer…

Yes I think v3 is pretty solid, and it's nice that they are still supporting v1 and v2. But I think the "migration" from v1 to v2 was the "painful" part. Not because it was too hard to migrate (I guess you don't even have to, since it's still supported), but because it introduced a very different approach, that was supposed to be the future of influx, that was just basically dropped in the next release. I think some…

For v3 we're supporting InfluxQL natively in addition to SQL. The InfluxQL implementation is actually just a front end on top of DataFusion, the SQL engine we use.

We really wanted to bring Flux along too, but found that it was too difficult in the near term to have it work well with v3. We spent a bunch of time building a gRPC API that Flux uses to talk to v3 (the same thing we have in our Cloud v2 product), but that API was designed with the previous storage engine in mind. It ended up being brittle and performed very poorly.

So at this point the long term supported languages for InfluxQL and SQL, but we're continuing to support Flux for our customers.

Re: Influxdb made the switch from Go to Rust

#78

Post by voiper1 »

d3w4s9 wrote: post #73

This is really surprising for a commercial company. Not about the choice of language itself, but the fact that they prioritize such rewrites over features, similar to the concerns from other commenters. They mention "performance" and "garbage collector" and "error handling" which are almost technical details. A company can make good money for quite a while as long as the performance is "good enough", and usually focu…

The first comment is from the Cofounder and he lists the _features_ that motivated a rewrite:

>Then there's the question of why we did a rewrite at all. We wanted to get at some important requirements: >Unlimited cardinality >Analytics queries against time series at the performance of a columnar DB >Use object store as the durability layer for historical data (i.e. separate compute from storage) >SQL and broader ecosystem compatibility >All of that stuff taken together meant that we'd be rewriting most of the core of the database. ...

Re: Influxdb made the switch from Go to Rust

#79

Post by pauldix »

d3w4s9 wrote: post #73

This is really surprising for a commercial company. Not about the choice of language itself, but the fact that they prioritize such rewrites over features, similar to the concerns from other commenters. They mention "performance" and "garbage collector" and "error handling" which are almost technical details. A company can make good money for quite a while as long as the performance is "good enough", and usually focu…

In this case, the features we kept getting asked for by our customers necessitated a change in underlying database architecture. I talk about that quite a bit in the reddit thread.

I totally agree that a rewrite is risky. It's not something I'd choose to do again, but at the time we didn't really see any way around rewriting the bulk of the database (even if we kept it implemented in Go).

Using Rust and the Arrow ecosystem of projects (Parquet, DataFusion, Flight) meant that there were a ton of things we didn't have to do from scratch. One of our staff engineers, Andrew Lamb, has called it a toolkit for building databases. Thanks in part to his contributions, I think he's right.

Re: Influxdb made the switch from Go to Rust

#80

Post by mardifoufs »

pauldix wrote: post #74
mardifoufs wrote: post #7

I love influx but damn do they like moving (too?) fast and quickly changing stuff. In a way, it's pretty cool since it means that they don't get stuck with bad decisions for backwards compatibility reasons, but it's a bit of a roller coaster for users. Not sure what's the best solution though. Having a "stable" but fundamentally limited product (I guess influxdb v1) or breaking stuff in hopes of ending up with a way…

We're trying to make the transition from v1 to v3 easier by brining the write and query APIs from that version forward. We wanted to do the same for Flux, but found it was too difficult in the near term. We might be able to do something in the future, but for now we're focused on making core improvements to the v3 engine. We'll have data migration tools for v1 and v2 into v3 later this year/early next.

Thanks for your reply! Dumb question (I couldn't find a definitive answer) but will v3 InfluxQL be compatible with v1? Is there an article about the changes between v1 and v3?
Post reply on HN