Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Influxdb made the switch from Go to Rust

old.reddit.com

Post reply on HN
Print view View original

151–160 of 162 posts

Re: Influxdb made the switch from Go to Rust

#151

Post by jokethrowaway »

jacobgorm wrote: post #135

My problem with Rust is that compilation is too slow, as is downloading the gazillion crates needed to go anywhere. When I heard it was going to replace C I was expecting similar build times, but in reality Rust builds much slower than C++. With C/C++ (and CMake + Ninja) it seemed we were finally getting to a point where incremental builds would complete before hitting the 400ms attention span "Doherty Threshold", an…

It's way better compared to the early days and we have incremental compilation in dev. I don't consider it a problem anymore.

People got the memo that slapping Serialize on everything has a cost an unless you're doing huge native dependencies which require long compilation time, it's pretty snappy.

Re: Influxdb made the switch from Go to Rust

#152

Post by hagen1778 »

manicennui wrote: post #145
hagen1778 wrote: post #131

Earlier quoted context omitted.

That's only a matter of time when they hire younger engineers who are familiar with modern monitoring systems and eager to apply their knowledge in practice.

What is it that you think prometheus offers over other solutions? It is more likely that the younger engineer is going to learn that companies don't care about what is popular on HN.

> What is it that you think prometheus offers over other solutions?

I like Prometheus and think this is a great piece of software. But even if we won't go into actual details, Prometheus is baked in into Kubernetes monitoring [0]. That's the first monitoring system young engineers will meet with when learning k8s. Although, k8s and Prometheus are both CNCF projects which means both of them will be promoted in synergy with each other.

> It is more likely that the younger engineer is going to learn that companies don't care about what is popular on HN.

This is not what I think younger engineers do :)

[0] https://kubernetes.io/docs/tasks/debug/debug-cluster/resourc...

Re: Influxdb made the switch from Go to Rust

#153

Post by lbruder »

asah wrote: post #138

How big are y'all that you need InfluxDB? For one database that receives 100M 600 byte JSON records/day, a single node AWS PostgreSQL RDS instance is handling it effortlessly, and the DBA work is very part-time. We keep year+ of summaries and 48 hours of detail, unloading the rest to S3 as parquet files, queryable by Athena if we need. AFAICT, we're spending p.s. a buddy at a top-3 TV streaming service is also doing…

we're storing 120M records a day in influx 1.8, offloading cold data to S3, all on a single m5.xlarge instance that runs other backend services on the side. Less than 400 bucks per month overall aws cost, with lots of other VMs in there. We could use RDS too, but why change it to something more expensive if it already works...

Re: Influxdb made the switch from Go to Rust

#154

Post by lelanthran »

pauldix wrote: post #65
techn00 wrote: post #4

tl;dr - No garbage collector - Fearless concurrency (thanks Rust compiler) - Performance - Error handling - Crates - they thought they were gonna use C++ and wanted interop (ended up not using C++?) - ecosystem: Apache Arrow DataFusion - "I thought that if we're going to rewrite most of the database anyway, we might as well do it in the best language choice in 2020" But the real reason might be: "Rust good, Go bad" /…

I still love Go, but I think Rust is a better fit for very performance sensitive systems software like a database.

If you're scaling horizontally it's doubtful that language choice of rust Vs Go is going to make a difference.

Not enough, anyway, to make a complete rewrite more profitable over adding features.

After all, it's not as easy if influx was plagued by concurrency bugs, was it?

A move to a safer language than go just doesn't seem worth it, and there's little to no performance gain when you can just throw more hardware for that tiny performance difference.

Re: Influxdb made the switch from Go to Rust

#155

Post by kiitos »

manicennui wrote: post #124
GauntletWizard wrote: post #5

Earlier quoted context omitted.

I am about the biggest Prometheus stan that you can find, but I will not mock or denigrate influx db. They are a runner-up, but there is room in this market for runner-ups, and their feature set does match some that Prometheus is not good at.

I think you'd find that Prometheus is way behind a number of solutions in the corporate world, including Graphite, DogStatsD, etc. Popularity on HN does not translate to real world popularity.

In the year of our lord 2023, Graphite, Statsd, etc. are anachronisms. Prometheus is wildly superior to those tools.

Re: Influxdb made the switch from Go to Rust

#156

Post by pbronez »

efxhoy wrote: post #144
mardifoufs wrote: post #142

Earlier quoted context omitted.

What's the difference between the watermark and a time column? Not the person you were replying to but I'm curious since I also thought that TimeseriesDB had a similar "timestamp" to influx.

TS requires you to specify a time column in each hypertable or view (continuous aggregate) where you want it to work its magic. It then stores an internal watermark that it compares to the time column in the table to figure out from where to read when refreshing. My issue was that for a grand total I didn’t have a time column, so I couldn’t define my query as a continuous aggregate and the query had to start counting…

Perhaps: add a time column with an artificially huge time range? This defines the grand total as an interval sum that happens to include all possible intervals.

Re: Influxdb made the switch from Go to Rust

#157

Post by pauldix »

roygbiv2 wrote: post #137
pauldix wrote: post #101

Earlier quoted context omitted.

We announced the availability of the v3 successor to Enterprise v1. It supports the v1 API. We're still building data migration tooling, but if you're interested in testing it out just email support or your sales rep.

How does one upgrade from v2 beta to the latest v2? The docs for doing that seem to no longer exist https://github.com/influxdata/influxdb/issues/24393

To be honest I'm not sure. Upgrading individual releases on the way should take you there, but the v2 beta was quite a while ago.

Re: Influxdb made the switch from Go to Rust

#158

Post by brabel »

pauldix wrote: post #67
brabel wrote: post #30

Earlier quoted context omitted.

The author was looking for an excuse to use Rust for something since 2018: https://www.influxdata.com/blog/rust-can-be-difficult-to-lea... The rewrite started in 2020... they rationalize now, but it's pretty clear they just really wanted Rust and found the reasons they list later as a post-decision justification. Nothing wrong with that, if you don't mind risking the future of your business on a risky rewrite... thou…

To be fair, we didn't drop everything and do a rewrite. Over the last 3.5 years (the length of time for this project), our total engineering team has ranged from 50-90 people. For the first year it was me and two other people. Then for the 2 years following that it was 9 people total. It wasn't until late last year that we made the decision to go all in on the rewrite and made that the focus of everyone in engineerin…

I think that was a risky move, but sometimes you need to take risks to get high rewards, so I agree it's sometimes the best move given your circumstances... I imagine you made the calculation that this was a risk worth taking... do you already know if that is the case? Are your developers "happier"? Customers praising improved performance? More people applying for jobs due to Rust? Fewer bugs? Easier to implement features?

Re: Influxdb made the switch from Go to Rust

#159

Post by asah »

lbruder wrote: post #153
asah wrote: post #138

How big are y'all that you need InfluxDB? For one database that receives 100M 600 byte JSON records/day, a single node AWS PostgreSQL RDS instance is handling it effortlessly, and the DBA work is very part-time. We keep year+ of summaries and 48 hours of detail, unloading the rest to S3 as parquet files, queryable by Athena if we need. AFAICT, we're spending p.s. a buddy at a top-3 TV streaming service is also doing…

we're storing 120M records a day in influx 1.8, offloading cold data to S3, all on a single m5.xlarge instance that runs other backend services on the side. Less than 400 bucks per month overall aws cost, with lots of other VMs in there. We could use RDS too, but why change it to something more expensive if it already works...

thanks! killer example.

how are you dealing with failover?

how are you dealing with interactive reporting?

how large are your records?

is the s3 data in cold storage? how much is stored in S3 right now? is it warm-enough to query via (e.g.) athena?

Re: Influxdb made the switch from Go to Rust

#160

Post by smabie »

brabel wrote: post #30
zzzeek wrote: post #9

> So this isn't the approach I'd recommend for this kind of project, but we started fresh from scratch. wow, the from scratch rewrite. I can't even imagine that for a major piece of software

The author was looking for an excuse to use Rust for something since 2018: https://www.influxdata.com/blog/rust-can-be-difficult-to-lea... The rewrite started in 2020... they rationalize now, but it's pretty clear they just really wanted Rust and found the reasons they list later as a post-decision justification. Nothing wrong with that, if you don't mind risking the future of your business on a risky rewrite... thou…

Smaller job pool?

How does learning Rust reduce your chances of getting a new job? Learning new stuff increases your chances, not decreasea them.

Post reply on HN