Live data from Hacker News

Datafuse – Modern Real-Time Data Processing in Rust

github.com

11–15 of 15 posts

Re: Datafuse – Modern Real-Time Data Processing in Rust

#11
Curious what the motivation behind rebuilding it in rust is, versus contributing more to Clickhouse? Obviously memory safety is a big one, but is that the only reason?

What are the other goals of the project?

Personally, I'd love to see an easier-to-manage system with replication considered as a first-class feature rather than bolted on at the end.

Re: Datafuse – Modern Real-Time Data Processing in Rust

#13

I almost mixed this up with Apache Arrow DataFusion for a second: https://github.com/apache/arrow-datafusion

That doesn't surprise me at all. I have politely requested that the project considers renaming to avoid this.

https://github.com/datafuselabs/datafuse/issues/654

Re: Datafuse – Modern Real-Time Data Processing in Rust

#14
post #11

Curious what the motivation behind rebuilding it in rust is, versus contributing more to Clickhouse? Obviously memory safety is a big one, but is that the only reason? What are the other goals of the project? Personally, I'd love to see an easier-to-manage system with replication considered as a first-class feature rather than bolted on at the end.

Well. 1. With the improvement of the rust ecosystem, using rust has made database development faster and easy, for example datafuse use the tokio to implement the pipeline https://github.com/datafuselabs/datafuse/tree/master/fuseque....

2. Couldn't agree with you more with easier-to-manage as a first-class feature, but some times easier-to-manage is built on stability, that's what datafuse is trying to do.

Re: Datafuse – Modern Real-Time Data Processing in Rust

#15
post #11

Curious what the motivation behind rebuilding it in rust is, versus contributing more to Clickhouse? Obviously memory safety is a big one, but is that the only reason? What are the other goals of the project? Personally, I'd love to see an easier-to-manage system with replication considered as a first-class feature rather than bolted on at the end.

Well. 1. With the improvement of the rust ecosystem, using rust has made database development faster and easy, for example datafuse use the tokio to implement the pipeline https://github.com/datafuselabs/datafuse/tree/master/fuseque... . 2. Couldn't agree with you more with easier-to-manage as a first-class feature, but some times easier-to-manage is built on stability, that's what datafuse is trying to do.

Awesome, great to hear! I've been using clickhouse for a long time and although we haven't contributed significantly to development, random bugs and issues have been quite painful in the past. Looking forward to what you're able to do!

p.s. Please don't add in-process DNS caching ;-)

https://github.com/ClickHouse/ClickHouse/issues/5287

Post reply on HN