Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meilisearch 1.0 – Open-source search engine built in Rust

blog.meilisearch.com

Post reply on HN
Print view View original

121–130 of 186 posts

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#121

Post by tmikaeld »

marban wrote: post #21
tmikaeld wrote: post #20

My team tried to use Meilisearch for large datasets, unfortunately, it's impossible to plan the RAM usage. If you have very little searches, it consumed very little, but if you have a lot of search traffic, it may consume more than we could provision beforehand. This made it too unpredictable and too expensive, so we went with Manticore instead. I don't know if this has been addressed in 1.0, hopefully it has.

Do you have any numeric definitions for few and lots?

Our index was aimed at handling 20 000 documents at total of 35MB of CSV, this would balloon into 0.7GB to 1GB of RAM and we expected at least 1000 of these indexes, which would require dedicated servers with 1TB of RAM. This was when Meili was at version 0.27.

With manticore, we've tried to run into these issues in benchmarks, but the only problem we got was temporary high IO load when indexes need to be re-indexed with new or changed documents. In total it's at 50-70% of the RAM usage compared to Meili.

We'd be happy to re-visit, but looking at the docs - it seems to be about the same as it was back then (a year ago).

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#122

Post by tmikaeld »

rsstack wrote: post #114
sandstrom wrote: post #111

Earlier quoted context omitted.

You could have a look at https://github.com/lucaong/minisearch/

Wow, this might fit our needs much better! Thanks!

Hot tip, we experimented with running minisearch in RAM on cloudflare workers and it works excellent for up to 5MB of index due to it being under the 50ms CPU time.

This means, 10M search requests for 5$. The only drawback is that it's expensive to re-index, but if your use case don't require that, it's hard to beat!

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#124

Post by schappim »

We’ve used Meilisearch in production and it is the closest thing to self hosted Algolia you can get, which in itself is pretty amazing.

Unfortunately the performance of indexing (constantly changing records) wasn’t great and Meilisearch would fall behind on indexing records for hours.

Meilisearch has been amazingly great for projects where records don’t change all that much (eg docs, or even a customer database), but if you have for example a fast paced ecommerce system with 50k records constantly changing (eg product inventory), it falls over pretty quick. We had to transition over to Elastic for this aspect of our app.

The other issue we faced is their Rails gems falling out of step with the server, and when fixes came out, the Rails gem was incompatible for a while.

I really really hope 1.0 increases performance to the point where it becomes production ready, because the initial out of the box performance (before getting bogged down with indexing) was pretty amazing. Better than Elastic and on par with Algolia.

I recommend keeping Meilisearch on your radar. It is going to be great.

I wish the best for the Meili team and hope they succeed!

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#125

Post by mcronce »

Yoric wrote: post #105
drcongo wrote: post #17

Earlier quoted context omitted.

I'm not a Rust dev, but I am the target market for this product and I kinda care that it's written in Rust. That gives me some (possibly entirely wrong) confidence that it's likely to be a single binary, easily installed, fast and relatively safe. More broadly, if there had been two headlines on the front page today and the other said "Open source search engine written in Node / JS" I would make assumptions about the…

For context (and I say that as a Rust developer), please note that Rust has the same strength/weakness as Node in terms of dependencies. There is ongoing work to strengthen this. I do not know the status.

Yes and no - deploying a Node project, I need to install all its dependencies. Deploying a Rust project, I still typically only need to pull down the binary. The general attitude is still to pull in dependencies to do a job instead of inventing your own solution, which I consider a good thing, but not everybody agrees.

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#126

Post by Yoric »

drcongo wrote: post #110
Yoric wrote: post #105

Earlier quoted context omitted.

For context (and I say that as a Rust developer), please note that Rust has the same strength/weakness as Node in terms of dependencies. There is ongoing work to strengthen this. I do not know the status.

Thanks. Would I be correct in assuming that there should be less of a burden on me as an end user with Rust though as I only need to update the one binary that I installed?

How did you install the binary? If you installed it from source, you'll need to `cargo update && cargo build`.

If you downloaded a binary or installed it from your distro's repo, generally you just need to update that one binary, yes.

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#127

Post by bayesian_horse »

wiradikusuma wrote: post #123

I see comparison against other search engines, but how does it compare to RDBMS full text search e.g. Postgre's? I know it's not apple-to-apple, but most people start with RDMBS.

As far as I understand it a search engine like this is meant to perform well on "Human" queries that are hard to formalize.

SQL queries, asking for records based on something like field a has to contain b or something like that are easy to formalize and fulfill by an RDBMS. But the SQL queries get hairier and hairier when the query involves multiple fields or even multiple unrelated tables. Or free form text. And those queries are harder to index.

On top of all of that, Humans often want things sorted in an order that isn't straight-forward to express in SQL. What is "relevancy"? All of that can be done in SQL, but it's not what RDBMS engines shine at.

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#128

Post by Kerollmops »

schappim wrote: post #124

We’ve used Meilisearch in production and it is the closest thing to self hosted Algolia you can get, which in itself is pretty amazing. Unfortunately the performance of indexing (constantly changing records) wasn’t great and Meilisearch would fall behind on indexing records for hours. Meilisearch has been amazingly great for projects where records don’t change all that much (eg docs, or even a customer database), but…

Thank you very much for this amazing feedback, really appreciated.

We did a lot of improvement to the indexing part of the engine and now can auto-batch updates which gaves incredible improvements. We will continue to work on this in 2023. Can I know the version you were using?

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#129

Post by bayesian_horse »

Yoric wrote: post #101
chimen wrote: post #10

Is Rust that important that you have to place "built in Rust" in the title? Is this like a cult following that we only bet on traffic and interest coming from other evangelists where Rust is the only feature that matter? 4 months ago: " Meilisearch, open-source alternative to Algolia in Rust lands a $15M Series A" It's not the first time I see, there are at least 2-3 daily submissions reaching the FP in this manner s…

I believe that there is a good reason. Rust is currently in the process of trying to eat some of C++'s cake (as well as that of Java, C# or Go). The usual response from C++ (Java, etc.) devotees is that Rust hasn't been tried on large projects so it cannot be compared. Which absolutely makes sense. Each large scale project that demonstrates that Rust can be used successfully in a domain where C++ (Java, etc.) traditi…

People overreact to hype and anti-hype. Rust is already a useful tool and has momentum. But it's not trivial to get into and it won't replace C++ codebases overnight either.

Based on historical data, a good lower bound for its future could be Ruby. According to TIOBE, Rust overtook Ruby in popularity, while Ruby has maintained roughly the same popularity for years. At worst, I expect Ruby to stay about as relevant as Ruby on Rails. But it doesn't look that way...

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#130

Post by megous »

pdimitar wrote: post #93
megous wrote: post #91

Earlier quoted context omitted.

Sorry, I don't know any important C codebase that omits necessary bounds checks casually. Often times it would not even make any sense. (If you want to iterate over an array, you don't just iterate until infinity, lol.) String manipulation is often wrapped in functions that handle length/storage size/reallocation/etc in the background.

Nobody is saying they do it casually. People genuinely believe they are without fault, which leads to stuff like Heartbleed (and many others; from 2017 to 2020 there was a number of HN submissions about various well-known pieces of software having buffer under/over-flows). I heard the ideal theory you cite, many many times. Yet many people still do mistakes. How does that fit in your world-view?

Not sure what ideal theory you mean. I didn't state any.

Most of the code in my Linux distro is written in C, yet I don't see many segfaults or data corruption in my favorite tools, even those exposed to the internet. It just works. Supposed buffer overflows and double-frees don't affect me daytoday despite 95%+ code I run being written in C, "catastrophic" issues like heartbleed notwithstanding.

People make mistakes, sure. They'll make them with "safe" languages, too. Rust programs are not immune from mistakes. They'll just be of a different kind.

PHP is memory safe, and there were many easily exploited (not just exploitable) vulnerabilities in software written with PHP. (and it doesn't even have escape hatches out of its memory safety)

Post reply on HN