Can you rewrite it in Rust?
Meilisearch 1.0 – Open-source search engine built in Rust
41–50 of 186 posts
Re: Meilisearch 1.0 – Open-source search engine built in Rust
#42Earlier quoted context omitted.
It's important, yes. If it was written in C/C++ I'll fully expect to have my servers pwned due to a memory safety bug that these (and many other) languages don't protect against. There's a number of technical people with decision-making powers that pay attention. And a part of them prioritize Rust-written projects. It's a sound (literally) and safe investment. I don't get the people getting ticked off by the "written…
Milli uses unsafe for what it's worth so it's not as safe as you may think. I do assume any post with "written in rust" does better on hacker news.
Question is: do you realize that Rust's `unsafe` is still not as unsafe as C without bound checks, double `free`-s, and others?
Point is, it's still an improvement. I am regularly amazed as to why that factually correct and (more and more) time-proven argument is always skipped when criticizing Rust.
Re: Meilisearch 1.0 – Open-source search engine built in Rust
#43How does Meilisearch compare to ElasticSearch from an operational point of view? I've experienced ElasticSearch to be quite painful to maintain, requiring lots of manual tweaking to balance shards and careful design of indices.
I've been using Meilisearch in production for quite some time now, and TBH it has been one of the easiest service to maintain (I mean, it's just a single statically linked binary) and with close to zero configuration.
Re: Meilisearch 1.0 – Open-source search engine built in Rust
#44Earlier quoted context omitted.
typesense did their own comparison here: https://typesense.org/typesense-vs-algolia-vs-elasticsearch-...
Unfortunately, the comparison with Meilisearch is not up to date in this link. Also, we have to keep in mind that every comparison written by a company is always oriented.
I maintain that comparison page on the Typesense side. I just updated it as recently as yesterday, based on my observation.
But let me know which ones need updating for Meilisearch. Happy to update.
While we’re on the topic, reminder about some of the outdated information in your comparison pages: https://twitter.com/typesense/status/1620825236055932928?s=4...
Re: Meilisearch 1.0 – Open-source search engine built in Rust
#45My team tried to use Meilisearch for large datasets, unfortunately, it's impossible to plan the RAM usage. If you have very little searches, it consumed very little, but if you have a lot of search traffic, it may consume more than we could provision beforehand. This made it too unpredictable and too expensive, so we went with Manticore instead. I don't know if this has been addressed in 1.0, hopefully it has.
Do you have any numeric definitions for few and lots?
I think that they might have fixed it. I noted this as a problem with earlier meilisearch releases as well, but reading through the documentation it looks like they don't require the entire index to be in memory any more, allowing it to be a memory mapped file.
https://docs.meilisearch.com/learn/advanced/storage.html#lmd...
>For the best performance, it is recommended to provide the same amount of RAM as the size the database takes on disk, so all the data structures can fit in memory.
> [...]
>It is important to note that there is no reliable way to predict the final size of a database. This is true for just about any search engine on the market—we're just the only ones saying it out loud.
Looks like a 10MB document is taking ~200MB, from their docs. I don't think that scales linearly though, since it's a reverse index it is going to scale based on the number of unique words it finds, with each document adding a bit on top of that. You'd expect it to have a pretty big index to cover common english words, and then each document adds a bit on top of that.
Definitely seems like somewhere they could make some improvements though. Some transparent compression could probably help, and with zstd's dictionary feature it can be fine tuned to the data they're actually seeing.
Not about to replace xapian in kiwix (offline wikipedia reader) any time soon, I think.
Re: Meilisearch 1.0 – Open-source search engine built in Rust
#46Earlier quoted context omitted.
Milli uses unsafe for what it's worth so it's not as safe as you may think. I do assume any post with "written in rust" does better on hacker news.
Question is: do you realize that Rust's `unsafe` is still not as unsafe as C without bound checks, double `free`-s, and others? Point is, it's still an improvement. I am regularly amazed as to why that factually correct and (more and more) time-proven argument is always skipped when criticizing Rust.
I'm not criticizing Rust, I really like Rust and think it's great, along with the trend of more things written in it.
At the same time assuming you'll never get a memory issue is over the top.
Re: Meilisearch 1.0 – Open-source search engine built in Rust
#47This looks really cool and I might try the self hosted option out on my small website as an upgrade from Postgres’ full text search. I was hoping the cloud version would be more appealing, granted there seems to be a generous free tier but the next option is $1200 a month?!
Sorry, it might not be obvious, but you can go over the free tier and pay for the usage at 0.25$ for each 1000 searches/documents :)
Ah yep sorry I missed that! Good to know. I just saw the next option was $1200 and my eyes became fixated on the number.
Maybe I will try out the cloud version then even though I expect my site would probably be well in the free tier limit, like I said it seems like a very generous tier.
Re: Meilisearch 1.0 – Open-source search engine built in Rust
#48Earlier quoted context omitted.
Question is: do you realize that Rust's `unsafe` is still not as unsafe as C without bound checks, double `free`-s, and others? Point is, it's still an improvement. I am regularly amazed as to why that factually correct and (more and more) time-proven argument is always skipped when criticizing Rust.
I'm not criticizing Rust, I really like Rust and think it's great, along with the trend of more things written in it. At the same time assuming you'll never get a memory issue is over the top.
Dunno, I think it should be clear for all experienced devs that the "never" part is still not achieved.
Personally though -- and in my work -- I'll take any improvement that I can. I am sick of reading about yet another important piece of software having yet another memory safety zero day pwnage bug.
Re: Meilisearch 1.0 – Open-source search engine built in Rust
#49compared to https://typesense.org/ ?
they are very similar. I've tested both intensively a few months ago, ended up w Typesense for performance reason.
My test data set is 1.5M doc * 3-10 fields * 10-50 characters. Meilisearch has slightly better multi-language support, but typesense is much better on batch reindex speed and ram usage while a bit shy on supporting asian languages. The query speed is similar in light to medium load, I didn't stress test on query.
Re: Meilisearch 1.0 – Open-source search engine built in Rust
#50Earlier quoted context omitted.
Milli uses unsafe for what it's worth so it's not as safe as you may think. I do assume any post with "written in rust" does better on hacker news.
Question is: do you realize that Rust's `unsafe` is still not as unsafe as C without bound checks, double `free`-s, and others? Point is, it's still an improvement. I am regularly amazed as to why that factually correct and (more and more) time-proven argument is always skipped when criticizing Rust.
Yes, but do you realize you don't have to write C code without bounds checks? You can add your own.