Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meilisearch 1.0 – Open-source search engine built in Rust

blog.meilisearch.com

Post reply on HN
Print view View original

71–80 of 186 posts

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#71

Post by jabo »

curquiza wrote: post #55
jabo wrote: post #44

Earlier quoted context omitted.

I maintain that comparison page on the Typesense side. I just updated it as recently as yesterday, based on my observation. But let me know which ones need updating for Meilisearch. Happy to update. While we’re on the topic, reminder about some of the outdated information in your comparison pages: https://twitter.com/typesense/status/1620825236055932928?s=4...

We sent mails but we got no updates on them.

Hmmm, I remember those emails and I did reply to gmourier, and made almost all of the changes he pointed out, to our comparison page. Here's [1] the exact commit with the changes I made.

The only one change I didn't make is the one about Meilisearch not being constrained by RAM, because of reports like this [2] I've seen in the past and because I saw this in your docs:

https://docs.meilisearch.com/learn/advanced/storage.html#mem...

>For the best performance, it is recommended to provide the same amount of RAM as the size the database takes on disk, so all the data structures can fit in memory.

[1] https://github.com/typesense/typesense-website/commit/0103ff...

[2] https://news.ycombinator.com/item?id=34708658

Let me know which other ones need updating.

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#72

Post by remram »

traverseda wrote: post #51
jabo wrote: post #44

Earlier quoted context omitted.

I maintain that comparison page on the Typesense side. I just updated it as recently as yesterday, based on my observation. But let me know which ones need updating for Meilisearch. Happy to update. While we’re on the topic, reminder about some of the outdated information in your comparison pages: https://twitter.com/typesense/status/1620825236055932928?s=4...

I'd say that the bit where typesense can only work with data that fits in ram is actually a pretty big problem for a lot of use cases, as an aside. That feature alone would discount typesense for basically all of my personal projects. Might be a trade off I'd be willing to make on a professional project given the other features but it seems really wasteful. Personally I find the meilisearch comparison to be more usef…

That is also MeiliSearch's recommended setup though:

> For the best performance, it is recommended to provide the same amount of RAM as the size the database takes on disk

https://docs.meilisearch.com/learn/advanced/storage.html#lmd...

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#73

Post by kristiandupont »

dureuill wrote: post #62
kristiandupont wrote: post #36

I am using the core (called "Milli") in a local indexer that I run on my repositories and Obsidian files. It works like a charm and I am very happy with it. Obviously that's a use case with very little traffic but just indexing my repositories folder is quite a bit of work and it does it surprisingly fast. The only real thing I am missing is a typeahead feature.

Hello from a Meilisearch team member, wow your project looks very interesting. How do you handle things like the filesystem changing while your indexer is offline? Do you reindex from scratch at startup? Regarding typeahead, is this what we call "query suggestions"[1]? At the moment, we think that this is something that frontends and SDK can provide rather than the engine, so that means you wouldn't find it at the Mi…

Thank you! Yes, I reindex. I store the file timestamp along with the contents, so it's not quite as involved as it could seem but startup does take a bit. And, I don't have a good way of discovering deleted files at the moment. Not a big deal as it is, but something I will look into.

And yes, query suggestions are exactly what I mean. Thank you for informing me, I guess I will have to look into how I can make it myself :-)

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#74

Post by hu3 »

adamnemecek wrote: post #69
dividedbyzero wrote: post #18

Earlier quoted context omitted.

Compared to what?

Anything. Legit no language comes even close in terms of how easy it is to git clone something and get it to build.

I'd argue Go projects tend to be easier to build since they require nightly Go builds much less frequently (I don't even remember a project that ever required nightly Go tbh).

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#75

Post by dureuill »

kristiandupont wrote: post #73
dureuill wrote: post #62

Earlier quoted context omitted.

Hello from a Meilisearch team member, wow your project looks very interesting. How do you handle things like the filesystem changing while your indexer is offline? Do you reindex from scratch at startup? Regarding typeahead, is this what we call "query suggestions"[1]? At the moment, we think that this is something that frontends and SDK can provide rather than the engine, so that means you wouldn't find it at the Mi…

Thank you! Yes, I reindex. I store the file timestamp along with the contents, so it's not quite as involved as it could seem but startup does take a bit. And, I don't have a good way of discovering deleted files at the moment. Not a big deal as it is, but something I will look into. And yes, query suggestions are exactly what I mean. Thank you for informing me, I guess I will have to look into how I can make it myse…

You could maybe use something equivalent to the "index hot swap"[1] feature we have at the Meilisearch level at startup, so that you make the reindexing in a another index at startup, and then atomatically swap this fresh index with the old one when it is ready? That way, you have fast startup at the cost of having possibly out-of-date information for a while after startup.

(you could even reindex from scratch completely in the background at startup, so no need to discover deleted files at all)

[1]: https://blog.meilisearch.com/zero-downtime-index-deployment/

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#76

Post by traverseda »

jabo wrote: post #68
traverseda wrote: post #51

Earlier quoted context omitted.

I'd say that the bit where typesense can only work with data that fits in ram is actually a pretty big problem for a lot of use cases, as an aside. That feature alone would discount typesense for basically all of my personal projects. Might be a trade off I'd be willing to make on a professional project given the other features but it seems really wasteful. Personally I find the meilisearch comparison to be more usef…

Typesense follows a memory model similar to Redis - you need sufficient RAM to hold the entire dataset. I don't want to speak for the Meilisearch team, but from observing user reports like this [1], it seems to me like you'd need at least X-2X RAM to run Meilisearch, if X is the size of your dataset, if you want it to not slow down as it swaps content from Disk to RAM. [1] https://news.ycombinator.com/item?id=3470865…

I mean that user report is from me, and was about a very very early meilisearch version. Maybe wrong link?

> if you want it to not slow down as it swaps content from Disk to RAM.

Obviously it's going to be fastest to run with your entire dataset in RAM, that's never in doubt. Part of why I find the whole typesense comparison page disingenuous is that you're making the ability to swap to disk sound like an anti-feature. The whole things just sounds biased in a way that the meilisearch comparison doesn't.

There are some killer features in typesense for sure, just my first impression of it is that it's very much aimed at someone other than me.

>Typesense follows a memory model similar to Redis

The difference is that redis is primarily being used as a cache, or for IPC, or as a task-queue. You're not loading a whole bunch of data into, and you expect that the data you have in it will either be short-lived (IPC, queue) or can be evicted with no issues (caching).

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#78

Post by sodapopcan »

wejick wrote: post #65
spoiler wrote: post #40

Earlier quoted context omitted.

As some people mentioned: I generally expect a higher standard of software when I see "build in Rust". That expectation includes a few things such as stability and operational UX (ie how easy it is to run and maintain). And these (in my experience as a Rust developer) stems from the fact that it's much easier to get the MVP and business logic taken care of becau I'm not bogged down by the drudgery of menial tasks tha…

While I can agree on the argument that rust offer many ergonomics and keep us away on many classes of security and memory management related issues. The quality of UX/DX is more defined by product and design requirements, not much to do with language of choice. Similar like restful api, good or bad is on the designer hand most of the time, not because it's implemented on some esoteric language.

OP said "devtime cost", ie, time to implement. They made no mention or insinuation of quality.

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#79

Post by kristiandupont »

dureuill wrote: post #75
kristiandupont wrote: post #73

Earlier quoted context omitted.

Thank you! Yes, I reindex. I store the file timestamp along with the contents, so it's not quite as involved as it could seem but startup does take a bit. And, I don't have a good way of discovering deleted files at the moment. Not a big deal as it is, but something I will look into. And yes, query suggestions are exactly what I mean. Thank you for informing me, I guess I will have to look into how I can make it myse…

You could maybe use something equivalent to the "index hot swap"[1] feature we have at the Meilisearch level at startup, so that you make the reindexing in a another index at startup, and then atomatically swap this fresh index with the old one when it is ready? That way, you have fast startup at the cost of having possibly out-of-date information for a while after startup. (you could even reindex from scratch comple…

That's a great idea, thank you!

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#80

Post by Kerollmops »

rockwotj wrote: post #39
curquiza wrote: post #16

Earlier quoted context omitted.

Hello! For me, "built in Rust" can be a real marketing argument. Indeed, Rust is a language that has proved its safety in the past. Building a technical product in Rust guarantees stability and safety (no memory issues in general) and performance (no garbage collector issue), so it brings more trust to the users.

FWIW milli does use unsafe in places. Also I would recommend not conflating no GC and performance. There are lots of reasons for Rust being fast and many have nothing to do with no GC. The main reasons a lot of languages with GC are slower is due to allocating on the heap as opposed to the stack, and in general Rust does a lot of static linking and the compiler has the full amount of information to optimize calls wit…

I am the co-founder and maintainer of the engine, and I confirm we have some localized unsafe blocks for when we interface with the C library: LMDB.

However, I prefer having a few unsafe blocks that I can review carefully than a single one encapsulating the primary function.

Post reply on HN