Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meilisearch 1.0 – Open-source search engine built in Rust

blog.meilisearch.com

Post reply on HN
Print view View original

91–100 of 186 posts

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#91

Post by megous »

pdimitar wrote: post #52
megous wrote: post #50

Earlier quoted context omitted.

Yes, but do you realize you don't have to write C code without bounds checks? You can add your own.

No, I never would have imagined. /s Point is, many don't do it. And they don't tell anyone. And they deploy important software. And then we get unpleasantly surprised years (or decades) later. Same argument as with C++: proponents say that the modern C++ is almost like Rust which is cool and I am happy for them, but there are literal hundreds of millions of C++ coding lines out there that will never get upgraded. Hav…

Sorry, I don't know any important C codebase that omits necessary bounds checks casually. Often times it would not even make any sense. (If you want to iterate over an array, you don't just iterate until infinity, lol.) String manipulation is often wrapped in functions that handle length/storage size/reallocation/etc in the background.

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#92

Post by sidmitra »

tpayet wrote: post #12
msvan wrote: post #11

How does Meilisearch compare to ElasticSearch from an operational point of view? I've experienced ElasticSearch to be quite painful to maintain, requiring lots of manual tweaking to balance shards and careful design of indices.

That's the point! We don't ambition to compete with Elastic on everything (logs, analytics, etc). We are doing search for front-end users with a strong focus on relevancy, speed & developer experience. You can read a bit more on our documentation https://docs.meilisearch.com/learn/what_is_meilisearch/compa...

A quick question, are there any limits around number of separate indexes we can have with meilisearch? I'm thinking atleast say 20-30K separate indexes to start with.

My use case is that i want to start creating some indexes that are "per-user" and some "per-company" where a company(customer) might have many users. This is to do some sort of double tenant isolation. I will create different keys that have permission to specific indexes and deliver those to the user somehow. My current solution does hacky things with Elasticsearch like adding query filters by user/company-id attributes in the background automatically. But since meilisearch would be customer facing, i need stronger guarantees around permissions per index.

I tried this out a year ago on Meilsearch locally, but haven't stress tested it by creating thousands of them like production.

Or is there a better way to do this. This is also a reason where memory-only systems like Typesense didn't make sense to me. I'm fine with taking a performance hit by going to disk to pull the right index. Not every index will be used all the time. I might also look at sharding/partitioning features if present.

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#93

Post by pdimitar »

megous wrote: post #91
pdimitar wrote: post #52

Earlier quoted context omitted.

No, I never would have imagined. /s Point is, many don't do it. And they don't tell anyone. And they deploy important software. And then we get unpleasantly surprised years (or decades) later. Same argument as with C++: proponents say that the modern C++ is almost like Rust which is cool and I am happy for them, but there are literal hundreds of millions of C++ coding lines out there that will never get upgraded. Hav…

Sorry, I don't know any important C codebase that omits necessary bounds checks casually. Often times it would not even make any sense. (If you want to iterate over an array, you don't just iterate until infinity, lol.) String manipulation is often wrapped in functions that handle length/storage size/reallocation/etc in the background.

Nobody is saying they do it casually. People genuinely believe they are without fault, which leads to stuff like Heartbleed (and many others; from 2017 to 2020 there was a number of HN submissions about various well-known pieces of software having buffer under/over-flows).

I heard the ideal theory you cite, many many times. Yet many people still do mistakes. How does that fit in your world-view?

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#94

Post by groestl »

nicoburns wrote: post #70
groestl wrote: post #61

Earlier quoted context omitted.

Safety yes, but stability? It's crash early model wrt stack overflows and out-of-memory errors seems to trade off availability for safety. Not proficient in Rust as a user nor as a developer, so an honest question.

> It's crash early model wrt stack overflows and out-of-memory errors seems to trade off availability for safety It's worth noting that while you can catch these kind of errors in C, very little software actually does so. The only software I'm aware of that does this is SQLite. Your C software will more than likely crash in OOM and StackOverflow situations too.

> while you can catch these kind of errors in C

Tbh C would have never crossed my mind as an alternative for Rust in this case. I guess that says something.

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#95

Post by traverseda »

jabo wrote: post #84
traverseda wrote: post #76

Earlier quoted context omitted.

I mean that user report is from me, and was about a very very early meilisearch version. Maybe wrong link? > if you want it to not slow down as it swaps content from Disk to RAM. Obviously it's going to be fastest to run with your entire dataset in RAM, that's never in doubt. Part of why I find the whole typesense comparison page disingenuous is that you're making the ability to swap to disk sound like an anti-featur…

> Part of why I find the whole typesense comparison page disingenuous is that you're making the ability to swap to disk sound like an anti-feature. Didn’t intend it that way. In fact, we recommend that users configure swap space even in Typesense as a safety mechanism. May I know which part of the comparison table makes it sound like that? The one under Index location says: “Disk with Memory Mapped files” for Meilise…

>May I know which part of the comparison table makes it sound like that?

Well a few things. Normally I'd try to coach these a bit kinder and all that, but I hope you don't mind if I just come out and talk about the issues. Keep in mind that these are just my interpretations after a quick read through.

# Bias

> Instant Search-as-you-type Experiences for up to a few hundred thousand records, that don't require a production-grade highly-available setup.

Seeing as meilisearch is your biggest competitor saying that they're not "production grade" sounds biased. "Production grade" is subjective, and I understand why it's written that way you define production grade to include high-availability multi-node configurations. I don't necessarily disagree but I think you need to drop "production grade" from that sentence and just say "high availability". Maybe add a row to your overview talking about high availability since it seems to be one of the factors you consider to be a significant differentiator.

>Only supports a single-node setup, which creates a potential single point of failure and so is not production-ready, despite the v1.x versioning.

This here is another spot where you seem biased. Remove the "despite v1.x versioning", it comes off as petty. I'd also remove the part where you say "is not production ready". You seem to have a very concrete idea of what production looks like, but for me one example of production looks like a raspberry pi in a school house in rural africa (internet in a box project). Under those constraints typesense isn't production ready.

I get what you're saying about "production ready" but there must be another way to word it?

The whole "production ready" line of reasoning comes off as arrogant and petty in general.

>Runtime Dependencies [...] Recommends use of nginx, apache or the like as a reverse proxy in front

Meilisearch is also a single self-contained binary with an embedded http server. I don't think either of you support https. Do you really not recommend the use of a reverse proxy? How do you route subdomains? I guess you're assuming it's running on a stand alone computer with a public facing IP and no SSL? Are you not providing a frontend/dashboard? You've made this sound like a draw back, if you had of put "None. Self-contained binary" in front of it like you did for yourself that would be fine but for this you mention a feature that you have while ignoring what looks to me like the same feature in your competitor.

>Language support

This is also a bit confusing, and I can't help but think it's probably not completely honest. What makes meilisearch different so that it doesn't support "all languages", but elasticsearch does? Meilisearch certainly claims to support all languages where words are seperated by spaces, do you support languages that don't have words separated by spaces?

This implication for this line seems to be that meilisearch isn't indexing on unicode, or something. Just weird, needs more detail probably.

This user claims that meilisearch has better multi-language support: https://news.ycombinator.com/item?id=34708802

So what's the difference?

>Number of Documents

This is completely fine! Good job linking to the pertinent issue and everything. This is how you mention significant drawbacks without seeming biased or petty.

# Target use case

This also seems to be pretty firmly aimed at large enterprise clients. If that's not the impression you're going for, well change the memory model but there's some language in this comparison that can probably help.

> CDN-like Geo-Distributed clusters

Just sounds buzz-wordy to me. Might be fine if I didn't get the impression for the previous paragraphs that my use cases weren't "production ready (webscale?)" and that I'm using it wrong if it's not on a server with 24 TB of ram.

This is more about who the intended customer is than about bias though, so I don't think it's really an issue. Your intended customer isn't some bloke running wordpress on a VPS, it's large scale enterprise and that's fine. If you want to soften that there's a few more things you'll need to change, but when combined with the above stuff about "production ready" it leaves a bit of a bad taste in my mouth, like you'd really rather I be paying you exorbitant rates to run this in your cloud than just using it.

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#97

Post by dureuill »

sidmitra wrote: post #92
tpayet wrote: post #12

Earlier quoted context omitted.

That's the point! We don't ambition to compete with Elastic on everything (logs, analytics, etc). We are doing search for front-end users with a strong focus on relevancy, speed & developer experience. You can read a bit more on our documentation https://docs.meilisearch.com/learn/what_is_meilisearch/compa...

A quick question, are there any limits around number of separate indexes we can have with meilisearch? I'm thinking atleast say 20-30K separate indexes to start with. My use case is that i want to start creating some indexes that are "per-user" and some "per-company" where a company(customer) might have many users. This is to do some sort of double tenant isolation. I will create different keys that have permission t…

Hello!

> A quick question, are there any limits around number of separate indexes we can have with meilisearch?

Yes! In v1.0, about 180 indexes under Linux in the same instance[1]. The good news is that I'm personally working on lifting this limitation for v1.1 (planned to release in the beginning of April), which should be able to accommodate an unlimited number of indexes[2] (disk space permits, of course).

Note that having many indexes does have an impact on performance and will keep doing so even after v1.1.

> Or is there a better way to do this.

If it works for your use case, you can try using a single index (or a few indexes) with tenant tokens[3] for multitenancy.

Hope this helps :-)

[1]: https://docs.meilisearch.com/learn/advanced/known_limitation...

[2]: https://github.com/meilisearch/meilisearch/issues/3382

[3]: https://docs.meilisearch.com/learn/security/tenant_tokens.ht...

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#99

Post by groestl »

cies wrote: post #63
groestl wrote: post #61

Earlier quoted context omitted.

Safety yes, but stability? It's crash early model wrt stack overflows and out-of-memory errors seems to trade off availability for safety. Not proficient in Rust as a user nor as a developer, so an honest question.

So Zig then?

I'd have chosen JVM for safety, ease of deployment, stability and performance, and Java for maturity of the toolchain (fully aware that required memory would be a concern). With that in mind, can you pitch Zig to me?

Re: Meilisearch 1.0 – Open-source search engine built in Rust

#100

Post by AlexAltea »

paraboul wrote: post #60
AlexAltea wrote: post #57

Earlier quoted context omitted.

Is it really "just a single statically linked binary"? I'd love to use Meilisearch as you describe, but their so-called SDKs are just for clients, so you still need the Meilisearch server listening on localhost. I would love to see something like SQLite based on Meilisearch (i.e. a fully self-contained search library like https://github.com/mchaput/whoosh ). Do you know if such a thing exists?

I was referring to the server daemon, not the client libraries. But ofc, it's a process not an embeddable library, so you can't just link it against your app like you would with SQLite or rocksdb. Although it looks like it's build around their core library "milli" though ( https://github.com/meilisearch/milli/ ), so probably something doable in the future?

Thank you very much, that's precisely what I was looking for!
Post reply on HN