Sonic: Fast, lightweight and schemaless search back end in Rust • Hacker News Board

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#21

Post by FridgeSeal » Sun, Mar 24, 2019, 9:25 AM UTC

mverwijs wrote: post #15

FridgeSeal wrote: post #14

Earlier quoted context omitted.

Write speed is fine, it’s more the fact that the dataset is reasonably large, and to run an instance with enough capacity and nodes (even with spill to disk), is silly expensive.

Being in a simular situation what do you consider "reasonably large"?

250-300GB.

Not large by absolute standards sure, but large enough to cause issues.

I’m sure there’s some kind of solution that involves re-architecting the ES cluster and indices and re-architecting the data flows and stuff. But if our options are go through all that, or seriously slim down our architecture and costs by just running Sonic + our data warehouse, I’m definitely going to give it a go. After all, worst comes to worst we can go down the re-architecting ES route if Sonic doesn’t work out.

¯\_(ツ)_/¯

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#22

Post by javitury » Sun, Mar 24, 2019, 9:35 AM UTC

Performance figures are awesome. Also language support is great.

> Sonic only keeps the N most recently pushed results for a given word

This index discards old entries. This is fine for messages, in which aging items lose relevance. Yet the developer uses it for a help desk, which I think should give equal importance to all items.

In this area I would say the main comperitor is Groonga. It can be integrated into PostgreSQL with the PGroonga extension, and it indexes all of the data. However it consumes way more ram.

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#23

Post by manigandham » Sun, Mar 24, 2019, 10:07 AM UTC

Always nice to see more alternatives to Elasticsearch. That project could be so much better with some proper planning and focus.

There's also Toshi: https://github.com/toshi-search/Toshi which is built on top of Tantivy: https://github.com/tantivy-search/tantivy

And for C++, there's Xapiland: https://github.com/Kronuz/Xapiand

And for Go, there's Blast: https://github.com/mosuka/blast built on Bleve: https://github.com/blevesearch/bleve

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#24

Post by Xylakant » Sun, Mar 24, 2019, 10:23 AM UTC

marmaduke wrote: post #17

This looks like a breath of fresh air. Elasticsearch won’t even start if it can’t preallocate 2 GB

That’s just false. The default config might set the JVM HEAP to 2GB (though I’m fairly certain it’s 1GB) but ES will start up with half a GB of heap with no issue.

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#25

Post by Xylakant » Sun, Mar 24, 2019, 10:30 AM UTC

FridgeSeal wrote: post #21

mverwijs wrote: post #15

Earlier quoted context omitted.

Being in a simular situation what do you consider "reasonably large"?

250-300GB. Not large by absolute standards sure, but large enough to cause issues. I’m sure there’s some kind of solution that involves re-architecting the ES cluster and indices and re-architecting the data flows and stuff. But if our options are go through all that, or seriously slim down our architecture and costs by just running Sonic + our data warehouse, I’m definitely going to give it a go. After all, worst co…

I’d be curious what your expectations and constraints are, but from my experience of running clusters in the double digit TB-Size my ballpark figure for that amount of data would be 2 medium size data nodes and a small tiebreaker. Alternatively, if you can live with the reduced resilience and availability, even a single node might just do. Depends on the expectated churn though, ES really does not like document updates.

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#26

Post by FridgeSeal » Sun, Mar 24, 2019, 11:52 AM UTC

Xylakant wrote: post #25

FridgeSeal wrote: post #21

Earlier quoted context omitted.

250-300GB. Not large by absolute standards sure, but large enough to cause issues. I’m sure there’s some kind of solution that involves re-architecting the ES cluster and indices and re-architecting the data flows and stuff. But if our options are go through all that, or seriously slim down our architecture and costs by just running Sonic + our data warehouse, I’m definitely going to give it a go. After all, worst co…

I’d be curious what your expectations and constraints are, but from my experience of running clusters in the double digit TB-Size my ballpark figure for that amount of data would be 2 medium size data nodes and a small tiebreaker. Alternatively, if you can live with the reduced resilience and availability, even a single node might just do. Depends on the expectated churn though, ES really does not like document updat…

Is that double digit TB on ElasticSearch?

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#27

Post by cinbun8 » Sun, Mar 24, 2019, 12:30 PM UTC

Xylakant wrote: post #25

FridgeSeal wrote: post #21

Earlier quoted context omitted.

250-300GB. Not large by absolute standards sure, but large enough to cause issues. I’m sure there’s some kind of solution that involves re-architecting the ES cluster and indices and re-architecting the data flows and stuff. But if our options are go through all that, or seriously slim down our architecture and costs by just running Sonic + our data warehouse, I’m definitely going to give it a go. After all, worst co…

I’d be curious what your expectations and constraints are, but from my experience of running clusters in the double digit TB-Size my ballpark figure for that amount of data would be 2 medium size data nodes and a small tiebreaker. Alternatively, if you can live with the reduced resilience and availability, even a single node might just do. Depends on the expectated churn though, ES really does not like document updat…

That does not sound like a good idea. You can't even maintain a quorum of 2 replicas with n=3 on a cluster like that. Losing one data node would be disastrous.

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#28

Post by marmaduke » Sun, Mar 24, 2019, 1:01 PM UTC

Xylakant wrote: post #24

marmaduke wrote: post #17

This looks like a breath of fresh air. Elasticsearch won’t even start if it can’t preallocate 2 GB

That’s just false. The default config might set the JVM HEAP to 2GB (though I’m fairly certain it’s 1GB) but ES will start up with half a GB of heap with no issue.

It was the default behavior on my install, documented recommendation is half the system memory and the logs don’t provide useful info when the heap allocation fails.

Perhaps the upstream default is 1 GB but this was not the default, and there is much confusion on setting these correctly

https://stackoverflow.com/a/40333263

Not everyone has time to dig into it; of course if you did that’s good for you

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#29

Post by fnordsensei » Sun, Mar 24, 2019, 1:57 PM UTC

Xylakant wrote: post #25

FridgeSeal wrote: post #21

Earlier quoted context omitted.

250-300GB. Not large by absolute standards sure, but large enough to cause issues. I’m sure there’s some kind of solution that involves re-architecting the ES cluster and indices and re-architecting the data flows and stuff. But if our options are go through all that, or seriously slim down our architecture and costs by just running Sonic + our data warehouse, I’m definitely going to give it a go. After all, worst co…

I’d be curious what your expectations and constraints are, but from my experience of running clusters in the double digit TB-Size my ballpark figure for that amount of data would be 2 medium size data nodes and a small tiebreaker. Alternatively, if you can live with the reduced resilience and availability, even a single node might just do. Depends on the expectated churn though, ES really does not like document updat…

From what I've learned, running any cluster on fewer than four nodes is not really recommended.

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#30

Post by d33 » Sun, Mar 24, 2019, 3:10 PM UTC

sidcool wrote: post #8

I would like to understand the code for the project. What approach would help?

Personally, I would advise finding an itch to scratch - something you'd like to see improved. Then try to understand the code from this perspective - where would you put the functionality, which pieces would it be connected to? Try to follow the lead, make notes as you read the code. You'll eventually get a feel of the infrastructure, going from the deal to the big picture - this is my default way of navigating projects.

If at any moment you feel lost, look through related issues and merge requests, as well as Git history. Perhaps you'll see how things get changed in the project, patterns intrinsic to it.

Also, keep in mind that you can always try to contact the community/author or invite someone to try figuring out your goal with you - once you collaborate, you'll get a solution tailored to the way you think. It does engage other people, but it also makes coding social and (at least to me) more satisfying.

Let me know what you think of this approach!