Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sonic: Fast, lightweight and schemaless search back end in Rust

github.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#31

Post by Xylakant »

FridgeSeal wrote: post #26
Xylakant wrote: post #25

Earlier quoted context omitted.

I’d be curious what your expectations and constraints are, but from my experience of running clusters in the double digit TB-Size my ballpark figure for that amount of data would be 2 medium size data nodes and a small tiebreaker. Alternatively, if you can live with the reduced resilience and availability, even a single node might just do. Depends on the expectated churn though, ES really does not like document updat…

Is that double digit TB on ElasticSearch?

Yes, certainly.

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#32

Post by Xylakant »

cinbun8 wrote: post #27
Xylakant wrote: post #25

Earlier quoted context omitted.

I’d be curious what your expectations and constraints are, but from my experience of running clusters in the double digit TB-Size my ballpark figure for that amount of data would be 2 medium size data nodes and a small tiebreaker. Alternatively, if you can live with the reduced resilience and availability, even a single node might just do. Depends on the expectated churn though, ES really does not like document updat…

That does not sound like a good idea. You can't even maintain a quorum of 2 replicas with n=3 on a cluster like that. Losing one data node would be disastrous.

That’s really not how ES replica works. The quorum is formed on the master eligible nodes (hence a tie-breaker) and is only required to elect a master. The elected master designates a primary shard and as many replica as you configure. However, replica shards are replica and may lag. There’s no read quorum or reconciliation or anything happening. If a primary fails, an (in-sync, depending on the version of ES) replica is auto-promoted. The master keeps track of in-sync replica and you can request that writes are on a number of replica before a write returns, but still, no true quorum.

You can absolutely run 2 data/master-eligible nodes plus a single master-eligible tie-breaker node as a safe configuration. The only constraint is that you should have an uneven number of master-eligibile node to avoid a split brain. You also need to understand what the resilience guarantees are for any given number of replica (roughly: each replica allows for the loss of a single random node) and how many replica you can allocate on a given cluster (at most one per data node). That would allow you to run a 2-datanode cluster in a configuration that survives the loss of one node.

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#33

Post by Xylakant »

fnordsensei wrote: post #29
Xylakant wrote: post #25

Earlier quoted context omitted.

I’d be curious what your expectations and constraints are, but from my experience of running clusters in the double digit TB-Size my ballpark figure for that amount of data would be 2 medium size data nodes and a small tiebreaker. Alternatively, if you can live with the reduced resilience and availability, even a single node might just do. Depends on the expectated churn though, ES really does not like document updat…

From what I've learned, running any cluster on fewer than four nodes is not really recommended.

I’ve run quite a few cluster on such a configuration or alternatively 3 data/master-eligible nodes. It’s a safe configuration unless you manage to overload the elected master. But if you’re fighting that issue, you’ll have to go beyond 4 nodes and have a triplet of dedicated master-eligible nodes plus whatever data nodes you need.

I pretty much specifically avoid 4 node clusters. You’d have to either designate 3 of the four nodes as master-eligibile with a quorum of 2 or have all of them master-eligible with a quorum of 3. Both options allow for failure of a single node before the cluster becomes unavailable. Any other configuration would either fail immediately on a node loss (quorum 4) or be unsafe (quorum of 2, allows for split-brain)

I’d much rather opt for 4 data/master eligible nodes plus a dedicated master eligible node with a quorum of 3.

You also need to pick the number of replica suitably: each replica allows for the loss of a single random(!) data node while retaining all your data. Note that if losses are not random but you want to safeguard against loss of a rack or an availability zone or such, configurations are possible that distribute primary and replica suitably (“keep a full copy on either side”)

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#34

Post by Xylakant »

marmaduke wrote: post #28
Xylakant wrote: post #24

Earlier quoted context omitted.

That’s just false. The default config might set the JVM HEAP to 2GB (though I’m fairly certain it’s 1GB) but ES will start up with half a GB of heap with no issue.

It was the default behavior on my install, documented recommendation is half the system memory and the logs don’t provide useful info when the heap allocation fails. Perhaps the upstream default is 1 GB but this was not the default, and there is much confusion on setting these correctly https://stackoverflow.com/a/40333263 Not everyone has time to dig into it; of course if you did that’s good for you

What your system install does and whether the default is good for what you intend to do is a bit different from what the system allows. If you want to use a piece of software, you’re expected to somewhat understand what it does. At least reading the “setup” chapter of the documentation which explains which ENV var to set should be recommended. (I specifically link to the 1.7 docs to show that it’s been in there for quite some time) https://www.elastic.co/guide/en/elasticsearch/reference/1.7/...

The recommendation is btw: at most half of the systems memory, no more than 31g (due to how the jvm compresses heap pointers), and keep at least as much memory available for disk buffers/caches as you allocate to the heap.

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#35

Post by manigandham »

O_H_E wrote: post #11
mellab wrote: post #9

Wow. I’ve spent the last three weeks building a custom search solution in Kotlin - in my case I’m using tokenizers from Lucene and using a radix trie as an index. I actually looked at using Bleve (another rust search lib) initially but it didn’t have the right language support Glancing over this it looks like a nearly perfect fit for my use case I just wish I had seen this a couple of weeks earlier!

And that's why I believe that the search/discovery problem is not solved yet by google

Google is a public service to search the internet. This product and thread is more about adding search functionality to other applications and private data.

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#36

Post by cinbun8 »

Xylakant wrote: post #32
cinbun8 wrote: post #27

Earlier quoted context omitted.

That does not sound like a good idea. You can't even maintain a quorum of 2 replicas with n=3 on a cluster like that. Losing one data node would be disastrous.

That’s really not how ES replica works. The quorum is formed on the master eligible nodes (hence a tie-breaker) and is only required to elect a master. The elected master designates a primary shard and as many replica as you configure. However, replica shards are replica and may lag. There’s no read quorum or reconciliation or anything happening. If a primary fails, an (in-sync, depending on the version of ES) replic…

I was not saying that's how they work. Most prod clusters are configured for high availability and multiple replicas. They have at least 3 nodes and 2 replicas configured for the index. Sure you can run this configuration, but do you really want all your traffic to hit this one instance when the other one goes down?

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#38

Post by StavrosK »

MosheZada wrote: post #37
StavrosK wrote: post #4

This looks pretty cool! I'd like to try it, but it looks like I'll have to wait for the Python client library first.

I just wrote a basic one https://github.com/moshe/asonic

That looks great, thank you! I'll try it out and send you any feedback I have. Also, you might want to link to Sonic in the README, for people unfamiliar with it.

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#39

Post by Xylakant »

cinbun8 wrote: post #36
Xylakant wrote: post #32

Earlier quoted context omitted.

That’s really not how ES replica works. The quorum is formed on the master eligible nodes (hence a tie-breaker) and is only required to elect a master. The elected master designates a primary shard and as many replica as you configure. However, replica shards are replica and may lag. There’s no read quorum or reconciliation or anything happening. If a primary fails, an (in-sync, depending on the version of ES) replic…

I was not saying that's how they work. Most prod clusters are configured for high availability and multiple replicas. They have at least 3 nodes and 2 replicas configured for the index. Sure you can run this configuration, but do you really want all your traffic to hit this one instance when the other one goes down?

> Most prod clusters are configured for high availability and multiple replicas

I've been doing ES consulting and running clusters since 0.14 and I see very few clusters that run more than a single replica. Most 3 node clusters I see are run with three nodes because you can then have three identical configurations at the cost of throwing more hardware at the problem.

> but do you really want all your traffic to hit this one instance when the other one goes down

Whether that's a problem really really depends on whether your cluster is write-heavy or read heavy. Basically all ELK clusters are write heavy and in that case, loosing one of two nodes would also cut writes in half (due to the write amplification that replica cause). Other clusters just have replica for resilience and can survive the read load with half of the nodes available. Whether that is the case for your cluster(s) depends - that's why I explicitly asked what constraints the OP had.

Re: Sonic: Fast, lightweight and schemaless search back end in Rust

#40

Post by MosheZada »

StavrosK wrote: post #38
MosheZada wrote: post #37

Earlier quoted context omitted.

I just wrote a basic one https://github.com/moshe/asonic

That looks great, thank you! I'll try it out and send you any feedback I have. Also, you might want to link to Sonic in the README, for people unfamiliar with it.

Thanks, feedback is appreciated. I've added a link back to Sonic
Post reply on HN