11–20 of 28 posts
So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services instead? What an awful idea! I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services instead? What an awful idea! I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
You can easily emulate object storage with e.g. s3 compatible APIs. And probably it will just support whatever file system, including nfs (currently not supported). So, that should not be a show stopper. And technically this could actually reduce your cost by a lot because you will be able to use some central network storage instead of having to have a lot of high end SSDs. You might still want to use those for cachi…
A cloud offering that decouples storage from compute makes this a lot easier and becomes more of a no brainer for 90% of the use cases.
If you're an outlier with TBs or PBs of search data this probably you can keep using on prem if you want. Though I don't really immediately grant that it's worse for your usecase, especially when object storage is insanely cheap and supposing they provide a compat layer for S3 you can get away with everything from R2, S3, Ceph, Minio, Backblaze, etc. This is very much a case you should benchmark/analyze as a proper engineer.
I've managed many TB ES clusters. This would have reduced my cost and time to manage it so I could focus on other features that would have benefitted my users.
So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services instead? What an awful idea! I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
You can easily emulate object storage with e.g. s3 compatible APIs. And probably it will just support whatever file system, including nfs (currently not supported). So, that should not be a show stopper. And technically this could actually reduce your cost by a lot because you will be able to use some central network storage instead of having to have a lot of high end SSDs. You might still want to use those for cachi…
This seems right in theory, but when you actually look at the details it stops being true.
Specifically, Lucene today is best served through memory mapping the files and using crazy amounts of RAM.
My guess, Elastic just no longer cares about latency sensitive users. Even going from ES 5 to ES 7 there was a performance reduction for queries using any multi-word synonyms. It’s likely this trend is set to continue with remote storage.
I still think it’s the correct trade off for ES. Most of their clients use them for latency-insensitive analytics/log type stuff.
Earlier quoted context omitted.
You can easily emulate object storage with e.g. s3 compatible APIs. And probably it will just support whatever file system, including nfs (currently not supported). So, that should not be a show stopper. And technically this could actually reduce your cost by a lot because you will be able to use some central network storage instead of having to have a lot of high end SSDs. You might still want to use those for cachi…
How is Elasticsearch "closed source"? The code is literally right here: https://github.com/elastic/elasticsearch
And for those who cannot wait, there is quickwit :-) https://quickwit.io
So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services instead? What an awful idea! I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
Fingers crossed they support any S3-compatible local object storage system (e.g. Ceph) then all you need to do is completely change your storage system and probably duplicate it’s capacity during migration. If this is the future of ES I don’t see a happy path for non-cloud customers…
Do you mean they "only support"?
And for those who cannot wait, there is quickwit :-) https://quickwit.io
This is music to my ears! Analytical databases are all so damn complicated to manage with a dozen of different stateful node types
Many of the people commenting don't realize that for the majority of ES users, it is a pain in the butt to manage and scale their cluster storage in lock step with their compute. Most companies use ES for relatively small scale search with people who are not experts in managing an ES cluster, or really infra at all. A cloud offering that decouples storage from compute makes this a lot easier and becomes more of a no…
Many of the people commenting don't realize that for the majority of ES users, it is a pain in the butt to manage and scale their cluster storage in lock step with their compute. Most companies use ES for relatively small scale search with people who are not experts in managing an ES cluster, or really infra at all. A cloud offering that decouples storage from compute makes this a lot easier and becomes more of a no…
I'm not familiar with this space, but I would have thoguht existing managed cloud offerings already decoupled storage from compute as far as the customer is concerned. But not true, or not as much true as it could be with new architecture?
In the case of both services, they make the processes easier but you still have time and expense considerations to them, if only just waiting time and not active work on your part. But it also means you have a lot more monitoring concerns you have to implement and then account for.
Plus your choice in storage IOPS greatly effects query time so you also have to consider that and high IOPS storage is expensive.
It's a lot of things for someone who just wants a workable search for their product to have to know.
The new architecture they're proposing seems to allow for much more seamless scaling as your data grows without a bunch of manual intervention, monitoring and infra knowledge to make sure things don't fall over on you.