Stateless – Evolving the architecture of Elasticsearch to simplify deployment
1–10 of 28 posts
Re: Stateless – Evolving the architecture of Elasticsearch to simplify deployment
#2So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services instead?
What an awful idea!
I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
Re: Stateless – Evolving the architecture of Elasticsearch to simplify deployment
#3So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services instead? What an awful idea! I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
Fingers crossed they support any S3-compatible local object storage system (e.g. Ceph) then all you need to do is completely change your storage system and probably duplicate it’s capacity during migration.
If this is the future of ES I don’t see a happy path for non-cloud customers…
Re: Stateless – Evolving the architecture of Elasticsearch to simplify deployment
#4So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services instead? What an awful idea! I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
Fingers crossed they support any S3-compatible local object storage system (e.g. Ceph) then all you need to do is completely change your storage system and probably duplicate it’s capacity during migration. If this is the future of ES I don’t see a happy path for non-cloud customers…
minio
Re: Stateless – Evolving the architecture of Elasticsearch to simplify deployment
#5So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services instead? What an awful idea! I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
You can easily emulate object storage with e.g. s3 compatible APIs. And probably it will just support whatever file system, including nfs (currently not supported). So, that should not be a show stopper. And technically this could actually reduce your cost by a lot because you will be able to use some central network storage instead of having to have a lot of high end SSDs. You might still want to use those for caching of course. But getting rid of some of the data replication probably might actually help lower your cost.
As Elasticsearch is closed source, you can also choose to switch to Opensearch which will of course not get any of these changes. Though I would not be surprised to see this move mirrored on their side as it makes a lot of sense to do this. But it would end up being an independent implementation of the same concept.
As a long time Elasticsearch user, this stateless architecture makes a lot of sense to me. Especially for very large clusters. Basically, it vastly simplifies scaling and cluster operations. You can literally auto scale nodes both on the indexing and querying tiers. That's a big deal. You want it faster? Add more nodes. Likewise upgrades are a lot easier. Simply bring new nodes online and re-index to some new objects. Once it is done, take down the old ones. Also it simplifies testing. You can simply bring up a few test nodes and query your production data without having to worry about affecting production loads. Everything gets easier.
It will be interesting to see how they will bring this to market. This reads like they are starting the work on this, not like they are ready to deploy this. I guess this would be part of a future major release and they just had their previous one fairly recently.
Re: Stateless – Evolving the architecture of Elasticsearch to simplify deployment
#6So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services instead? What an awful idea! I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
I believe it would be cheaper to run stateless (given the reduced overhead of devops)? It is an open secret that the most expensive commodity in a datacenter (servers) is also the most under utilised [0]. Hence, folks like me prefer to pay per-query cost.
Elastic had to do this in face of competition from the likes of quickwit.io and snowflake search. AWS might have something similar up their sleeve, because I don't believe Athena/Trino can do super fast searches, yet.
Re: Stateless – Evolving the architecture of Elasticsearch to simplify deployment
#7Earlier quoted context omitted.
Fingers crossed they support any S3-compatible local object storage system (e.g. Ceph) then all you need to do is completely change your storage system and probably duplicate it’s capacity during migration. If this is the future of ES I don’t see a happy path for non-cloud customers…
minio
...which means another layer that eats up performance?
Re: Stateless – Evolving the architecture of Elasticsearch to simplify deployment
#8Quickwit is another project aiming at this sort of stateless search: https://quickwit.io/ . I've been keeping an eye on it for a possible project that would involve wanting to offer search over a lot of data for an application that wouldn't have very many users, such that keeping a big ES box around all the time would be needlessly expensive; I had been looking for a search situation where my only always-on costs were for storage, and I could pay for compute as needed (and ideally also separate paying for indexing compute, which doesn't happen often in this application, from search compute).
Interesting that ES might also end up with this kind of an offering.
Re: Stateless – Evolving the architecture of Elasticsearch to simplify deployment
#9And for those who cannot wait, there is quickwit :-) https://quickwit.io
Re: Stateless – Evolving the architecture of Elasticsearch to simplify deployment
#10Quickwit is another project aiming at this sort of stateless search: https://quickwit.io/ . I've been keeping an eye on it for a possible project that would involve wanting to offer search over a lot of data for an application that wouldn't have very many users, such that keeping a big ES box around all the time would be needlessly expensive; I had been looking for a search situation where my only always-on costs wer…
Managing storage and compute together is one of the biggest issues with ElasticSearch it makes managing a cluster a fairly complex task, I think it's natural to see them move in this direction.
I imagine that, especially after the OpenSearch debacle, they're keen to make sure they don't lose even more marketshare so they're paying particular attention to the way the wind blows and wanted to do this PoC so they can at least compete with people already in the stateless space like Quickwit.