21–28 of 28 posts
And for those who cannot wait, there is quickwit :-) https://quickwit.io
I’m not well versed with elastic search and the like but have a project in mind… quickwit mentions logs in the first header. Is it definitely for generic searching or is it for searching logs?
Earlier quoted context omitted.
I'm not familiar with this space, but I would have thoguht existing managed cloud offerings already decoupled storage from compute as far as the customer is concerned. But not true, or not as much true as it could be with new architecture?
Not as much true as it could be, with the offerings I'm aware of (AWS OpenSearch, Elasti, Co's own cloud), and having used OpenSearch more extensively, you're still basically fully responsible for managing the cluster topology and the "data" node type is what handles storage and the important compute for search. If you need more storage you'll have to bump the EBS volume sizes on a new cluster and then do replication…
This, absolutely. I'll add that I once tried to look at the Elasti.co offer for an ELK solution that could handle the logs we are currently managing internally (on EC2 managed by us with some automation), and you basically had to specify all the details of the topology to get a quote. It was basically a layer on top of AWS Cost Calculator with their margin baked in.
No way you can tell them "I send on average X bytes/day, I want a quick answer for Y days and a slower answer for Z days and a global retention of XY days" and get a "you will pay aprox NNN dollars/month in the HA config, MMM in the non-HA config"
And for those who cannot wait, there is quickwit :-) https://quickwit.io
As search engine that operates at any kind of scale needs to skip through very large files to evaluate a query. You need very low-latency, high-bandwidth access to disk. A search engine instance that accesses files on a local SSD is an order of magnitude faster than one that puts files on EBS.
They make some mention of local caching, but the devil is in the details here. Does all data get copied to local cache? What is the performance here?
How is it that the S3 API is remotely fast enough to make this work? As search engine that operates at any kind of scale needs to skip through very large files to evaluate a query. You need very low-latency, high-bandwidth access to disk. A search engine instance that accesses files on a local SSD is an order of magnitude faster than one that puts files on EBS. They make some mention of local caching, but the devil i…
I know Presto isn't focused on search, but Athena (AWS branded Presto) can do some really fast queries over S3, the issue is coldstart time on the compute, for a similar solution focused on search maybe you keep the compute always warm and work from there.
How is it that the S3 API is remotely fast enough to make this work? As search engine that operates at any kind of scale needs to skip through very large files to evaluate a query. You need very low-latency, high-bandwidth access to disk. A search engine instance that accesses files on a local SSD is an order of magnitude faster than one that puts files on EBS. They make some mention of local caching, but the devil i…
If you are doing multiple dependent loads, e.g. loading an index that tells you which part of the data to load which tells you which other related table to look into (e.g. a complex join)... that would be bad.
So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services instead? What an awful idea! I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.
So you wouldn't need to use S3 or some s3 emulator like other comments have suggested.
Give the ES cluster a disk it can write to that is guaranteed to be replicated (ie a RAID cluster) and thats it. The limits of object storage (no appending, no in place editing) tend to make those clusters cheaper so you could also run something like Ceph locally. It's probably going to do a better job of persisting data than ES also.
Earlier quoted context omitted.
How is Elasticsearch "closed source"? The code is literally right here: https://github.com/elastic/elasticsearch
Probably meant "Source available" since neither Elasticsearch code licenses are "Open source" (going by OSI approval).
Elasticsearch meets the above definition.
The permissiveness of the modification and enhancement is a different topic.