61–70 of 109 posts
Earlier quoted context omitted.
I've managed a 100+ node cluster for years without seeing any corruption. Where are you getting this from?
I'm actually struggling to imagine exactly what warrants a 100+ node cluster of ES?
No, of course not. But the question is, do you need a database? A database is a big proposition: transactions, indexes, query processing, replication, distribution, etc. A fair number of use cases are just "Take this data and give it back to me when I ask for it". ES (or any other not-a-database) might not be a full-bore DBMS. But it might be what you need.
Rule of thumb: Whenever you think you don't need relational database features, you will later discover why you do. The one thing relational databases don't have, that you might need, is scaling. Maintaining data consistency implies a certain level of non-concurrency. Conversely, maintaining perfect concurrency implies a certain level of data inconsistency.
(Obviously I'm referring to a famous YouTube video on the subject)
No, of course not. But the question is, do you need a database? A database is a big proposition: transactions, indexes, query processing, replication, distribution, etc. A fair number of use cases are just "Take this data and give it back to me when I ask for it". ES (or any other not-a-database) might not be a full-bore DBMS. But it might be what you need.
Rule of thumb: Whenever you think you don't need relational database features, you will later discover why you do. The one thing relational databases don't have, that you might need, is scaling. Maintaining data consistency implies a certain level of non-concurrency. Conversely, maintaining perfect concurrency implies a certain level of data inconsistency.
You could maybe consider Rel if you have a particular type of workload, but, realistically, just use a tablational database. It will be a lot easier and is arguably better.
It has an index? It has data that can be queried with indexes? it is a database. PERIOD. Let's not turn the word database into a buzzword. It should obviously NOT be a "main" database but part of an ETL pipeline for search purposes for instance.
It is much too late for that, but you're right that we'd be wise to put effort into undoing that. This is exactly how you end up with people using Elasticsearch as a primary datastore. When someone hears that they need a database, a database is what you are going to see them pick.
If we regularly used the proper terminology with appropriate specificity then those without the deep technical knowledge required to understand all the different kinds of databases and the tradeoffs that come them are able to narrow their search to the solutions that fit within the specification.
Ofcourse it is not meant as a primary database. What baffles me is that people use it as log storage. As an application scales, storage and querying logs become the bottleneck if elasticsearch is used. I was dealing with a system that could afford only 1 week of log retention!
Metrics are much more efficient and are the tool of choice for longer term storage and debugging.