Earlier quoted context omitted.
Nah it's all about Microsoft Access '97 if you actually care about your data
I said a real database (Excel)
stream to a csv file!
CERN swaps out databases to feed its petabyte-a-day habit
41–50 of 57 posts
Re: CERN swaps out databases to feed its petabyte-a-day habit
#42Earlier quoted context omitted.
This is puzzling. I'm not sure how VictoriaMetrics solved the cardinality problem? When running an aggregate query that sums up some counters for a single metric over the dimension of instances in a time window of larger than a few hours, VictoriaMetrics would barf with error for the querying having too many time series (or data points? I forgot the exact wording). This clearly shows that 1/ Victoria Metrics does not…
https://github.com/VictoriaMetrics/VictoriaMetrics#cardinali... If I understand this correctly, it deals with high cardinality by dropping data. The operators need to monitor for this and adjust their data to lower the cardinality.
From your link:
> By default VictoriaMetrics doesn't limit the number of stored time series.
They have put out some benchmarks showing VictoriaMetrics ingesting 40M time series: https://valyala.medium.com/high-cardinality-tsdb-benchmarks-...
Re: CERN swaps out databases to feed its petabyte-a-day habit
#43We've been using and building with VictoriaMetrics for a while at Batteries Included. I have probably created and torn down 100+ clusters now. It's a remarkably easy to use piece of software for something with the capabilities.
Re: CERN swaps out databases to feed its petabyte-a-day habit
#44Earlier quoted context omitted.
This is puzzling. I'm not sure how VictoriaMetrics solved the cardinality problem? When running an aggregate query that sums up some counters for a single metric over the dimension of instances in a time window of larger than a few hours, VictoriaMetrics would barf with error for the querying having too many time series (or data points? I forgot the exact wording). This clearly shows that 1/ Victoria Metrics does not…
https://github.com/VictoriaMetrics/VictoriaMetrics#cardinali... If I understand this correctly, it deals with high cardinality by dropping data. The operators need to monitor for this and adjust their data to lower the cardinality.
This is the only longer term scalable solution. High cardinality for TSDB's have to be dealt with by dropping. Or you run out of storage, write rate, memory, or network.
It's possible to smooth this loss out (by assuming a normal distribution of lost data) if it's noticed and limited. Though I do not think there's any commercial TSDB that does that automatically.
Re: CERN swaps out databases to feed its petabyte-a-day habit
#45Earlier quoted context omitted.
This is puzzling. I'm not sure how VictoriaMetrics solved the cardinality problem? When running an aggregate query that sums up some counters for a single metric over the dimension of instances in a time window of larger than a few hours, VictoriaMetrics would barf with error for the querying having too many time series (or data points? I forgot the exact wording). This clearly shows that 1/ Victoria Metrics does not…
There are a number of circuit breakers in VictoriaMetrics that limit the number of time series/datapoints in queries to limit CPU/RAM usage. These can be tweaked with the -search.max* command line flags. I think the default query limits for data points per series are 30M and number of series is 300K. What do you consider reasonable cardinalities or a true TSDB?
In my particular example, the cardinality of the instances (i.e., the number of of unit instance count in a query's time range) should not even matter. I was summing a counter over all the instances, and VictoriaMetrics should just add up the counters for every time unit while scanning all the data points -- this is implemented by pretty much all the OLAP engines. Or put it another way, logically I was query over a single time series. It's just that each value in the time series was associated with multiple values. Given such logical model, a good time series database should not not even bother me with any concern of cardinality.
Re: CERN swaps out databases to feed its petabyte-a-day habit
#46I also dropped InfluxDB at work due to its terrible performance. VictoriaMetrics is great I was using Promscale (TimescaleDB) but they EOL'd Promscale which forced us to Victoria. But either way both of these are much faster than Influx Don't get fooled into the latest InfluxDB rewrite. I think the latest is cloud hosted only too? So stupid
InfluxData is killing InfluxDB with their changes. Their v1->v2->v3 changes are beyong insane. They revealed that flux is deprecated in v3, their main selling point for v2. In database domain you would like stability not breaking change in every 2-3 years. See https://news.ycombinator.com/item?id=37206194
[dead]
Re: CERN swaps out databases to feed its petabyte-a-day habit
#47I also dropped InfluxDB at work due to its terrible performance. VictoriaMetrics is great I was using Promscale (TimescaleDB) but they EOL'd Promscale which forced us to Victoria. But either way both of these are much faster than Influx Don't get fooled into the latest InfluxDB rewrite. I think the latest is cloud hosted only too? So stupid
Honestly the database isn't half as useful as the tool they wrote to grab the metrics. At least I think telegraf was written by the same people? It seems to have the exact opposite design philosophy.
Yup, telegraf is what keeps me with InfluxDB, even if I couldn't care less about the database or the pain in the ass it is to operate.
Re: CERN swaps out databases to feed its petabyte-a-day habit
#48Earlier quoted context omitted.
What version of Influx were you running? I'm interested if v3 will be more competitive than v2.
1.8, migration path to 2.0 was a no-no. Don't remember exact reasons back then but we decided to have wait-and-see approach and see how alternatives grow up as our data generally grows in predictable rate Also frankly Prometheus support is a massive positive. For better or worse industry standarized on apps using Prometheus as ingest for metrics, and also most of the materials related to that will of course give exam…
Flux is noisy yes. But promql isn’t exactly the pinnacle of query languages either. Simple stuff is simple, but once you start having more complex queries like sum by/without, it makes absolutely no sense where the parenthesis go and you can place some of the operators on either side of the inner expression.
Re: CERN swaps out databases to feed its petabyte-a-day habit
#49I really like VictoriaMetrics's architecture vmagent takes care of all the pesky edge things like emulating prometheus config parsing and various scraping bits. It also does buffering in case you lose network connection for a while, and accept vast spread of different protocols vminsert/vmselect scale separately from eachother and your queries don't bother your ingest all that much. vmstorage does just that, storage.…
Can you write alerting rules in Prometheus syntax also? And use with Prometheus alert manager?
Re: CERN swaps out databases to feed its petabyte-a-day habit
#50One of my fondest memories as a summer student at CERN in 1993 (in the Electronics and Computing for Physics department) was the visit to the basement beneath the main computing facility, where a colossal tape robot was in operation. Even at that time, CERN was grappling with exceedingly vast amounts of data.
I was a summer student this summer, working right next to that exact building. Unfortunately we did not get to see the tape robot, we did get visits to the experiments though.
I can say from what I've seen that the amount of data they have to deal with is still the #1 problem. Before the multi layered filters they generate a petabyte of data per second.