This is nothing compared to what dragnet surveillance has to deal with.
And that's all on MSSQL or RDS, right?
CERN swaps out databases to feed its petabyte-a-day habit
11–20 of 57 posts
Re: CERN swaps out databases to feed its petabyte-a-day habit
#12One of my fondest memories as a summer student at CERN in 1993 (in the Electronics and Computing for Physics department) was the visit to the basement beneath the main computing facility, where a colossal tape robot was in operation. Even at that time, CERN was grappling with exceedingly vast amounts of data.
Re: CERN swaps out databases to feed its petabyte-a-day habit
#13Missing from the title: leaving InfluxDB and Prometheus for VictoriaMetrics.
Re: CERN swaps out databases to feed its petabyte-a-day habit
#14Weird that the title talks about the petabyte a day, while the article is actually about their monitoring tooling, not the thing ingesting the data from experiments, iiuc.
Re: CERN swaps out databases to feed its petabyte-a-day habit
#15Missing from the title: leaving InfluxDB and Prometheus for VictoriaMetrics.
This is puzzling. I'm not sure how VictoriaMetrics solved the cardinality problem? When running an aggregate query that sums up some counters for a single metric over the dimension of instances in a time window of larger than a few hours, VictoriaMetrics would barf with error for the querying having too many time series (or data points? I forgot the exact wording). This clearly shows that 1/ Victoria Metrics does not treat a time series with multiple dimensions as a single time series; 2/ VictoriaMetrics does not perform hierarchical aggregation.
That is, VictoriaMetrics has not really built a true time series DB that handles reasonable cardinalities.
Re: CERN swaps out databases to feed its petabyte-a-day habit
#16OPENSOURCE, APACHE2 LICENSE
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast...
Re: CERN swaps out databases to feed its petabyte-a-day habit
#17OPENSOURCE, APACHE2 LICENSE https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast...
[deleted]
Re: CERN swaps out databases to feed its petabyte-a-day habit
#18I also dropped InfluxDB at work due to its terrible performance. VictoriaMetrics is great I was using Promscale (TimescaleDB) but they EOL'd Promscale which forced us to Victoria. But either way both of these are much faster than Influx Don't get fooled into the latest InfluxDB rewrite. I think the latest is cloud hosted only too? So stupid
I saw the writing on the wall with InfluxDB v2 (doubling down on closed platform / SaaS) and advocated exploring VictoriaMetrics, even though we had some Influx v1 running. No regrets.
I also prefer the golang-esque simplicity of the Prometheus ecosystem. Monitoring is the last place I want unnecessary abstraction layers and complicated configuration files.
Re: CERN swaps out databases to feed its petabyte-a-day habit
#19I really like VictoriaMetrics's architecture vmagent takes care of all the pesky edge things like emulating prometheus config parsing and various scraping bits. It also does buffering in case you lose network connection for a while, and accept vast spread of different protocols vminsert/vmselect scale separately from eachother and your queries don't bother your ingest all that much. vmstorage does just that, storage.…
What version of Influx were you running? I'm interested if v3 will be more competitive than v2.
Re: CERN swaps out databases to feed its petabyte-a-day habit
#20I can do this on my laptop
/tumbleweed...