Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using machine learning to choose compression algorithms

vks.ai

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Using machine learning to choose compression algorithms

#51

Post by kootenpv »

md_ wrote: post #18
kootenpv wrote: post #14

Earlier quoted context omitted.

Spot on (I briefly touch on this in the article)! This is why I try to work with cheap-to-compute features. I used to calculate how unique all values were, but ended up taking a sample instead to speed that part up for large data!

How does that compare to just doing compression on a sample of data? :)

Perfect interview questions to get to the juice details haha!

The problem is that choice of compression is very much dependent on the sample size, so this is why just choosing the algorithm based on running benchmarks on the sample will be off.

However, computing certain statistics on the sample and then doing machine learning makes a lot of sense!

Obviously not for simple/cheap to compute features such as number of rows/columns which actually do not take longer to compute as the data gets larger.

But I actually do this for predicting the uniqueness of values! You basically want to get an idea of how many unique values you have per column. But.. if you take a sample this will give you completely wrong numbers.

You can see my approach here: https://github.com/kootenpv/shrynk/blob/6a8675061d82aa65fc3b... Basically the formula calculates the uniqueness on a sample (e.g. 10000 rows), but then extrapolates the result to your actual data size. E.g. it finds 100 unique values in a sample of 10000, but in reality that means you maybe have 500 unique values.

Re: Using machine learning to choose compression algorithms

#52

Post by eru »

proverbialbunny wrote: post #17
pedrocx486 wrote: post #3

Is there a way to do the reverse? There's a quite "legendary" Game Boy Advance game out there (Klonoa - Densetsu no Star Medal) that never got a translation to English because it has some sort of in-house created compression by Namco applied to the game that was made so it could fit into a GBA cartridge. AFAIK no one was ever able to crack it open and release the code to de/compress it. A while ago I had a "bounty" o…

Anyone who does malware analysis professionally has to deal with packed data in new and funky ways, as malware binaries tend to be packed (ie compressed) in unique ways to get past antivirus software. If the bounty was high enough there are people out there who do this sort of thing professionally and would probably jump on the opportunity.

For something like an obscure GBA game, you can probably write up some blogposts to add reputation to the minuscule monetary value of the bounty.

Re: Using machine learning to choose compression algorithms

#53

Post by numbol »

Funny, it deeply resembles the last episodes of "Silicon Valley"

More seriously, the generalized compression algorithm can be one of the keys or even one of the definitions of generalized artificial intelligence

And Kolmogorov’s complexity was present by a subtle reference in the very last episode ...

Re: Using machine learning to choose compression algorithms

#54

Post by mebr »

The wording could be improved to make it easier understand what is being offered more quickly.

This is using machine learning to predict on the given heterogeneous tabular data which compression algorithm will yield a higher compression rate.

There are multiple other ways to do this approximation.

Re: Using machine learning to choose compression algorithms

#55

Post by md_ »

ma2rten wrote: post #29
md_ wrote: post #18

Earlier quoted context omitted.

How does that compare to just doing compression on a sample of data? :)

Running the compression algorithms is O(kn) where k is the number of compression algorithms. Taking the machine learning approach is O(n).

As the author noted after you posted this, it’s not a given that the ML algorithm is O(n). It may be constant time (by looking only at column headers and a sample of data, say).

That said, I was really more interested in practical runtimes. Like, in practice the ML may have a high startup cost (e.g., due to cost of loading a model), whereas for most sized datasets linear complexity may be fine...

Re: Using machine learning to choose compression algorithms

#56

Post by proverbialbunny »

JamesBarney wrote: post #44
proverbialbunny wrote: post #19

Earlier quoted context omitted.

Does anyone know what is the leading time-series database people use today? Like, eg, bar / tick data. (I use PostgreSQL, due to my ignorance.)

I've been happy with influxdb, but I've also noticed that a lot of people switch to a time series database way before they need to. You can go pretty far with postgres/oracle/sql server, and it has the advantage you don't need to manage different databases.

Thanks for the response. I've avoided influxdb because it drops data. Bar and tick data needs to be ACID or at least very close to ACID.

InfluxDB is great for server usage stats though.

Re: Using machine learning to choose compression algorithms

#57

Post by JamesBarney »

proverbialbunny wrote: post #56
JamesBarney wrote: post #44

Earlier quoted context omitted.

I've been happy with influxdb, but I've also noticed that a lot of people switch to a time series database way before they need to. You can go pretty far with postgres/oracle/sql server, and it has the advantage you don't need to manage different databases.

Thanks for the response. I've avoided influxdb because it drops data. Bar and tick data needs to be ACID or at least very close to ACID. InfluxDB is great for server usage stats though.

I don't know much about bar or tick data. Mind going into more detail about dropping data?

We didn't really have a need for transactions because we were just recording a bunch of sensor data from oil and gas wells from the around the world and then running computations and then displaying and alerting on the results.

Re: Using machine learning to choose compression algorithms

#58

Post by kootenpv »

md_ wrote: post #55
ma2rten wrote: post #29

Earlier quoted context omitted.

Running the compression algorithms is O(kn) where k is the number of compression algorithms. Taking the machine learning approach is O(n).

As the author noted after you posted this, it’s not a given that the ML algorithm is O(n). It may be constant time (by looking only at column headers and a sample of data, say). That said, I was really more interested in practical runtimes. Like, in practice the ML may have a high startup cost (e.g., due to cost of loading a model), whereas for most sized datasets linear complexity may be fine...

Models are cached and not large so the setup time is very low. I'll time it when I get to a PC

Re: Using machine learning to choose compression algorithms

#59

Post by proverbialbunny »

JamesBarney wrote: post #57
proverbialbunny wrote: post #56

Earlier quoted context omitted.

Thanks for the response. I've avoided influxdb because it drops data. Bar and tick data needs to be ACID or at least very close to ACID. InfluxDB is great for server usage stats though.

I don't know much about bar or tick data. Mind going into more detail about dropping data? We didn't really have a need for transactions because we were just recording a bunch of sensor data from oil and gas wells from the around the world and then running computations and then displaying and alerting on the results.

Bar and tick data is financial, so dealing with numbers. It has to be precise, like not 0.0000001 off and dropped data isn't the end of the world, but it's pretty bad.

Re: Using machine learning to choose compression algorithms

#60

Post by pedrocx486 »

eru wrote: post #52
proverbialbunny wrote: post #17

Earlier quoted context omitted.

Anyone who does malware analysis professionally has to deal with packed data in new and funky ways, as malware binaries tend to be packed (ie compressed) in unique ways to get past antivirus software. If the bounty was high enough there are people out there who do this sort of thing professionally and would probably jump on the opportunity.

For something like an obscure GBA game, you can probably write up some blogposts to add reputation to the minuscule monetary value of the bounty.

Well, the bounty nowadays is $0. :-)

I have given up until I have the proper time along with a friend to try to crack the game insides in our own. If anyone did this for now, it'd be for the community.

Post reply on HN