101–107 of 107 posts
Earlier quoted context omitted.
Rigid structures and schemas are nice, but having document oriented data also has its advantages.
> having document oriented data also has its advantages. As the VC said to the founder who wanted to do things the naive way: "well, I've seen quite a few people try that over the last few decades. It'll be interesting to see if you can make it work this time."
Earlier quoted context omitted.
Does anyone have a simple explanation of how it structures the log data?
If you have exactly two logging operations in your entire program: log(“Began {x} connected to {y}”) log(“Ended {x} connected to {y}”) We can label the first one logging operation 1 and the second one logging operation 2. Then, if logging operation 1 occurs we can write out: 1, {x}, {y} instead of “Began {x} connected to {y}” because we can reconstruct the message as long as we know what operation occurred, 1, and th…
Earlier quoted context omitted.
Does anyone have a simple explanation of how it structures the log data?
1) Use Zstandard with a generated dictionary kept separate from the data, but moreover: 2) Organize the log data into tables with columns, and then compress by column (so, each column has its own dictionary). This lets the compression algorithm perform optimally, since now all the similar data is right next to itself. (This reminds me of the Burrows-Wheeler transform, except much more straightforward, thanks to how s…
Earlier quoted context omitted.
1) Use Zstandard with a generated dictionary kept separate from the data, but moreover: 2) Organize the log data into tables with columns, and then compress by column (so, each column has its own dictionary). This lets the compression algorithm perform optimally, since now all the similar data is right next to itself. (This reminds me of the Burrows-Wheeler transform, except much more straightforward, thanks to how s…
Heh, my hacked version of this was used for apache logging. I just ran Mysql on top of a ZFS volume with compression turned on and had a column for timestamp, IP, referral IP, URL, action, and return code. I was amazed at how fast it was, how easy/fast it was to query, disk storage efficiency, and was overall quite impressed at how it was nearly as useful as a standard web traffic analysis tool that took significant…
Also, I assume you used a smallish blocksize in ZFS because of the frequent small writes?
Earlier quoted context omitted.
Heh, my hacked version of this was used for apache logging. I just ran Mysql on top of a ZFS volume with compression turned on and had a column for timestamp, IP, referral IP, URL, action, and return code. I was amazed at how fast it was, how easy/fast it was to query, disk storage efficiency, and was overall quite impressed at how it was nearly as useful as a standard web traffic analysis tool that took significant…
I wonder what would happen if you stored the columns in separate tables (perhaps pairs of columns?) and queried them with a join off a shared ID (perhaps a view or materialized view?) in order to really take advantage of compression’s ability to compress highly self-similar data located together, highly. Also, I assume you used a smallish blocksize in ZFS because of the frequent small writes?
Timestamps, especially north of 1000 hits/sec have many bits in common. URL, Referrer, and IP address where all just indexes. That worked really well because it was storage efficient, and made various queries like "who hit this URL", "who is our top referrer" and the like very efficient. Things that used to require ingesting a months worth of logs and spitting out a report would often be answered with a simple SQL query.
All in all using indexed columns was a huge win.