Reducing logging cost by two orders of magnitude using CLP
1–10 of 107 posts
Re: Reducing logging cost by two orders of magnitude using CLP
#2This just in, Uber rediscovers what all us database people already knew, structured data is usually way easier to compress and store and index and query than unstructured blobs of text, which is why we kept telling you to stop storing json in your databases.
Re: Reducing logging cost by two orders of magnitude using CLP
#3Always bugged me that highly repetitive logs take up so much space!
I'm curious, are there any managed services / simple to use setups to take advantage of something like this for massive log storage and search? (Most hosted log aggregators I've looked at charge by the raw text GB processed)
Re: Reducing logging cost by two orders of magnitude using CLP
#4This just in, Uber rediscovers what all us database people already knew, structured data is usually way easier to compress and store and index and query than unstructured blobs of text, which is why we kept telling you to stop storing json in your databases.
Word!
Storing JSON is so often the most direct and explicit way of accruing technical debt:
"We don't really know what structure the data we'll get should have, just specify that it's going to be JSON"...
Re: Reducing logging cost by two orders of magnitude using CLP
#5This just in, Uber rediscovers what all us database people already knew, structured data is usually way easier to compress and store and index and query than unstructured blobs of text, which is why we kept telling you to stop storing json in your databases.
Yeah it’s silly, even when Spark is writing unstructured logs, that doesn’t mean that you can‘t parse them after-the-fact and store them in a structured way. Even if it doesn’t work for 100% of the cases, it’s very easy to achieve for 99% of them, in which case you’ll still keep a “raw_message” column which you can query as text.
Next up: Uber discovers column oriented databases are more efficient for data warehouses.
Re: Reducing logging cost by two orders of magnitude using CLP
#6This just in, Uber rediscovers what all us database people already knew, structured data is usually way easier to compress and store and index and query than unstructured blobs of text, which is why we kept telling you to stop storing json in your databases.
Word! Storing JSON is so often the most direct and explicit way of accruing technical debt: "We don't really know what structure the data we'll get should have, just specify that it's going to be JSON"...
But if you’re not storing data as JSON, can you really say you’re agile? /s
Re: Reducing logging cost by two orders of magnitude using CLP
#7Earlier quoted context omitted.
Word! Storing JSON is so often the most direct and explicit way of accruing technical debt: "We don't really know what structure the data we'll get should have, just specify that it's going to be JSON"...
But if you’re not storing data as JSON, can you really say you’re agile? /s
Look, we'll just get it in this way for now, once it's live we'll have all the time we need to change the schema in the background
Re: Reducing logging cost by two orders of magnitude using CLP
#8This just in, Uber rediscovers what all us database people already knew, structured data is usually way easier to compress and store and index and query than unstructured blobs of text, which is why we kept telling you to stop storing json in your databases.
There's nothing wrong with storing json in your database if the tradeoffs are clear and it's used in a sensible way.
Having structured data and an additional json/jsonb column where it makes sense can be very powerful. There's a reason every new release of Postgres improves on the performance and features available for the json data type. (https://www.postgresql.org/docs/9.5/functions-json.html)
Re: Reducing logging cost by two orders of magnitude using CLP
#9Man, I was expecting constraint linear programming.
Re: Reducing logging cost by two orders of magnitude using CLP
#10Earlier quoted context omitted.
But if you’re not storing data as JSON, can you really say you’re agile? /s
Look, we'll just get it in this way for now, once it's live we'll have all the time we need to change the schema in the background
We don’t have a use case yet, but let’s just collect all the data and figure out what to do with it later!
It’s funny how these cliches repeat everywhere in the industry, and it’s almost impossible for people to figure this out beforehand. It seems like everyone needs to deal with data lakes (at scale) at least once in their life before they truly appreciate the costs of the flexibility they offer.