This is pretty great. Some folks in the thread are suggesting using a binary format, and that's certainly a good idea for some. My business sells software that collects moderately sized (10-100TB/day) , and this data is collected as PROTOBUF. The format is great, and coordinating the backend and client side stuff with protobuf is bliss. However, we store the data in various relational and document databases. Neither…
I've written translation layers for such systems and it's not too bad. See this project from $job - 1: https://github.com/CaperAi/pronto It allowed us to have a single model for storage in the DB, for sending between services, and syncing to edge devices.
But if you're talking about data lakes, like splunk, or various other similar systems - there's no standard way for writing a parser like this across all of them and you end up implementing the same thi ng a ton of different times.