Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The four pillars of data observability: metrics, metadata, lineage, and logs

metaplane.dev

Post reply on HN
Print view View original

21–30 of 30 posts

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#21

Post by vbezhenar »

xcambar wrote: post #2

Off topic: there seems to be a growing trend at HN of posts reaching the homepage with a reasonable number of upvotes yet without comments. I don't know how to proceed with these posts (and this one), yet the temptation of mentally flagging these as friendly upvotes or point hoarders is strong, and I must admit that such posts receive less attention and more suspicion from me. YMMV.

My guess is that people are interested with this topic and want to read discussions from other people, but don't really have anything to add right now. Sometimes I upvote topics using those thoughts.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#22

Post by Archelaos »

kzh_ wrote: post #12
metadat wrote: post #7

Earlier quoted context omitted.

Edit: I went and read TFA, and must say there were some red flags. CS people who add "PhD" beside their name are not only pretentious, but are trying to throw their academic weight around instead of letting their ideas and presentation stand on its own. Filled with more marketing fluff than useful information. Ugh. I'm siding with you on this. I've "undowned" you and upvoted instead; Sorry xcamber! -- If you're reall…

OP here, I only try to write and share things that I find personally interesting, so if it came across as marketing fluff that was the opposite of what I was aiming for :/. But I do appreciate you reading the whole thing. FWIW I also thought including PhD might be pretentious.

I would say inlcuding CEO is far more pretentious. A PhD at least means something more substantial, because it requires an external certification.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#23

Post by pm90 »

pbowyer wrote: post #15

We're good at logging text, but how do you handle logging assets (images, audio - anything non-textual but generated) and associating them with your logs? For example an image processing pipeline. You don't always want to log (it'd never scale) but as part of a trace you might want to keep the intermediate files so you can track down where the problem is. You've already got text logging for each step, recording metri…

Maybe wild idea: generate a unique text identifier for the image + an s3 url, log that identifier rather than the image. I guess its logging metadata rather than the actual data.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#24

Post by SnowHill9902 »

pbowyer wrote: post #15

We're good at logging text, but how do you handle logging assets (images, audio - anything non-textual but generated) and associating them with your logs? For example an image processing pipeline. You don't always want to log (it'd never scale) but as part of a trace you might want to keep the intermediate files so you can track down where the problem is. You've already got text logging for each step, recording metri…

- unused memory is wasted, you may be able to store the raw image. - if your process is deterministic store a hash. - store a low resolution image.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#25

Post by awinter-py »

nobody needs this new saas stuff. I prefer the traditional pillars of: emails from users, live chat feature in product where users shame you publicly if something is wrong, twitter search 'is X down', and 4) having laptop open in passenger seat on commute, tethered to blackberry, and periodically hitting F5 on the page which hits the most APIs

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#26

Post by swordsmith8 »

Seems like the the key pillars are: freshness, volume, schema, distribution, and lineage.

Makes more sense this way, I think...

If you think about metrics, traces, and logs (software observability pillars) as three distinct things, it's hard to view metadata separate from metadata, lineage, or logs. Metadata is kind of the glue that holds everything together.

This article has more relevant sources, IMO, even if it is from a SaaS vendor.

https://www.montecarlodata.com/blog-what-is-data-observabili...

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#27

Post by Dwolb »

Archelaos wrote: post #22
kzh_ wrote: post #12

Earlier quoted context omitted.

OP here, I only try to write and share things that I find personally interesting, so if it came across as marketing fluff that was the opposite of what I was aiming for :/. But I do appreciate you reading the whole thing. FWIW I also thought including PhD might be pretentious.

I would say inlcuding CEO is far more pretentious. A PhD at least means something more substantial, because it requires an external certification.

Adding “PhD” and other credentialed titles is standard SEO practice these days.

The thought is Google sees the article as from a “credible source” and ranks you higher.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#28

Post by Dowwie »

There's not enough information shared about what metrics people monitor and why. Accurately measuring complex system performance and utilization over time is hard work! Observability platforms such as DataDog and New Relic are very expensive. Engineering teams repeat mistakes, recording too much information, getting the bill, and rushing high-priority pruning exercises through engineering to stop the financial bleeding. I encourage everyone to share their monitoring setups!

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#29

Post by MonkeyMalarky »

xcambar wrote: post #2

Off topic: there seems to be a growing trend at HN of posts reaching the homepage with a reasonable number of upvotes yet without comments. I don't know how to proceed with these posts (and this one), yet the temptation of mentally flagging these as friendly upvotes or point hoarders is strong, and I must admit that such posts receive less attention and more suspicion from me. YMMV.

I often upvote articles because I have a passing interest in the topic and want to see the content/author get intellectually flogged by HN users.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#30

Post by AishwaryaVenkat »

Dowwie wrote: post #28

There's not enough information shared about what metrics people monitor and why. Accurately measuring complex system performance and utilization over time is hard work! Observability platforms such as DataDog and New Relic are very expensive. Engineering teams repeat mistakes, recording too much information, getting the bill, and rushing high-priority pruning exercises through engineering to stop the financial bleedi…

We use Atatus, that can help with application performance troubleshooting and optimization, helping us to identify slow database queries and optimize query performance.
Post reply on HN