Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The four pillars of data observability: metrics, metadata, lineage, and logs

metaplane.dev

Post reply on HN
Print view View original

11–20 of 30 posts

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#11

Post by kzh_ »

xcambar wrote: post #2

Off topic: there seems to be a growing trend at HN of posts reaching the homepage with a reasonable number of upvotes yet without comments. I don't know how to proceed with these posts (and this one), yet the temptation of mentally flagging these as friendly upvotes or point hoarders is strong, and I must admit that such posts receive less attention and more suspicion from me. YMMV.

OP here, I posted this a few days ago and was surprised to see it on the front page this morning. Not sure why it says I submitted 4 hours ago when I wasn’t awake, maybe the second-chance pool (https://news.ycombinator.com/item?id=26998308)?

But I’m also generally skeptical of high upvote/comment ratios, because as a long-time HNer too I also want to read things that are genuinely interesting. In this case, I can promise you neither I nor anyone on the team is soliciting upvotes for this post.

On that note, if anyone has any comments about the content itself, happy to discuss further.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#12

Post by kzh_ »

metadat wrote: post #7
xcambar wrote: post #2

Off topic: there seems to be a growing trend at HN of posts reaching the homepage with a reasonable number of upvotes yet without comments. I don't know how to proceed with these posts (and this one), yet the temptation of mentally flagging these as friendly upvotes or point hoarders is strong, and I must admit that such posts receive less attention and more suspicion from me. YMMV.

Edit: I went and read TFA, and must say there were some red flags. CS people who add "PhD" beside their name are not only pretentious, but are trying to throw their academic weight around instead of letting their ideas and presentation stand on its own. Filled with more marketing fluff than useful information. Ugh. I'm siding with you on this. I've "undowned" you and upvoted instead; Sorry xcamber! -- If you're reall…

OP here, I only try to write and share things that I find personally interesting, so if it came across as marketing fluff that was the opposite of what I was aiming for :/. But I do appreciate you reading the whole thing. FWIW I also thought including PhD might be pretentious.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#13

Post by xcambar »

xiande04 wrote: post #9
xcambar wrote: post #4

Earlier quoted context omitted.

What makes you assume I did not read? How shocking is it that, on some article and topic, I do not have a comment I deem interesting enough to share? Besides, if you do not appreciate my specificly off-topic contribution, then so be it.

> What makes you assume I did not read? OP didn't say you didn't read it. They said that you didn't read it AND attempt to start a meaningful discussion about it, which exactly what you were complaining about, right?

If you mean meaningful with regards to the topic, I'd agree with you and knew it from the start, hence the "off-topic" warning.

My intention was to confront my experience and behavior (towards certain categories of posts of high upvotes/comment ratio) with the rest of the HN crowd, in a contextualized environment where it applies.

I'm sorry that the conversation now revolves around my own comment. Kinda ironic.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#14

Post by xcambar »

kzh_ wrote: post #11
xcambar wrote: post #2

Off topic: there seems to be a growing trend at HN of posts reaching the homepage with a reasonable number of upvotes yet without comments. I don't know how to proceed with these posts (and this one), yet the temptation of mentally flagging these as friendly upvotes or point hoarders is strong, and I must admit that such posts receive less attention and more suspicion from me. YMMV.

OP here, I posted this a few days ago and was surprised to see it on the front page this morning. Not sure why it says I submitted 4 hours ago when I wasn’t awake, maybe the second-chance pool ( https://news.ycombinator.com/item?id=26998308 )? But I’m also generally skeptical of high upvote/comment ratios, because as a long-time HNer too I also want to read things that are genuinely interesting. In this case, I can p…

Thanks for commenting constructively. As you have umderstood, my intention was never to point fingers at you or your article, but rather use it as a suitable context to confront with the HN crowd.

Thanks for having seen this from the start :)

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#15

Post by pbowyer »

We're good at logging text, but how do you handle logging assets (images, audio - anything non-textual but generated) and associating them with your logs?

For example an image processing pipeline. You don't always want to log (it'd never scale) but as part of a trace you might want to keep the intermediate files so you can track down where the problem is. You've already got text logging for each step, recording metrics like duration and which filters were involved. I have saved files and referenced them in the logfile, but no log viewers I've seen understand anything beyond text. So I then have to build my own UI or open the images in turn.

Is there a pattern to handle this?

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#16

Post by ctxc »

kzh_ wrote: post #12
metadat wrote: post #7

Earlier quoted context omitted.

Edit: I went and read TFA, and must say there were some red flags. CS people who add "PhD" beside their name are not only pretentious, but are trying to throw their academic weight around instead of letting their ideas and presentation stand on its own. Filled with more marketing fluff than useful information. Ugh. I'm siding with you on this. I've "undowned" you and upvoted instead; Sorry xcamber! -- If you're reall…

OP here, I only try to write and share things that I find personally interesting, so if it came across as marketing fluff that was the opposite of what I was aiming for :/. But I do appreciate you reading the whole thing. FWIW I also thought including PhD might be pretentious.

Hi OP! It would be nice to have some examples to go with the article. Some set of minimum data and sample "lineage" etc.

This has broadened my perception of data though, I never linked this with the good old thermodynamic principles.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#17

Post by kzh_ »

ctxc wrote: post #16
kzh_ wrote: post #12

Earlier quoted context omitted.

OP here, I only try to write and share things that I find personally interesting, so if it came across as marketing fluff that was the opposite of what I was aiming for :/. But I do appreciate you reading the whole thing. FWIW I also thought including PhD might be pretentious.

Hi OP! It would be nice to have some examples to go with the article. Some set of minimum data and sample "lineage" etc. This has broadened my perception of data though, I never linked this with the good old thermodynamic principles.

Thanks for reading! Including examples is a great point, because otherwise the article can be kind of abstract, especially because each person has a different mental model of data. I'll add some later on.

Maybe thermodynamics is a hammer that makes all things seem like nails, but the connections pop up all over the place. Entropy is another highly applicable concept to data systems.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#18

Post by nerdponx »

pbowyer wrote: post #15

We're good at logging text, but how do you handle logging assets (images, audio - anything non-textual but generated) and associating them with your logs? For example an image processing pipeline. You don't always want to log (it'd never scale) but as part of a trace you might want to keep the intermediate files so you can track down where the problem is. You've already got text logging for each step, recording metri…

It might be interesting to have something like "statistical logging", which saves the intermediate image files 1% of the time and discards them after 30 days.

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#19

Post by [deleted] »

pbowyer wrote: post #15

We're good at logging text, but how do you handle logging assets (images, audio - anything non-textual but generated) and associating them with your logs? For example an image processing pipeline. You don't always want to log (it'd never scale) but as part of a trace you might want to keep the intermediate files so you can track down where the problem is. You've already got text logging for each step, recording metri…

[deleted]

Re: The four pillars of data observability: metrics, metadata, lineage, and logs

#20

Post by kzh_ »

pbowyer wrote: post #15

We're good at logging text, but how do you handle logging assets (images, audio - anything non-textual but generated) and associating them with your logs? For example an image processing pipeline. You don't always want to log (it'd never scale) but as part of a trace you might want to keep the intermediate files so you can track down where the problem is. You've already got text logging for each step, recording metri…

+1 on existing log viewers being particularly well suited for text over non-textual assets. My experience here is limited but I believe Grafana has a dynamic image plugin if you store a link to an asset in blob storage or Base64 encode it.

I've also heard of people storing those links in a database like Snowflake then creating displays on top using Tableau or Looker, to avoid having to build a web app from scratch.

Post reply on HN