Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why and how GitHub is adopting OpenTelemetry

github.blog

Post reply on HN
Print view View original

51–60 of 113 posts

Re: Why and how GitHub is adopting OpenTelemetry

#51

Post by Philip-J-Fry »

StreamBright wrote: post #10

I think it would be great to show the performance impact of these SDKs because it is one of the really important aspects of monitoring (being non-intrusive).

Well, the second you do any database call or other service call you've already spent 100x longer doing that than you have recording some timings.

These clients will usually buffer the stats in memory and push them out asynchronously. Performance is definitely affected but I'm pretty sure it's negligible for most cases.

Best practice would be to reduce tracing ratio in production too. So most requests are literally just a timing.

Re: Why and how GitHub is adopting OpenTelemetry

#52

Post by rektide »

cogman10 wrote: post #50
rektide wrote: post #47

Earlier quoted context omitted.

I tend to think computing itself has been in kind of a dumb rut. Right now, OpenTelemetry is just logging & tracing, imagined as tools for ops, but over time, I fully expect we begin to see this as an event-stream in itself, something we can use for Event Sourcing, to watch & trigger new compute based off of. That's not in the cards today. But longer term, I think "knowing what computers are doing" is big business. A…

Who will be doing this spying? It certainly wouldn't be opentelemetry itself as that's just a interface you add adapters to. Are you thinking a man in the middle would spy? How would that work? This information is pushed over secure connections on the backend likely in a VPN. On the front end, it'd be transmitted over HTTPS. Shouldn't we be more fearful of information collected from DNS than encrypted data sent over…

We're forming the best, most complete, competent view of computing-that's-happened that we've ever formed.

You have a bunch of weird straw men that I don't get. I tried to de-emphasize the role of behavioral analytics & user-tracking, because I think it's just one small part of what this will be used for. But I am fairly confident we will eventually start to do more user-tracking via these systems. I've used half a dozen different user-tracking products at various companies, and they all read like ultra-low-fi versions of the ops tools. Ops tools have been evolving, at a far faster rate, far more in the public domain, and at some point, it just wont make sense to instrument your product twice.

I want to re-iterate that I see this as one of the smallest, least interesting aspects of a coming Event-Sourcing-powered-by-tracing world. There's far more profound implications for what could happen to computing in general here (a de-wiring of the request/response microservice world & a shift towards async, reactive systems). But even today, it feels to me like folks work very hard to draw a distinction between their ops tools & their behavioral analytics tool. As a developer, they often work, I often interface with them in very very similar fashion. The desire to draw the distinction has felt illogical, and felt unsupported. Especially as the ops tools advance, I think it will be harder to reconcile the idea that there are & ought be separate systems of tracking/viewing/understanding.

Re: Why and how GitHub is adopting OpenTelemetry

#53

Post by thefounder »

ehershey wrote: post #46
thefounder wrote: post #17

Off topic: At some point my wasm app was consuming excessive memory bringing the browser to a halt. The issue was the OpenTelemetry package(Go-lang) used by various Google sdks. Forking/sanitizing the sdks fixed the issue.

What do you mean by “sanitizing” the sdk’s? Did you remove the OpenTelemetry package?

>> Did you remove the OpenTelemetry package?

Exactly!

Re: Why and how GitHub is adopting OpenTelemetry

#55

Post by sethammons »

StreamBright wrote: post #10

I think it would be great to show the performance impact of these SDKs because it is one of the really important aspects of monitoring (being non-intrusive).

it is a solid call out. One of our teams had to rip out and completely re-think our integration with OpenTracing because of allocations in the client at the time. I believe that's been fixed.

Re: Why and how GitHub is adopting OpenTelemetry

#56

Post by tignaj »

thefounder wrote: post #17

Off topic: At some point my wasm app was consuming excessive memory bringing the browser to a halt. The issue was the OpenTelemetry package(Go-lang) used by various Google sdks. Forking/sanitizing the sdks fixed the issue.

Please submit a bug at https://github.com/open-telemetry/opentelemetry-go/issues We want to make the SDKs rock-solid.

Re: Why and how GitHub is adopting OpenTelemetry

#57

Post by AYBABTME »

AlexandrB wrote: post #20
rektide wrote: post #12

A lot of very confused weird comments here. This article is about a fairly large sized tech company adopting a fairly recent but increasingly mainstream & popular tool that helps them understand their operations. It'll give them a standard way to see what their computers are doing, across their various systems. OpenTelemetry is one of the key emerging cloud standards, and I expect many many many many more articles li…

> As for concerns about user tracking & privacy, that's not generally what these operational tools are used for. I'm not sure that the intended use matters. Unless the telemetry systems are carefully designed to not capture PII at all they become yet another channel that must be secured as if they are collecting PII. For example, see Windows 10 telemetry vs. HIPPA: https://hipaaone.com/2015/09/22/windows-10-and-hipaa…

I suspect you're tripping on the flowers of the carpet. Telemetry here is not a word meaning "call home and report user data", it means "telemetry" in the sense that a Mars rover sends back positioning & orientation data back to NASA for control. The usage of this tool is equivalent to having a query engine for application logs, or a statsd server to render graphs.

It's just a protocol to see how systems are performing, but a better one than statsd or logs.

Re: Why and how GitHub is adopting OpenTelemetry

#58

Post by geofft »

yannoninator wrote: post #25
rektide wrote: post #12

A lot of very confused weird comments here. This article is about a fairly large sized tech company adopting a fairly recent but increasingly mainstream & popular tool that helps them understand their operations. It'll give them a standard way to see what their computers are doing, across their various systems. OpenTelemetry is one of the key emerging cloud standards, and I expect many many many many more articles li…

Wait, you said: > I haven't heard of a single case of them being used for user tracking or behavioral analytics. and then you said: > That we will start to see the potential to harvest these records for analytics… So this can be used to gather analytics of any sort of data, such as spying then? This is still worrisome.

In that sentence, "these records" means "information about how GitHub's servers are behaving," not, say, "user data."

It is technically true that such a tool could be used on any sort of data, in the same way that, say, Perl or SQL can be used on any sort of data.

Re: Why and how GitHub is adopting OpenTelemetry

#59

Post by scrollaway »

yannoninator wrote: post #21

I’m now gravely concerned about GitHub after reading this. So now I should be excited about telemetry and that GitHub uses this in their systems? This raises alarm bells into if I should be using them for my repositories at all. Unsure if I should trust GitHub after this now. Downvoters: I don’t understand here? So I should be excited about telemetry? Why?

You're getting downvoted because you are complaining about something you don't understand, which puts into question the "after reading this" part of your post.

Jumping on the first opportunity to rant about telemetry without even checking what the article is talking about will do that.

Re: Why and how GitHub is adopting OpenTelemetry

#60

Post by tedk-42 »

thefounder wrote: post #53
ehershey wrote: post #46

Earlier quoted context omitted.

What do you mean by “sanitizing” the sdk’s? Did you remove the OpenTelemetry package?

>> Did you remove the OpenTelemetry package? Exactly!

With your level of debugging, I'd say the problem was pebcak
Post reply on HN