Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transparent telemetry for open-source projects

research.swtch.com

Post reply on HN
Print view View original

31–40 of 314 posts

Re: Transparent telemetry for open-source projects

#31

Post by teraflop »

I am all for transparency and limited intrusiveness of telemetry.

But in practical terms, the problem with this approach -- if I'm understanding it correctly -- is that it has no way to detect and reject outliers, and therefore the data can't be validated in any way. It only makes sense if all your clients are 100% trustworthy.

Let's say you want to know whether to keep supporting ARMv5, and your data says 10% of users are using it. There's no way to tell whether that's accurate, or if you have 0.01% of die-hard users who modified their telemetry code to report 1000x as frequently as they're supposed to. Even if you suspect this is happening (and you might not), there's no way to identify the culprit and filter out their data without tracking personal identifiers such as IP addresses.

So even if most of the time the telemetry data is valid, over time it will trend toward uselessness, because it can be endlessly second-guessed unless it confirms a decision you wanted to make anyway.

Re: Transparent telemetry for open-source projects

#32

Post by bioemerl »

Lammy wrote: post #21
bioemerl wrote: post #16

Honestly, this may be unpopular with hacker news, but just add your own telemetry. If people don't like it they can turn it off, and telemetry is essential for a good product. Do let people turn it off though please.

> telemetry is essential for a good product Up until now, you've had to make these design decisions on your own, relying only on perplexing intangibilities like 'taste' and 'intuition'.

I think it's somewhat silly to fly blind on the assumption that your taste is better than any real world observations you can make.

Especially if you haven't had the chance to develop an intuition yet and are new to the field. Without data to correct you, how do you get better?

Re: Transparent telemetry for open-source projects

#33

Post by pcthrowaway »

bioemerl wrote: post #16

Honestly, this may be unpopular with hacker news, but just add your own telemetry. If people don't like it they can turn it off, and telemetry is essential for a good product. Do let people turn it off though please.

I think maybe even making telemetry mandatory with an open license, and customizable with a support license might be a sustainable way to run an open source company.

For many open source project (anything with an attached business model), either telemetry or tight communication around usage patterns will be necessary to inform development. The latter of those two options consumes business resources.

Re: Transparent telemetry for open-source projects

#34

Post by waboremo »

Lammy wrote: post #21
bioemerl wrote: post #16

Honestly, this may be unpopular with hacker news, but just add your own telemetry. If people don't like it they can turn it off, and telemetry is essential for a good product. Do let people turn it off though please.

> telemetry is essential for a good product Up until now, you've had to make these design decisions on your own, relying only on perplexing intangibilities like 'taste' and 'intuition'.

Those design decisions were never made in a vacuum. They relied on telemetry (or, less scarily, user testing, user feedback, user research, etc) to figure out what works best. As a reminder, our intuition doesn't come from nowhere, rather centuries of survival and expectations. If you do not know what these expectations are, and if you do not know how your users interact with your product based on these expectations, you cannot make a good product. Certainly, you can make a product that appeals only to you, but how many of yous exist?

The design of a teapot is a great one. It didn't magically appear with handles and a spout and a place to hold leaves, but after years of refining based on usage. As shocking as that is, tea wasn't even discovered on purpose, let alone having a specific vessel for it right out the gate.

So yes, telemetry is essential. Taste is personal.

Re: Transparent telemetry for open-source projects

#35

Post by bee_rider »

I don’t really see why a classic community-driven open source project would care about what non-contributing users are doing with the software. In that case, helpful users come with built-in telemetry (pull requests).

But I guess this could be helpful corporatized read-only repo projects, or other groups that aren’t sure if they are building a community or a customer base.

Re: Transparent telemetry for open-source projects

#36

Post by Spivak »

groestl wrote: post #19
bioemerl wrote: post #16

Honestly, this may be unpopular with hacker news, but just add your own telemetry. If people don't like it they can turn it off, and telemetry is essential for a good product. Do let people turn it off though please.

> If people don't like it they can turn it off If I, perchance, encounter software I use phoning home without my explicit permission it's done on my systems. Period.

Well according to our telemetry 0% of users turn it off so it seems pretty popular.

But more realistically what you gain in privacy you give up in having your voice heard by the devs. The decisions about the future of the product/project will be driven by the data, specifically the data from the kind of people who leave telemetry on.

Re: Transparent telemetry for open-source projects

#37

Post by JohnFen »

> When you hear the word telemetry, if you’re like me, you may have a visceral negative reaction to a mental image of intrusive, detailed traces of your every keystroke and mouse click headed back to the developers of the software you’re using.

But that's not my only objection to telemetry. Equally important to me is that so many bad decisions are justified based on telemetry. It's very easy to misunderstand the data, because telemetry leaves out so much, but developers often treat it as if it's giving a complete picture.

As an example, I have seen developers drop really important functionality on the basis that it is rarely used. While that was true, it was also true that when those rare times happen, that functionality was absolutely critical to have.

Re: Transparent telemetry for open-source projects

#38

Post by JohnFen »

rsc wrote: post #27
slimsag wrote: post #23

I've been a pretty strong advocate of the idea that analytics should always be minimal, 100% anonymous, aggregated, and open to the public - otherwise it’s spying. This is how we do analytics on our websites today[0][1], and how we plan to do it in games we release in the future. Maybe one day I will start a dedicated FOSS service that people can use for exactly this with some trusted reputation/transparency/auditabi…

I am concerned about run-time telemetry in libraries as well. It might make sense for language ecosystems to offer more data about library usage gathered at build time eventually, as a different system than the one I'm posting about today. I think when you get to that level of detail you probably need to start thinking hard about differential privacy and probably cryptographic solutions like ESA or Prio. I don't thin…

Telemetry embedded in libraries is simply abusive, in my opinion. At the very least, the decision about whether or not to include telemetry should be made by the application developers, not the toolmakers.

Re: Transparent telemetry for open-source projects

#39

Post by bioemerl »

pcthrowaway wrote: post #33
bioemerl wrote: post #16

Honestly, this may be unpopular with hacker news, but just add your own telemetry. If people don't like it they can turn it off, and telemetry is essential for a good product. Do let people turn it off though please.

I think maybe even making telemetry mandatory with an open license, and customizable with a support license might be a sustainable way to run an open source company. For many open source project (anything with an attached business model), either telemetry or tight communication around usage patterns will be necessary to inform development. The latter of those two options consumes business resources.

Mandatory with an open license might shoot yourself in the foot when people fork your code.

For me, the harshest you can go is to have telemetry and not prompt for the setting on start with the user opted in by default. Ideally, you ask on first load, and that's what I'd probably aim for.

You can't ever try to force people to do anything in open source. They'll run right by you and make it do what they want it to do with or without you.

I'd even imagine paying customers are broadly more happy with telemetry than open source ones. And their needs more important anyways.

Re: Transparent telemetry for open-source projects

#40

Post by JohnFen »

gen220 wrote: post #15
mordae wrote: post #13

I dunno. It sure makes sense to me to collect telemetry from free software installations, but I feel that having every platform or even piece of software to do it on its own with opt-out will inevitably lead to people being overwhelmed and angry. I would, personally, prefer a single non-profit service that would list publicly what is being collected and publish the results as open data for anyone to use. Applications…

I think the issue of telemetry is fundamentally a human issue of incentives and trust. The system you describe is wise because it recognizes this and attempts to address it. The difficulty with telemetry is that even if we design the perfect, privacy-preserving system to begin with, once the pattern of having a network port open is established, there's nothing to prevent us (humans) from changing our policies about w…

> The difficulty with telemetry is that even if we design the perfect, privacy-preserving system

The other difficult is what you mentioned: trust. Even if a piece of software really does telemetry in a perfect, privacy-preserving way -- as a user, I have to take the developer's word for that in the end. That's a hard hurdle to pass, because that trust has been violated so much in the past that nobody gets the benefit of the doubt anymore.

> Unfortunately, even though this problem has software as its medium, it is a problem that cannot be solved by clever software alone

I agree entirely. At the heart of it, this is not a technological problem. It's a human one.

Post reply on HN