Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More Memory Safety for Let's Encrypt: Deploying ntpd-rs

letsencrypt.org

Post reply on HN
Print view View original

31–40 of 177 posts

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#31

Post by hackernudes »

syncsynchalt wrote: post #21
motrm wrote: post #19

Earlier quoted context omitted.

If the pieces of state are all well known at build time - and trusted in terms of their content - it may be feasible to print out JSON 'manually' as it were, instead of needing to use a JSON library, print "{" print "\"some_state\": \""; print GlobalState.Something.to_text(); print "\", "; print "\"count_of_frobs\": "; print GlobalState.FrobsCounter; print "}"; Whether it's worth doing this just to rid yourself of a…

Even better to just use TSV. Hand-rolling XML or JSON is always a smell to me, even if it's visibly safe.

Do you mean TLV (tag-length-value)? I can't figure out what TSV is.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#32

Post by FredFS456 »

hackernudes wrote: post #31
syncsynchalt wrote: post #21

Earlier quoted context omitted.

Even better to just use TSV. Hand-rolling XML or JSON is always a smell to me, even if it's visibly safe.

Do you mean TLV (tag-length-value)? I can't figure out what TSV is.

Tab Separated Values, like CSV but tabs instead of commas.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#34

Post by oconnor663 »

cogman10 wrote: post #28

This seems like a weird place to be touting memory safety. It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many VMs without exploding memory for a while now. I'd think there are far more critical components to rewrite in a memory safe language than the clock synchronizer.

> it's been running on many VMs without exploding memory for a while now

Most of the security bugs we hear about don't cause random crashes on otherwise healthy machines, because that tends to get them noticed and fixed. It's the ones that require complicated steps to trigger that are really scary. When I look at NTP, I see a service that:

- runs as root

- talks to the network

- doesn't usually authenticate its traffic

- uses a bespoke binary packet format

- almost all network security depends on (for checking cert expiration)

That looks to me like an excellent candidate for a memory-safe reimplementation.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#35

Post by Faaak »

nubinetwork wrote: post #33

The problem with ntp isn't the client, it's the servers having to deal with forged UDP packets. Will ntpd ever become TCP-only? Sadly I'm not holding my breath. I stopped running a public stratum 3 server ~10 years ago.

On the contrary, I'm hosting a stratum 1 and 2 stratum 2s (at my previous company we offered 3 stratum 1s) on the ntp pool. It's useful, used, and still needed :-)

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#36

Post by cogman10 »

luma wrote: post #30
cogman10 wrote: post #28

This seems like a weird place to be touting memory safety. It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many VMs without exploding memory for a while now. I'd think there are far more critical components to rewrite in a memory safe language than the clock synchronizer.

It's present on loads of systems, it's a very common service to offer, it's a reasonably well-constrained use case, and the fact that nobody thinks about it might be a good reason to think about it. They can't boil the ocean but one service at a time is a reasonable approach. I'll flip the question around, why not start at ntpd?

> I'll flip the question around, why not start at ntpd?

Easy, because there are loads of critical infrastructure written in C++ that is commonly executed on pretty much every VM and exposed in such a way that vulnerabilities are disasterous.

For example, JEMalloc is used by nearly every app compiled in *nix.

Perhaps systemd which is just about everywhere running everything.

Maybe sshd, heaven knows it's been the root of many attacks.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#37

Post by cogman10 »

oconnor663 wrote: post #34
cogman10 wrote: post #28

This seems like a weird place to be touting memory safety. It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many VMs without exploding memory for a while now. I'd think there are far more critical components to rewrite in a memory safe language than the clock synchronizer.

> it's been running on many VMs without exploding memory for a while now Most of the security bugs we hear about don't cause random crashes on otherwise healthy machines, because that tends to get them noticed and fixed. It's the ones that require complicated steps to trigger that are really scary. When I look at NTP, I see a service that: - runs as root - talks to the network - doesn't usually authenticate its traff…

> runs as root

ntpd can (and should) run as a user

> talks to the network

Makes outbound requests to the network. For it to be compromised, the network itself or a downstream server needs to be compromised. That's very different from something like hosting an http server.

> doesn't usually authenticate its traffic

Yes it does. ntp uses TLS to communicate with it's well known locations.

> uses a bespoke binary packet format

Not great but also see above where it's talking to well known locations authenticated and running as a user.

It's a service that to be compromised requires state level interference.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#38

Post by jaas »

cogman10 wrote: post #28

This seems like a weird place to be touting memory safety. It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many VMs without exploding memory for a while now. I'd think there are far more critical components to rewrite in a memory safe language than the clock synchronizer.

I'm the person driving this.

NTP is worth moving to a memory safe language but of course it's not the single most critical thing in our entire stack to make memory safe. I don't think anyone is claiming that. It's simply the first component that got to production status, a good place to start.

NTP is a component worth moving to a memory safe language because it's a widely used critical service on a network boundary. A quick Google for NTP vulnerabilities will show you that there are plenty of memory safety vulnerabilities lurking in C NTP implementations:

https://www.cvedetails.com/vulnerability-list/vendor_id-2153...

Some of these are severe, some aren't. It's only a matter of time though until another severe one pops up.

I don't think any critical service on a network boundary should be written in C/C++, we know too much at this point to think that's a good idea. It will take a while to change that across the board though.

If I had to pick the most important thing in the context of Let's Encrypt to move to a memory safe language it would be DNS. We have been investing heavily in Hickory DNS but it's not ready for production at Let's Encrypt yet (our usage of DNS is a bit more complex than the average use case).

https://github.com/hickory-dns/hickory-dns

Work is proceeding at a rapid pace and I expect Hickory DNS to be deployed at Let's Encrypt in 2025.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#39

Post by brohee »

nubinetwork wrote: post #33

The problem with ntp isn't the client, it's the servers having to deal with forged UDP packets. Will ntpd ever become TCP-only? Sadly I'm not holding my breath. I stopped running a public stratum 3 server ~10 years ago.

When one can make a stratum 1 server for $100, there is very little reason for the continuous existence of public NTP servers. ISP can offer the service to their customers, and any company with a semblance of IT dept can have its own stratum 1.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#40

Post by maxbond »

fiedzia wrote: post #24
motrm wrote: post #19

Earlier quoted context omitted.

If the pieces of state are all well known at build time - and trusted in terms of their content - it may be feasible to print out JSON 'manually' as it were, instead of needing to use a JSON library, print "{" print "\"some_state\": \""; print GlobalState.Something.to_text(); print "\", "; print "\"count_of_frobs\": "; print GlobalState.FrobsCounter; print "}"; Whether it's worth doing this just to rid yourself of a…

> If the pieces of state are all well known at build time - and trusted in terms of their content .. than use library, because you should not rely on the assumption that next developer adding one more piece to this code will magically remember to validate it with json spec.

No magic necessary. Factor your hand-rolling into a function that returns a string (instead of printing as in the example), and write a test that parses it's return with a proper JSON library. Assert that the parsing was successful and that the extracted values are correct. Ideally you'd use a property test.
Post reply on HN