Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More Memory Safety for Let's Encrypt: Deploying ntpd-rs

letsencrypt.org

Post reply on HN
Print view View original

121–130 of 177 posts

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#121

Post by pjmlp »

ghshephard wrote: post #59
dheera wrote: post #55

Earlier quoted context omitted.

Why are C and C++ all of a sudden unsafe? Did I miss something? What is safe now? JavaScript? PyTorch?

One of the major drivers (if not the driver) for the creation of Rust the fact that C is not a memory-safe language. This has been known for decades, but it wasn't until 2010 that a serious attempt at writing a new system-language that was memory safe was attempted and got traction - Rust. https://kruschecompany.com/rust-language-concise-overview/#:... .

I would consider that serious attempts have been made since 1961, however all of them failed in the presence of a free beer OS, with free beer unsafe compilers.

The getting traction part is the relevance part.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#122

Post by westurner »

codetrotter wrote: post #105
1over137 wrote: post #81

Earlier quoted context omitted.

Never heard of it. Shockingly little on wikipedia for example.

Yeah. Seems it doesn’t even have its own article there. Only a short mention in the main article about NTP itself: > Network Time Security (NTS) is a secure version of NTPv4 with TLS and AEAD. The main improvement over previous attempts is that a separate "key establishment" server handles the heavy asymmetric cryptography, which needs to be done only once. If the server goes down, previous users would still be able…

"RFC 8915: Network Time Security for the Network Time Protocol" (2020) https://www.rfc-editor.org/rfc/rfc8915.html

"NTS RFC Published: New Standard to Ensure Secure Time on the Internet" (2020) https://www.internetsociety.org/blog/2020/10/nts-rfc-publish... :

> NTS is basically two loosely coupled sub-protocols that together add security to NTP. NTS Key Exchange (NTS-KE) is based on TLS 1.3 and performs the initial authentication of the server and exchanges security tokens with the client. The NTP client then uses these tokens in NTP extension fields for authentication and integrity checking of the NTP protocol messages that exchange time information.

From "Simple Precision Time Protocol at Meta" https://news.ycombinator.com/item?id=39306209 :

> How does SPTP compare to CERN's WhiteRabbit, which is built on PTP [and NTP NTS]?

White Rabbit Project: https://en.wikipedia.org/wiki/White_Rabbit_Project

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#123

Post by pjmlp »

hot_gril wrote: post #60
danudey wrote: post #10

Earlier quoted context omitted.

I hate writing code in golang, because I have to pepper every single function call with `if err != nil`, but then I think about how much C code I've seen that doesn't do that and I wonder how much of it should.

Golang's error handling is safer than C's, but it's more cumbersome than it needs to be. In high-level code, nearly every func you write can return an error, and 99% of the time you're just going to pass the error up. Webserver will catch all and send 4xx or 5xx, for example. Exceptions are a lot more convenient and encourage solid error handling. Rust chose a good in-between (the "?" unwrapping syntax). In mid or lo…

Every now and then I think about forking Go and adding something like ?, and Pascal/Modula-2 like enumerations.

However then I realise, why bother, and go back into using C#, Java, D instead.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#124

Post by stavros »

rnijveld wrote: post #113
akira2501 wrote: post #6

Why does your ntpd have a json dependency?

I don’t think our dependency tree is perfect, but I think our dependencies are reasonable overall. We use JSON for transferring metrics data from our NTP daemon to our prometheus metrics daemon. We’ve made this split for security reasons, why have all the attack surface of a HTTP server in your NTP daemon? That didn’t make sense to us. Which is why we added a readonly unix socket to our NTP daemon that on connecting…

If you're only dumping a string, couldn't you replace this dependency with some string concatenation?

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#125

Post by rnijveld »

cogman10 wrote: post #28

This seems like a weird place to be touting memory safety. It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many VMs without exploding memory for a while now. I'd think there are far more critical components to rewrite in a memory safe language than the clock synchronizer.

I do think that memory safety is important for any network service. The probability of something going horribly wrong when a network packet is parsed in a wrong way is just too high. NTP typically does have more access to the host OS than other daemons, with it needing to adjust the system clock.

Of course, there are many other services that could be made memory safe, and maybe there is some sort of right or smart order in which we should make our core network infrastructure memory safe. But everyone has their own priorities here, and I feel like this could end up being an endless debate of whatabout-ism. There is no right place to start, other than to just start.

Aside from memory safety though, I feel like our implementation has a strong focus on security in general. We try and make choices that make our implementation more robust than what was out there previously. Aside from that, I think the NTP space has had an under supply of implementations, with there only being a few major open source implementations (like ntpd, ntpsec and chrony). Meanwhile, NTP is one of those pieces of technology at the core of many of the things we do on the modern internet. Knowing the current time is one of these things you just need in order to trust many of the things we take for granted (without knowledge of the current time, your TLS connection could never be trusted). I think NTP definitely deserves this attention and could use a bunch more attention.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#126

Post by rnijveld »

hi-v-rocknroll wrote: post #114
ComputerGuru wrote: post #20

Unlike say, coreutils, ntp is something very far from being a solved problem and the memory safety of the solution is unfortunately going to play second fiddle to its efficacy. For example, we only use chrony because it’s so much better than whatever came with your system (especially on virtual machines). ntpd-rs would have to come at least within spitting distance of chrony’s time keeping abilities to even be up for…

You might be doing too much work at the wrong level of abstraction. VMs should use host clock synchronization. It requires some work and coordination, but it eliminates the need for ntp in VMs entirely. Hosts should then be synced using PTP or a proper NTP local stratum (just get a proper GNSS source for each DC if you have then funds). https://tsn.readthedocs.io/timesync.html Deploy chrony to bare metal servers wher…

Our project also includes a PTP implementation, statime (https://github.com/pendulum-project/statime/), that includes a Linux daemon. Our implementation should work as well or even better than what linuxptp does, but it's still early days. One thing to note though is that NTP can be made to be just as precise (if not more precise), given the right access to hardware (unfortunately most hardware that does timestamping only does so for PTP packets). The reason for this precision is simple: NTP can use multiple sources of time, whereas PTP by design only uses a single source. This gives NTP more information about the current time and thus allows it to more precisely estimate what the current time is. The thing with relying purely on GNSS is that those signals can be (and are in practice) disrupted relatively easily. This is why time synchronization over the internet makes sense, even for large data centers. And doing secure time synchronization over the internet is only practically possible using NTP/NTS at this time. But there is no one size fits all solution for time synchonization in general.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#127

Post by rnijveld »

1over137 wrote: post #81
timmytokyo wrote: post #41

Earlier quoted context omitted.

I really wish more internet infrastructure would switch to using NTS. It addresses these kinds of issues.

Never heard of it. Shockingly little on wikipedia for example.

I'm afraid this is a pretty common sentiment. NTS has been out for several years already and is implemented in several implementations (including our ntpd-rs implementation, and others like chrony and ntpsec). Yet its usage is low and meanwhile the fully unsecured and easily spoofable NTP remains the default, in effect allowing anyone to manipulate your clock almost trivially (see our blog post about this: https://tweedegolf.nl/en/blog/121/hacking-time). Hopefully we can get NTS to the masses more quickly in the coming years and slowly start to decrease our dependency on unsigned NTP traffic, just as we did with unencrypted HTTP traffic.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#128

Post by tialaramex »

patmorgan23 wrote: post #80
orf wrote: post #15

Earlier quoted context omitted.

Would you rather it had a JSON dependency to parse a config file, or yet another poorly thought out, ad-hoc homegrown config file format?

Why isn't there a decent parser in the standard library? More than 50% of programs will probably touch json at this point.

Rust's stdlib is (at least notionally) forever. Things in the standard library (including core and alloc) get deprecated but must be maintained forever, which means that "at this point" isn't enough.

In 2003 those programs would have used XML, in 1993 probably .INI files. Are you sure that despite all its shortcomings JSON is the end of history? I don't believe you.

If you want "at this point" you can, as software does today, just use a crate. Unlike the stdlib, if next week Fonzie files are huge and by 2026 "nobody" is using JSON because Fonzie is cool, the JSON config crate merely becomes less popular.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#129

Post by rnijveld »

dequan wrote: post #74
johnklos wrote: post #72

[flagged]

I agree that it would be great if the ecosystem was a bit slower to use every new version and it does seem like things are beginning to tend in that direction as many foundational crates have begun declaring MSRVs of !LATEST. However I don't think the pace of updates really changes anything in terms of tool chain security. If Rust decided to go to a 36 week release cycle, each release would just have 6x as much stuff…

I also agree that there is too much churn in the Rust ecosystem and that we should try and slow things down in the coming years. ntpd-rs also does this: our MSRV is 1.70 right now (that was released over a year ago) and we test our code on CI against this version (as well as the current stable release). And we go a little further. Using the `direct-minimal-versions` (nightly only right now unfortunately) flag we downgrade our dependencies to the minimal ones we've specified in our `Cargo.toml` and test against those dependencies as well, as well as the latest dependencies specified in `Cargo.lock` which we update regularly. This allows us to at least partially verify that we still work against old versions of our dependencies, allowing our upstream packagers to more easily match their packages against our own. Of course we all should update to newer versions whenever possible, but sometimes that is hard to do (especially for package maintainers in distributions such as Fedora and Debian, who have to struggle with so many packages at the same time) and we shouldn't create unnecessary work when its not needed. Hopefully this is our way of helping the ecosystem slow down a little and focus more on security and functionality and less on redoing the same thing all over again every year because of some shiny new feature.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#130

Post by ssl-3 »

brohee wrote: post #39
nubinetwork wrote: post #33

The problem with ntp isn't the client, it's the servers having to deal with forged UDP packets. Will ntpd ever become TCP-only? Sadly I'm not holding my breath. I stopped running a public stratum 3 server ~10 years ago.

When one can make a stratum 1 server for $100, there is very little reason for the continuous existence of public NTP servers. ISP can offer the service to their customers, and any company with a semblance of IT dept can have its own stratum 1.

One can build a GPS-backed stratum 1 server for a lot less than $100 in hardware (and I have done so in the past). It was a fun little project for me, but it involved a confluence of skillsets that many [especially smaller] companies may not collectively possess. And even then, it needs to be documented so someone else can work on it, and that maintenance has to actually-happen, and it needs a view of the sky in order for it to chooch, and it also needs redundancy. This takes time. (And if this IT department doesn't work for free, then the cost is very quickly a lot more than $100.)

And going full-assed with one or more actually-outside antennas can also be problematic, since it's a Bad Day when (eg) grounding is done improperly and lightning comes by to fuck their shit up.

And ISPs can (and certainly should!) provide decent NTP servers. That's the logical place for shared servers, network-wise. But the one's choice of ISP is often limited by geography, and the most-viable ISP here stopped publishing what their NTP servers are -- if they still exist in a form that customers can use, they don't publish this information. (It wasn't always this way and I remember using them years ago when they were more forthcoming. They were shitty NTP servers with high jitter. Good enough for some things, I suppose, but not as good as the members of the public NTP pool tend to be.)

I mean: Many ISPs can't even manage to handle DNS queries quickly. When public servers like 8.8.8.8 and 1.1.1.1 (or the semi-public ones like 4.2.2.1) are faster than the ISP's own servers, then that's a problem. And it's a stupid problem, and it should not happen. But it does happen.

So thus, public NTP servers are useful for many -- including those who have a tinkered-together NTP server with a GPS antenna in a window somewhere, where public servers can be used as backup.

It's good to have options, and it's nice that some organizations provide some options for the greater network.

Post reply on HN