Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More Memory Safety for Let's Encrypt: Deploying ntpd-rs

letsencrypt.org

Post reply on HN
Print view View original

111–120 of 177 posts

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#111

Post by rnijveld »

ComputerGuru wrote: post #20

Unlike say, coreutils, ntp is something very far from being a solved problem and the memory safety of the solution is unfortunately going to play second fiddle to its efficacy. For example, we only use chrony because it’s so much better than whatever came with your system (especially on virtual machines). ntpd-rs would have to come at least within spitting distance of chrony’s time keeping abilities to even be up for…

I would encourage you to take a look at some of our testing data and an explanation of our algorithm in our repository (https://github.com/pendulum-project/ntpd-rs/tree/main/docs/a...). I think we are very much in spitting distance of Chrony in terms of synchronization performance, sometimes even beating Chrony. But we’d love for more people to try our algorithm in their infrastructure and report back. The more data the better.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#112

Post by comex »

Spivak wrote: post #98
Gigachad wrote: post #88

Earlier quoted context omitted.

This looks like the exact kind of thing that results in unexpected exploits.

Hand rolled JSON input processing, yes. Hand rolled JSON output, no. You're gonna have a hard time exploiting a text file output that happens to be JSON.

> You're gonna have a hard time exploiting a text file output that happens to be JSON.

If you’re not escaping double quotes in strings in your hand-rolled JSON output, and some string you’re outputting happens to be something an attacker can control, then the attacker can inject arbitrary JSON. Which probably won’t compromise the program doing the outputting, but it could cause whatever reads the JSON to do something unexpected, which might be a vulnerability, depending on the design of the system.

If you are escaping double quotes, then you avoid most problems, but you also need to escape control characters to ensure the JSON isn’t invalid. And also check for invalid UTF-8, if you’re using a language where strings aren’t guaranteed to be valid UTF-8. If an attacker can make the output invalid JSON, then they can cause a denial of service, which is typically not considered a severe vulnerability but is still a problem. Realistically, this is more likely to happen by accident than because of an attacker, but then it’s still an annoying bug.

Oh, and if you happen to be using C and writing the JSON to a fixed-size buffer with snprintf (I’ve seen this specific pattern more than once), then the output can be silently truncated, which could also potentially allow JSON injection.

Handling all that correctly doesn’t require that much code, but it’s not completely trivial either. In practice, when I see code hand-roll JSON output, it usually doesn’t even bother escaping anything. Which is usually fine, because the data being written is usually not attacker-controlled at all. For now. But code has a tendency to get adapted and reused in unexpected ways.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#113

Post by rnijveld »

akira2501 wrote: post #6

Why does your ntpd have a json dependency?

I don’t think our dependency tree is perfect, but I think our dependencies are reasonable overall. We use JSON for transferring metrics data from our NTP daemon to our prometheus metrics daemon. We’ve made this split for security reasons, why have all the attack surface of a HTTP server in your NTP daemon? That didn’t make sense to us. Which is why we added a readonly unix socket to our NTP daemon that on connecting dumps a JSON blob and then closes the connection (i.e. doing as little as possible), which is then usable by our client tool and by our prometheus metrics daemon. That data transfer uses json, but could have used any data format. We’d be happy to accept pull requests to replace this data format with something else, but given budget and time constraints, I think what we came up with is pretty reasonable.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#114

Post by hi-v-rocknroll »

ComputerGuru wrote: post #20

Unlike say, coreutils, ntp is something very far from being a solved problem and the memory safety of the solution is unfortunately going to play second fiddle to its efficacy. For example, we only use chrony because it’s so much better than whatever came with your system (especially on virtual machines). ntpd-rs would have to come at least within spitting distance of chrony’s time keeping abilities to even be up for…

You might be doing too much work at the wrong level of abstraction. VMs should use host clock synchronization. It requires some work and coordination, but it eliminates the need for ntp in VMs entirely.

Hosts should then be synced using PTP or a proper NTP local stratum (just get a proper GNSS source for each DC if you have then funds).

https://tsn.readthedocs.io/timesync.html

Deploy chrony to bare metal servers wherever possible.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#116

Post by dataking »

dheera wrote: post #61
ghshephard wrote: post #59

Earlier quoted context omitted.

One of the major drivers (if not the driver) for the creation of Rust the fact that C is not a memory-safe language. This has been known for decades, but it wasn't until 2010 that a serious attempt at writing a new system-language that was memory safe was attempted and got traction - Rust. https://kruschecompany.com/rust-language-concise-overview/#:... .

How is C not memory safe? If I access memory I didn't allocate the OS shuts the program down. Is that not memory safety? (Unless you're running it on bare metal ...)

Highly recommend Alex Gaynor's intro to memory unsafety https://alexgaynor.net/2019/aug/12/introduction-to-memory-un...

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#117

Post by masklinn »

syncsynchalt wrote: post #21
motrm wrote: post #19

Earlier quoted context omitted.

If the pieces of state are all well known at build time - and trusted in terms of their content - it may be feasible to print out JSON 'manually' as it were, instead of needing to use a JSON library, print "{" print "\"some_state\": \""; print GlobalState.Something.to_text(); print "\", "; print "\"count_of_frobs\": "; print GlobalState.FrobsCounter; print "}"; Whether it's worth doing this just to rid yourself of a…

Even better to just use TSV. Hand-rolling XML or JSON is always a smell to me, even if it's visibly safe.

Hand-rolling TSV is no better. The average TSV generator does not pay any mind to data cleaning, and quoting / escaping is non-standard, so what the other wide will do with it is basically playing russian roulette.

Using C0 codes is likely safer at least in the sense that you will probably think to check for those and there is no reason whatsoever for them to be found in user data.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#118

Post by cowsandmilk »

TeMPOraL wrote: post #104
kortilla wrote: post #68

Earlier quoted context omitted.

This is not a question for HN at this point. It’s like asking why SQL query forming via string concatenation of user inputs is unsafe. Google it, C memory boundary issues have been a problem for security forever.

> It’s like asking why SQL query forming via string concatenation of user inputs is unsafe. To be honest, that's a surprisingly deep question, and the answer is something I'm yet to see any developer I worked with understand. For example, did you know that SQL injection and XSS are really the same problem? And so is using template systems[0] like Mustache? In my experience, very few people appreciate that the issue w…

Really, the problem is in combining tainted input strings with string concatenation. If you have certain guarantees on the input strings, concatenation can be safe. That said, I still wouldn’t use it since there are few guarantees that future code wouldn’t introduce tainted strings.

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#119

Post by rnijveld »

syncsynchalt wrote: post #25
ComputerGuru wrote: post #20

Unlike say, coreutils, ntp is something very far from being a solved problem and the memory safety of the solution is unfortunately going to play second fiddle to its efficacy. For example, we only use chrony because it’s so much better than whatever came with your system (especially on virtual machines). ntpd-rs would have to come at least within spitting distance of chrony’s time keeping abilities to even be up for…

The biggest danger in NTP isn't memory safety (though good on this project for tackling it), it's (a) the inherent risks in implementing a protocol based on trivially spoofable UDP that can be used to do amplification and reflection and (b) emergent resonant behavior from your implementation that will inadvertently DDOS critical infrastructure when all 100m installed copies of your daemon decide to send a packet to N…

I agree that amplification and reflection definitely are worries, which is why we are working towards NTS becoming a default on the internet. NTS would prevent responses by a server from a spoofed packet and at the same time would make sure that NTP clients can finally start trusting their time instead of hoping that there are no malicious actors anywhere near them. You can read about it on our blog as well: https://tweedegolf.nl/en/blog/122/a-safe-internet-requires-s...

One thing to note about amplification: amplification has always been something that NTP developers have been especially sensitive to. I would say though that protocols like QUIC and DNS have far greater amplification risks. Meanwhile, our server implementation forces that responses can never be bigger than the requests that initiated them, meaning that no amplification is possible at all. Even if we would have allowed bigger responses, I cannot imagine NTP responses being much bigger than two or three times their related request. Meanwhile I've seen numbers for DNS all the way up to 180 times the request payload.

As for your worries: I think being a little cautious keeps you alert and can prevent mistakes, but I also feel that we've gone out of our way to not do anything crazy and hopefully we will be a net positive in the end. I hope you do give us a try and let us know if you find anything suspicious. If you have any feedback we'd love to hear it!

Re: More Memory Safety for Let's Encrypt: Deploying ntpd-rs

#120

Post by pjmlp »

dheera wrote: post #55
jaas wrote: post #38

Earlier quoted context omitted.

I'm the person driving this. NTP is worth moving to a memory safe language but of course it's not the single most critical thing in our entire stack to make memory safe. I don't think anyone is claiming that. It's simply the first component that got to production status, a good place to start. NTP is a component worth moving to a memory safe language because it's a widely used critical service on a network boundary.…

Why are C and C++ all of a sudden unsafe? Did I miss something? What is safe now? JavaScript? PyTorch?

They have been unsafe from their very early days, Multics got a higher security score than UNIX thanks to PL/I, and C.A.R Hoare has addressed C's existence on his Turing Award in 1980, while Fran Allen has also made similar remarks back in the day.
Post reply on HN