Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

mender.io

Post reply on HN
Print view View original

11–20 of 41 posts

Re: Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

#11

Post by geofft »

Cyph0n wrote: post #10
geofft wrote: post #9

Earlier quoted context omitted.

I'm not sure what part you're referring to as FUD - the cost? I definitely agree with the rest of the comment. It's a good solution. I inherited a software product that worked like this at a past job, and it was great. (And now I'm trying to convince my current job that we want to move to this model for normal servers in datacenters, very much not IoT.) A couple of complications: - You should think about the fact tha…

How do you handle key and/or certificate storage at the client side? Depending on the threat model, the update verification step can be subverted.

You install manually / out-of-band (in my old job, you installed by copying to a USB stick from a desktop machine), and the updater runs within the OS, because you can safely write to the unused partition while the OS runs. So it has the full set of functionality that you ship with your OS - Python, OpenSSL, GPG, whatever. You're not downloading the update from the bootloader or anything (which would take too long).

Once the device has been installed, there is always at least one working partition on the device - the partition that was last booted. So you don't need a minimal recovery partition or anything. (You could build a recovery command-line option in, if you want, but it's just a custom way of booting the normal partition.)

Re: Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

#12

Post by Cyph0n »

geofft wrote: post #11
Cyph0n wrote: post #10

Earlier quoted context omitted.

How do you handle key and/or certificate storage at the client side? Depending on the threat model, the update verification step can be subverted.

You install manually / out-of-band (in my old job, you installed by copying to a USB stick from a desktop machine), and the updater runs within the OS, because you can safely write to the unused partition while the OS runs. So it has the full set of functionality that you ship with your OS - Python, OpenSSL, GPG, whatever. You're not downloading the update from the bootloader or anything (which would take too long).…

How do you protect keys and/or certificates stored on the device from being exfiltrated or replaced by an adversary?

Re: Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

#13

Post by kahlonel »

I wonder how will a Git based client-side agent fit in. Why not use an already proven tech for the rollout, and then use custom installation scripts (also within the git repo) for doing software setup? The "server" here will just be normal git server, with commit ID serving as version number.

Re: Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

#14

Post by ralphmender »

sjmulder wrote: post #4
jon-wood wrote: post #2

This is wonderfully timed, I’m currently looking for options when it comes to doing firmware updates on Linux based IoT devices. Does anyone have any recommendations?

A friend who is building an Linux appliance is using Yocto: https://www.yoctoproject.org/ . He also talked about Mender, mentioned in the other comment. I don't know how much overlap there is between these projects.

Disclaimer: I'm with Mender.io and author of the article.

The Yocto Project is a popular build system for your own embedded Linux distribution.

Mender integrates with the Yocto Project with a layer (https://github.com/mendersoftware/meta-mender), but is a separate project for end-to-end OTA, which includes the client and the management server. Both are licensed under Apache 2.0 thus it is freely available and you're not locked into a hosted-only backend.

Although this is an older blog post, here is how you can port the Mender client to a non-Yocto build system: https://mender.io/blog/porting-mender-to-a-non-yocto-build-s...

Re: Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

#15

Post by wmf »

Cyph0n wrote: post #12
geofft wrote: post #11

Earlier quoted context omitted.

You install manually / out-of-band (in my old job, you installed by copying to a USB stick from a desktop machine), and the updater runs within the OS, because you can safely write to the unused partition while the OS runs. So it has the full set of functionality that you ship with your OS - Python, OpenSSL, GPG, whatever. You're not downloading the update from the bootloader or anything (which would take too long).…

How do you protect keys and/or certificates stored on the device from being exfiltrated or replaced by an adversary?

Unless you have a TPM or something you just don't; this is an unsolvable problem in general.

Re: Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

#16

Post by tomc1985 »

geofft wrote: post #9
tomc1985 wrote: post #6

This is FUD. Partition twice the amount of space you need on IOT's fixed media, and trickle-download your update to the empty partition. Once that's finished, verify the download then switch your bootloader. I have to update 500+ field installations with a new OS and this is the approach I'd take if we had the space and bandwidth (Instead we're sending out an army of techs and account reps armed with USB sticks)

I'm not sure what part you're referring to as FUD - the cost? I definitely agree with the rest of the comment. It's a good solution. I inherited a software product that worked like this at a past job, and it was great. (And now I'm trying to convince my current job that we want to move to this model for normal servers in datacenters, very much not IoT.) A couple of complications: - You should think about the fact tha…

I looked at the document and it immediately reminded me of the kind of person who at meetings brings up tons of questions that may or may not be relevant to the actual task. There is a recurring thread of problems in CS being really simple but then needlessly complicated by well-intentioned nerds unable to see the forst for the trees. I say FUD because I see helpful blog posts promoting new tools and software when they should be promoting techniques.

Seriously those stack diagrams are massively overkill designs for OTA update infra.

As for your specific points, I left out a LOT of detail.

  - You should think about the fact that this means your root partition changes
Yes, or no, depending on your bootloader. GRUB and others can chainload to other partitions after control is given. Of course you'll need to think about how to re-provision user-specific data, though the first idea that pops in my mind would be to simply read it from the old install on first boot. (IRL we are solving this with a first-boot 'activation' app, which re-downloads user data on completion)

  - You do want a management server, as this document suggests...
yes, but this is way easier than needing to learn and roll out someone else's complicated piece of tech. An API server that can receive updates from the field and download scripts/run the commands to start the process.

Also, if you've been supporting this kind of product for a while you probably already have your own infra.

  - You want some mechanism for detecting if the new version doesn't work and rolling back;
of course. Though, if your hardware is uniform you can get away with a lot less of this than one might think, due to the updates being image-based and not package-based. Easiest way is to simply compute a SHA-hash on the download to make sure it wasn't corrupted (make sure the value you compare it with came from a trusted conversation with your server). If it fails to boot the newly provisioned code needs to know to re-point the bootloader at the previous install, though again with uniform hardware your main concerns are transport-related.

  - The updates should be signed etc. as described in the document.
Fairly trivial with pgp et al.

Re: Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

#17

Post by wmf »

kahlonel wrote: post #13

I wonder how will a Git based client-side agent fit in. Why not use an already proven tech for the rollout, and then use custom installation scripts (also within the git repo) for doing software setup? The "server" here will just be normal git server, with commit ID serving as version number.

That's addressed by the article. What about a management server, power/network loss, atomic updates, validation, etc? If you try to write custom installation scripts to do all that it's going to be a lot of work.

Re: Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

#18

Post by seanalltogether »

I've been contracting with an IoT company for 3 years now. It's interesting to see guys who coded exclusively for radio or infrared based remotes get pulled into the software development world.

V1: 5 years ago everything was raw sockets and custom messaging formats with hand coded firmware and all data stored in a custom vector format, builds were distributed on google drive and flashed by hand.

V2: 3 years ago we dragged them kicking and screaming into http and hand coded json apis, firmware was still custom, data still stored in custom vector format but updates were now done on a non secure server with a hash check.

V3: this past year they started on a small box with a micro linux distro, apis are provided by standardized library, data now stored in sql, updates done over https.

Things are better now, except they still expect to sell and support those first 2 options for the next 10 years.

Re: Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

#19

Post by eystein »

Cyph0n wrote: post #10
geofft wrote: post #9

Earlier quoted context omitted.

I'm not sure what part you're referring to as FUD - the cost? I definitely agree with the rest of the comment. It's a good solution. I inherited a software product that worked like this at a past job, and it was great. (And now I'm trying to convince my current job that we want to move to this model for normal servers in datacenters, very much not IoT.) A couple of complications: - You should think about the fact tha…

How do you handle key and/or certificate storage at the client side? Depending on the threat model, the update verification step can be subverted.

Disclaimer: I work on the Mender project.

Signing and verification in Mender is covered here: https://docs.mender.io/artifacts/signing-and-verification

Re: Software Updates for IoT Devices and the Hidden Costs of Homegrown Updaters [pdf]

#20

Post by geofft »

Cyph0n wrote: post #12
geofft wrote: post #11

Earlier quoted context omitted.

You install manually / out-of-band (in my old job, you installed by copying to a USB stick from a desktop machine), and the updater runs within the OS, because you can safely write to the unused partition while the OS runs. So it has the full set of functionality that you ship with your OS - Python, OpenSSL, GPG, whatever. You're not downloading the update from the bootloader or anything (which would take too long).…

How do you protect keys and/or certificates stored on the device from being exfiltrated or replaced by an adversary?

For all of my use cases, either physical access counts as game over and is out of the threat model, or we're using Secure Boot for verifying the bootloader and/or TPMs for keeping secrets, at which point this is a problem with known solutions that aren't specific to image-based updates. If you're using Secure Boot with read-only images, one thing to try is dm_verity, which is how Chrome OS solves this problem - it's a Merkle hash of the entire block device that's checked lazily as blocks are accessed. If a block has been tampered with, you can configure dm_verity to either panic the system or return an I/O error for that block. (Or you can just read the entire image and verify it up front at the cost of slower boot time.)

In particular, for all my use cases, there's already some mechanism for the device gaining a secure channel to the rest of the infrastructure, so if you're worried about keeping updates secret (which you may or may not be!), just protect updates by that mechanism.

Post reply on HN