41–48 of 48 posts
I'm curious about why you chose Yocto over buildroot for your official integration. I figured buildroot would be better, because Yocto's opkg system is superfluous on a device with full image updates.
Earlier quoted context omitted.
As someone that also implemented A/B booting for the Pi: I wonder how you roll back fully automated? I read a bit of the code but wasn't able to find that. Or is that already handled by u-boot? In my case, the first thing I do once an unverified version boots is to switch back to the other partition (so the known good version is active during the next boot), then run a detached reboot process that forces a reboot in…
I work on Mender, so I can tell you how automated rollback works there. The update is written to the inactive rootfs partition, uboot is configured to boot from it and the device is rebooted. Using the bootcount feature of uboot it is possible to roll back automatically if booting fails. Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll…
I see. Thanks for the info. I suspected that u-boot does have support for that, but I wasn't sure.
> Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll back.
Is there any deadline at all for that? I explicitly spawn a reboot command that ensures that even if everything gets stuck (in software, not in hardware) for whatever reason, the system falls back to the previous version (unless the reboot command gets killed too, in which case a manual restart is required). Any thoughts on that?
This is kind of a stupid question, but why not just use .rpm/dnf or .deb/apt and a custom repo?
I then proceeded to spend the next 2-3 hours reinstalling CentOS on my server, re-configuring oVirt and re-importing all of my VM's from the store.
Now, imagine an embedded device with a potentially flaky power source that could be interrupted in the middle of a system update - you really want to avoid that situation if at all possible. OSTree or Mender are a much better solution when you want an "all or nothing" upgrade, especially as they protect the system from FAILED updates and allow easy rollback.
Can someone summarize the difference between Mender and OSTree? I see that QtOTA chose OSTree as their underlying mechanism, which is significant in embedded automotive industry.
We have been evaluating OSTree as a potential building block for Mender, however the key challenges we've come across:
integrating OSTree into an existing device/OS seems quite invasive - https://ostree.readthedocs.io/en/latest/manual/adapting-exis...
block level signatures is not possible, which we feel is a requirement for an over-the-air updater
rollback is not built-in and can be quite challenging to implement reliably (bootloader level)
Think OSTree as more of a building block, like Git is for your development process. We might use it in the future but robustness and easy integration are our first priorities.
Can someone summarize the difference between Mender and OSTree? I see that QtOTA chose OSTree as their underlying mechanism, which is significant in embedded automotive industry.
We have been evaluating OSTree as a potential building block for Mender, however the key challenges we've come across: integrating OSTree into an existing device/OS seems quite invasive - https://ostree.readthedocs.io/en/latest/manual/adapting-exis... block level signatures is not possible, which we feel is a requirement for an over-the-air updater rollback is not built-in and can be quite challenging to implement re…
Earlier quoted context omitted.
I work on Mender, so I can tell you how automated rollback works there. The update is written to the inactive rootfs partition, uboot is configured to boot from it and the device is rebooted. Using the bootcount feature of uboot it is possible to roll back automatically if booting fails. Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll…
> Using the bootcount feature of uboot it is possible to roll back automatically if booting fails. I see. Thanks for the info. I suspected that u-boot does have support for that, but I wasn't sure. > Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll back. Is there any deadline at all for that? I explicitly spawn a reboot command that ens…
We have looked into hardware watchdog for this, but it is in the gray-zone of what an updater should be involved in. This is actually a more generic problem - maybe it hangs even when you did not deploy an update. There is varying support for hardware watchdogs across boards as well, unfortunately.
Most of the time it will not just hang, maybe it will crash or kernel panic and in those cases Mender will rollback. But the indefinite-hanging case is quite tricky and not yet handled.
Would be open to ideas here.
Earlier quoted context omitted.
We have been evaluating OSTree as a potential building block for Mender, however the key challenges we've come across: integrating OSTree into an existing device/OS seems quite invasive - https://ostree.readthedocs.io/en/latest/manual/adapting-exis... block level signatures is not possible, which we feel is a requirement for an over-the-air updater rollback is not built-in and can be quite challenging to implement re…
Why is it important to have signatures at the block level? Wouldn't signing an archive or binary diff be good enough?
For security-sensitive embedded devices (e.g. payment terminals), block level signatures would allow hardware verification during boot as well (1st stage bootloader verifies 2nd stage, then kernel, etc.) if designed correctly.
Earlier quoted context omitted.
> Using the bootcount feature of uboot it is possible to roll back automatically if booting fails. I see. Thanks for the info. I suspected that u-boot does have support for that, but I wasn't sure. > Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll back. Is there any deadline at all for that? I explicitly spawn a reboot command that ens…
This is a valid point. If booting just hangs after the bootloader but before the Mender daemon comes up is actually quite tricky to manage. We have looked into hardware watchdog for this, but it is in the gray-zone of what an updater should be involved in. This is actually a more generic problem - maybe it hangs even when you did not deploy an update. There is varying support for hardware watchdogs across boards as w…
The first step of our boot process was to enable the watchdog. We extend the timeout periodically during the boot process, but generally if userspace isn't reached within 30 seconds or so we reset. Once in userspace, the daemon validates that things look good (this includes things beyond just application of the update -- did services start up correctly? Is the hardware operating as we expect?) before disabling the watchdog and marking the update as a success, at which point rollback isn't possible. At this point we might consider applying new updates, etc.
We also modified our first stage bootloader to be resilient to bootloader update issues, and chainloaded our second stage bootloader from a stub which could rollback.
We also niced the update process to avoid resource contention, allowed the updates to be delayed until the network was quiet, and paused them when it became noisy to make for a good user experience. There was a server-side flag to force updates to apply regardless, with higher priority, as well as one to basically disable all other functionality in the case of a unforeseen serious, perhaps security related, issue.
We actually had a discrete watchdog service which was responsible for petting an always-on watchdog, to rescue the system if it locked up or became unresponsive (if certain processes were not running, or responding, the watchdog would not be pet).
All of this led to effectively 0 failures in the field, a seamless user experience (except for the 30-second reboot when inactive). I wish everything I owned worked this way.
I could talk ad nauseum about this stuff. It's very cool to see the designs of others. I feel this is an under appreciated and under explored problem space.