Live data from Hacker News

QEMU Internals

airbus-seclab.github.io

21–30 of 36 posts

Re: QEMU Internals

#21

I don't mean this to disparage Airbus in any way but after Boeing's issues with the 737 MAX I'd assumed a fairly poor culture of software at airplane manufacturers in general. Super glad to see work like this coming out of Airbus, really makes me rethink my earlier assumptions about software competence in the field.

Is "move fast and break things" a good culture for airplane manufacturer? Airbus is known for making good software, they earned their reputation by releasing the first fly by wire airliner (a320) in 84, which forced Boeing to go this route with the 777.

Making safety critical software is a totally different world than what is seen on HN. The culture needed is safety culture and it is all about doing boring code, following strict coding rules, doing tons of documentation and analysis prior coding and a doing tons of review of tests. I don't think it will arouse interest here.

Re: QEMU Internals

#22
post #18

Earlier quoted context omitted.

That is such a bizarre viewpoint from my perspective. The absolute deathtrap that is the 737 MAX had two software-related critical failures in 400,000 flights. That constitutes a whole system per-flight software reliability of 2 in ~400,000 or a ~99.9995%, 5 9s. Obviously that is still unacceptable as that is far below the software standard amongst all commercial airplanes where software has not been implicated in a…

That’s an interesting way to look at uptime no pun intended thou I wouldn’t buy a Toyota that exploded every 400,000 trips world wide Or bank with a bank that lost all my money every 400,000 transactions world wide

Well, Toyota had the sticking gas pedal issue 10 years ago: they did not implement a brake override when the gas pedal was stuck. This was a recommended feature by European manufacturers when they introduced the electronic throttle, apparently Toyota didn't get the memo.

Although I find the GM ignition key issue way worse than Toyota which was an oversight.

Re: QEMU Internals

#23
post #16
post #5

Earlier quoted context omitted.

Conventionally, one starts from the CHIP-8, which is indeed a virtual machine rather than a system in a strict sense. What I've found difficult is the step beyond that. NES and GameBoy are typical steps, however, I've been very frustrated by the confusing documentation of the GameBoy. There are 3/4 references, but one of them has significant mistakes, while another is incomplete. On the other hand, the Pan Docs shoul…

I built about a third of a NES emulator. The nesdev wiki is mostly decent, although there's a fair number of things where it seems like the first people to figure things out got stuff kind of backwards, and if you flip it, it's a lot easier, that's the sort of fighting the specifications I think you're talking about. All that said, emulating the CPU was pretty fun. There's a CPU test rom out there you can run with tr…

I did a GameBoy and similarly found the CPU enjoyable and the PPU a huge pain. Perhaps if I understood graphics better, I would have enjoyed it more, but like you say it just felt like a lot of steps.

Re: QEMU Internals

#24
post #20
post #18

Earlier quoted context omitted.

That is such a bizarre viewpoint from my perspective. The absolute deathtrap that is the 737 MAX had two software-related critical failures in 400,000 flights. That constitutes a whole system per-flight software reliability of 2 in ~400,000 or a ~99.9995%, 5 9s. Obviously that is still unacceptable as that is far below the software standard amongst all commercial airplanes where software has not been implicated in a…

Apples to oranges? The scale between AWS and 737s is several orders of magnitude different. Boeing has a critical issue every 200k flights, or let's say 3.8M hours of flight time (assuming all flights are 19h, which they are not). Assume AWS has 1M CPUs total (they have way more than that), if AWS saw a critical CPU bug every 3.8M hours of CPU time they would be having a 737 MAX crisis level every 3.8 hours.

One failure per 3.8M hours would be once per 433 CPU-years, so they probably actually do have somewhere between 10-100x that failure rate for their CPUs given that expected CPU lifetime is probably around 20-30 years. Even using a much more reasonable 2 hours per flight that is still ~45 CPU-years so still within the likely range of expected CPU errors. Also that is a comparison against a system so dangerous that it is unfit for use instead of the actual standard which is once per 50,000,000 flights or ~250x better.

Even ignoring that, I am discussing the uptime of a system using AWS which only guarantees 99.99% uptime for AWS service in any given AWS region and only a 10% refund (which is less than their profit margin) as long as they keep your system up more than 99% of the time. Downtime for a system due to AWS downtime in a region constitutes a critical failure of AWS to deliver expected service. That their lack of service does not result in deaths unlike an airplane is immaterial to a reliability analysis, it only tells us if their critical failures matter and what level of reliability we should require/demand when making reliability-cost tradeoffs. In other words, the probability and costs of failure are not actually related. It is just that costly failures result in more effort being spent on developing mitigations. In the case of airplanes, critical failure in the form of a crash is very costly, so they take great pains to minimize the whole-system risk of that failure mode.

Re: QEMU Internals

#25

I don't mean this to disparage Airbus in any way but after Boeing's issues with the 737 MAX I'd assumed a fairly poor culture of software at airplane manufacturers in general. Super glad to see work like this coming out of Airbus, really makes me rethink my earlier assumptions about software competence in the field.

Airbus is known to be excellent in airplane software development.

However, this is probably not about the airplane part of Airbus. Like Boeing, Airbus also have huge defense and space divisions.

Re: QEMU Internals

#26
post #18

Earlier quoted context omitted.

That is such a bizarre viewpoint from my perspective. The absolute deathtrap that is the 737 MAX had two software-related critical failures in 400,000 flights. That constitutes a whole system per-flight software reliability of 2 in ~400,000 or a ~99.9995%, 5 9s. Obviously that is still unacceptable as that is far below the software standard amongst all commercial airplanes where software has not been implicated in a…

That’s an interesting way to look at uptime no pun intended thou I wouldn’t buy a Toyota that exploded every 400,000 trips world wide Or bank with a bank that lost all my money every 400,000 transactions world wide

Indeed, a Toyota with a critical fatality-inducing safety defect every 200,000 trips would be rightfully viewed as a deathtrap. Given that the average trip is probably somewhere around ~30 miles that would be a fatality per 6M miles versus the standard of ~60M miles in the US, or about 10x more dangerous. However, when comparing a car versus airplanes, given that they both fulfill the niche of transportation and are to some degree substitutable, a more reasonable analysis would be fatalities/person-hour or fatalities/person-mile. For fatalities/person-hour the average flight is something like ~2 hours. In the same amount of time 200,000 cars for 2 hours at an average of 40 mph would be ~16M miles, so the 737 MAX is ~4x more dangerous on a person-hour basis than cars. If we go by distance the average flight is ~500 miles, so the 737 MAX had a fatality per 100M person-miles or is ~1.6x safer than driving. That is just how high our standards are with planes that a plane that is viewed as an absolute death machine that is totally unfit for use is safer than its primary alternative for an equivalent distance. A plane that is 100x worse than any other commercial plane is still better than the non-plane alternative on a per-distance basis.

Obviously, this does not excuse their actions as they still made a system at least 100x more dangerous than the standard, but it should give perspective on the difficulty of the problems actually being solved. It is not a bunch of amateurs or below-average engineers who need to adopt basic practices. It is a bunch of highly-skilled professionals developing systems with a level of reliability far beyond what most software developers even think is possible. Even the abysmal processes of the 737 MAX that are far below the standard in the airplane industry would, relative to most software, be very good. It is just that the problems they need to solve are very, very, very hard and very good does not cut it when lives, not data, are at stake.

Re: QEMU Internals

#27
post #5
post #3

Earlier quoted context omitted.

I've always been intrigued by virtual machines and emulation as well. I've always wanted to try and make an emulator of some kind. I don't know much about the internals of VirtualBox, but my suggestion would be to start "easy" with one CPU/Computer System/Game Console and go from there. That's what I finally did with the 6502 and Commodore 64.

Conventionally, one starts from the CHIP-8, which is indeed a virtual machine rather than a system in a strict sense. What I've found difficult is the step beyond that. NES and GameBoy are typical steps, however, I've been very frustrated by the confusing documentation of the GameBoy. There are 3/4 references, but one of them has significant mistakes, while another is incomplete. On the other hand, the Pan Docs shoul…

A subset of CP/M calls is a pretty simple "rest of the system" to implement on top of an 8080/Z80 CPU emulation. (It's a bit of a cheat - like qemu's "Linux user mode emulation" or early version of DOSBox, because you restrict software to interacting with a high-level software interface, there are no lower-level details to aim for fidelity with)

Re: QEMU Internals

#28
post #6
post #2

Thank you. On the same subject can someone recommend a book or any other resource to learn about virtual machine internals? My goal is to try to build a toy clone of VirtualBox/VMWare. So far I have found one -- Virtual Machines by James E. Smith and Ravi Nair.

I work on virtual machines at Google. I usually suggest "Hardware and Software Support for Virtualization" [1] to new team members without a virtualization background. [1] https://www.amazon.com/Hardware-Software-Virtualization-Synt...

This looks like a good read, thanks. I'm curious what your background is. How does one go about getting into that specialty at an org like Google? I've understood that Borg and GKE containers at Google generally always run in a VM. Is this where your work is(platform) or are you more research oriented?

Re: QEMU Internals

#29
post #6

Earlier quoted context omitted.

I work on virtual machines at Google. I usually suggest "Hardware and Software Support for Virtualization" [1] to new team members without a virtualization background. [1] https://www.amazon.com/Hardware-Software-Virtualization-Synt...

This looks like a good read, thanks. I'm curious what your background is. How does one go about getting into that specialty at an org like Google? I've understood that Borg and GKE containers at Google generally always run in a VM. Is this where your work is(platform) or are you more research oriented?

Generally "systems-y software" is my background. I joined Google for a semi-experimental operating system project and from there it was a small jump to virtualization when I decided I was interested in doing something else. I'm definitely on the platform side, but been doing some interesting stuff recently :)

Re: QEMU Internals

#30
post #29

Earlier quoted context omitted.

This looks like a good read, thanks. I'm curious what your background is. How does one go about getting into that specialty at an org like Google? I've understood that Borg and GKE containers at Google generally always run in a VM. Is this where your work is(platform) or are you more research oriented?

Generally "systems-y software" is my background. I joined Google for a semi-experimental operating system project and from there it was a small jump to virtualization when I decided I was interested in doing something else. I'm definitely on the platform side, but been doing some interesting stuff recently :)

Thanks. Do you have any other recommendations for "systems-y software" books or resources you think would be helpful for a people pursuing similar roles?
Post reply on HN