Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Porting Python to GRUB: “Networking without an operating system”

lwn.net

Post reply on HN
Print view View original

41–49 of 49 posts

Re: Porting Python to GRUB: “Networking without an operating system”

#41

Post by niftich »

JoshTriplett wrote: post #14
ThePhysicist wrote: post #11

Earlier quoted context omitted.

Did you modify the CPython binary to run in the BIOS, or did you write a new interpreter from scratch? What was the hardest part of implementing this?

We build CPython's source into BITS, which runs as an EFI binary. (Or as a non-EFI image bootable on a BIOS, but that version doesn't include network support.) We have almost no changes to CPython itself, other than a couple of minor fixes to enable some additional calling conventions in the ctypes module. The hardest part of porting Python in general was porting libffi to support the EFI calling convention, which re…

I apologize in advance for my lack of detailed knowledge about low-level IO, but,

Would a library like libuv/pyuv that provides an async interface to networking and file IO help in this regard?

I understand that your goals were to implement as much of the Python stdlib's APIs so that programs can run unmodified, but could you envision Python applications that are coded against pyuv running on a platform that bridges pyuv calls directly to EFI without the stdlib as an intermediary?

Re: Porting Python to GRUB: “Networking without an operating system”

#42

Post by JoshTriplett »

gvb wrote: post #31
JoshTriplett wrote: post #7

I'm one of the authors of BITS, and the presenter of the PyCon presentation this article discusses. Happy to answer any questions here.

How big is the base image, e.g. printing "hello world" to the console? Is it modular, e.g. if you don't need networking[1] how hard is it to not include networking? [1] I'm using networking as an example package, not knowing what other semi-optional packages could be used as an example.

> How big is the base image, e.g. printing "hello world" to the console?

46MB. However, that image includes three separate complete copies of all the compiled code (4.5-6MB each), so that it simultaneously boots on 32-bit BIOS, 32-bit EFI, and 64-bit EFI. It also includes the complete source code (24MB) needed to reproduce the image, to make it self-contained and redistributable without needing to have a separate distribution of source. And it includes .py files for all the BITS .pyc files (other than the Python standard library), to simplify debugging and hacking. A quick check suggests that you could pretty easily build an 8.4MB image, if you cared about the size.

> Is it modular, e.g. if you don't need networking[1] how hard is it to not include networking?

No. We haven't needed a smaller image, so we haven't worked on that. Typically, we boot it from a USB drive or hard drive, so storage space isn't a concern.

Networking, though, consists almost exclusively of Python code, and the compiled .pyc files for that don't take up much space; not a good target to start with if you wanted to reduce size.

Re: Porting Python to GRUB: “Networking without an operating system”

#43

Post by JoshTriplett »

niftich wrote: post #41
JoshTriplett wrote: post #14

Earlier quoted context omitted.

We build CPython's source into BITS, which runs as an EFI binary. (Or as a non-EFI image bootable on a BIOS, but that version doesn't include network support.) We have almost no changes to CPython itself, other than a couple of minor fixes to enable some additional calling conventions in the ctypes module. The hardest part of porting Python in general was porting libffi to support the EFI calling convention, which re…

I apologize in advance for my lack of detailed knowledge about low-level IO, but, Would a library like libuv/pyuv that provides an async interface to networking and file IO help in this regard? I understand that your goals were to implement as much of the Python stdlib's APIs so that programs can run unmodified, but could you envision Python applications that are coded against pyuv running on a platform that bridges…

> Would a library like libuv/pyuv that provides an async interface to networking and file IO help in this regard?

No, it wouldn't. libuv depends on OS facilities like select, poll, or epoll, and then builds a high-performance event loop on top of them. We needed to implement those facilities themselves.

In theory, if we didn't care about Python's standard library modules, we could potentially port libuv to support the asynchronous callback mechanism used by EFI. However, we also wanted to write as much of this code as possible in Python, rather than C, to make it easier to maintain. That's why, for instance, we didn't implement the Berkeley sockets API in C to use the CPython socketmodule.c, but instead implemented our own _socket.py in Python directly on top of the efi module.

Re: Porting Python to GRUB: “Networking without an operating system”

#44

Post by JoshTriplett »

djsumdog wrote: post #37
JoshTriplett wrote: post #18

Earlier quoted context omitted.

Originally, it was because we had a lot of one-off test cases not written in Python. Before BITS, most BIOS test programs consisted of one-off DOS programs, or later one-off EFI programs. If you wanted a new test, or you wanted a customer to quickly gather some information for you, you sent them a new test program, or a bootable image. Now, many of those one-off tests have become Python one-liners, and you can use Py…

Wow. I thought this was one of those "just because you can," projects, but it looks like there are some real practical uses for firmware developers/testers.

Yeah, we wrote BITS almost entirely for practical reasons. Several groups at Intel use it to test both the BIOS and the CPU itself. (You have to have enough functionality to boot an OS, but once you have that, BITS can easily test things like CPU power management or MSR configuration.)

The module that displays presentation slides using EFI_GRAPHICS_OUTPUT_PROTOCOL, on the other hand, was definitely "because we could". https://www.youtube.com/watch?v=bYQ_lq5dcvM#t=22m56s

Re: Porting Python to GRUB: “Networking without an operating system”

#45

Post by JoshTriplett »

niftich wrote: post #40
JoshTriplett wrote: post #7

I'm one of the authors of BITS, and the presenter of the PyCon presentation this article discusses. Happy to answer any questions here.

Awesome stuff! What are your future plans in this effort? Could you envision a (G)UI toolkit written in Python that could be used to write applications that run in EFI?

> Awesome stuff! What are your future plans in this effort?

On a small scale, moving as much as possible from C to Python using ctypes. Much of the C code pre-dates us implementing ctypes support, so it provides bindings to things like ACPI via the Python C API. We'd like to re-write that using ctypes, which will result in far less C code and no manual management of things like reference counts of Python objects.

On a larger scale, we're outgrowing GRUB's menu system. It's been incredibly helpful to get started, but we'd like a more usable and flexible UI than GRUB menus based on dynamically generated GRUB cfg files. (Several of the menus consist of lines like "source (python)/test.cfg", where "(python)" is an in-memory GRUB filesystem implemented in Python, and Python code enumerates the test hierarchy at boot time to generate test.cfg.) In particular, we'd like to be able to have simple UI elements to select one of a set of configuration options and display the current value directly in the menu.

So yes, I can envision a UI toolkit written in Python. :)

Re: Porting Python to GRUB: “Networking without an operating system”

#46

Post by mwcampbell »

JoshTriplett wrote: post #6
sandGorgon wrote: post #2

why would they not use lua ? which is pretty popular (and pretty much designed) as embedded scripting languages. Not to mention much more lightweight than python.. and has a much smaller footprint. Nginx is a pretty good example of this.

> why would they not use lua ? The target audience for this is BIOS developers and firmware engineers. Several existing development tools in that area already support Python, including Simics and the Intel ITP, so the target audience is used to it. Lua also has several other impedance mismatches with the target audience, such as 1-based array indexing (remember, the target audience writes assembly and C). While peopl…

> I much prefer the Python C API.

Your other reasons for going with Python make sense, but this one surprises me. Isn't the Python C API's reference counting tedious? I've written bindings to C libraries for both Python and Lua. With Python, I always use something higher level than the Python C API, like Cython or ctypes. With Lua, I've cranked out lots of bindings using the Lua C Api directly.

Re: Porting Python to GRUB: “Networking without an operating system”

#47

Post by JoshTriplett »

mwcampbell wrote: post #46
JoshTriplett wrote: post #6

Earlier quoted context omitted.

> why would they not use lua ? The target audience for this is BIOS developers and firmware engineers. Several existing development tools in that area already support Python, including Simics and the Intel ITP, so the target audience is used to it. Lua also has several other impedance mismatches with the target audience, such as 1-based array indexing (remember, the target audience writes assembly and C). While peopl…

> I much prefer the Python C API. Your other reasons for going with Python make sense, but this one surprises me. Isn't the Python C API's reference counting tedious? I've written bindings to C libraries for both Python and Lua. With Python, I always use something higher level than the Python C API, like Cython or ctypes. With Lua, I've cranked out lots of bindings using the Lua C Api directly.

The reference counting is annoying, yes; however, it's the cost of allowing C to hold references to Python objects.

I find a single call to PyArgs_ParseTuple at the top of a Python C entry point much nicer than referencing various values off the Lua stack as they move around.

That said, we're in the process of rewriting as many of those functions as possible from C into Python itself, using ctypes.

Re: Porting Python to GRUB: “Networking without an operating system”

#48

Post by yitchelle »

JoshTriplett wrote: post #18
yitchelle wrote: post #17

Earlier quoted context omitted.

I am always interested to hear when a language is ported to a new plaform, in this case Python (or a subset of) on GRUB. The question would be why? Is this because you have lots of test cases already in Python and can easily deployed those test cases on using this?

Originally, it was because we had a lot of one-off test cases not written in Python. Before BITS, most BIOS test programs consisted of one-off DOS programs, or later one-off EFI programs. If you wanted a new test, or you wanted a customer to quickly gather some information for you, you sent them a new test program, or a bootable image. Now, many of those one-off tests have become Python one-liners, and you can use Py…

Thanks for the detail answer.

Does your work have a high dependencies on Intel CPUs? ie, have your heard of it being used on other CPU architecture, eg ARM or PowerPC?

Re: Porting Python to GRUB: “Networking without an operating system”

#49

Post by JoshTriplett »

yitchelle wrote: post #48
JoshTriplett wrote: post #18

Earlier quoted context omitted.

Originally, it was because we had a lot of one-off test cases not written in Python. Before BITS, most BIOS test programs consisted of one-off DOS programs, or later one-off EFI programs. If you wanted a new test, or you wanted a customer to quickly gather some information for you, you sent them a new test program, or a bootable image. Now, many of those one-off tests have become Python one-liners, and you can use Py…

Thanks for the detail answer. Does your work have a high dependencies on Intel CPUs? ie, have your heard of it being used on other CPU architecture, eg ARM or PowerPC?

It could be ported to ARM systems running EFI with some effort, if someone wanted to do so. It wouldn't make much sense to port to a completely different BIOS architecture such as those on PowerPC, though.
Post reply on HN