The mystery of the fifteen-millisecond breakpoint instruction • Hacker News Board

Re: The mystery of the fifteen-millisecond breakpoint instruction

#41

Post by tjoff » Sat, Sep 05, 2015, 7:51 PM UTC

TheLoneWolfling wrote: post #40

tjoff wrote: post #38

Earlier quoted context omitted.

I'm going to guess that the Pi uses ext3/4 which I'm assuming does pretty well with appending files on a spinning disk. Yes, we need something tailored for flash and the advent of SSDs have put some effort into that. However, the controller of a SD-card is for understandable reasons not as advanced as one on an SSD. One of the issues that seemed prevalent of the Pi was that SD-cards, when issuing a TRIM command, just…

Or you have one filesystem that is geared towards dumb flash, and another that is geared towards smart flash. I don't see why people try to design filesystems that do everything. As usual for hybrids, they try to do everything and as a result don't work well anywhere. As long as you keep the limitations of dumb flash in mind (namely, that flipping bits in one direction is slow (things are a little more complex on MLC…

This is a tangent but you seem to be under the impression that filesystems grow on trees. I've been waiting for a modern COW linux filesystem for almost a decade now. Maybe in a few years it's mature enough to wait a few more years for it to become stable? (I'm looking at you BTRFS) ;)

I guess BTRFS would fall into the filesystems that do everything category... Though at the other side of the spectrum you have filesystems that have not been tested enough or isn't maintained.

Re: The mystery of the fifteen-millisecond breakpoint instruction

#42

Post by comex » Sat, Sep 05, 2015, 9:37 PM UTC

derf_ wrote: post #25

evmar wrote: post #24

He wrote: > Think about the common debugging scenario where the user sets a conditional breakpoint: "break if x > y". Testing that condition is going to take 15ms each time. Wouldn't the implementation of a software breakpoint look like: if x > y: bkpt so it'd only be slow on the case where it needed to break? I guess I can imagine you could implement it the other way (always break, check the condition after breaking…

I don't know what gdb does, but even on x86 a conditional breakpoint can be ridiculously slow. On a loop that executes a few hundred thousand times before meeting the condition, it can take minutes to evaluate something that executed instantly before. I often go back, edit the code to add something like if (condition) { foo = foo; } recompile, restart the debugger, and break on the foo = foo line. Even with all of th…

Incidentally, GDB can sometimes evaluate breakpoint conditions on the target:

https://sourceware.org/gdb/onlinedocs/gdb/Conditions.html (search for "target's side")

I don't know what targets support it, though. (Pretty sure OS X isn't one of them.)

Re: The mystery of the fifteen-millisecond breakpoint instruction

#43

Post by simoncion » Sat, Sep 05, 2015, 10:44 PM UTC

TheLoneWolfling wrote: post #32

tonyg wrote: post #31

Earlier quoted context omitted.

If I understood the article correctly, the ~1ms cost came from having rsyslog running, not from printk. Killing rsyslog led to a ~1ms speedup, but it was removing printk() that led to the rest. Printing to the console on the RPi could involve either or both of the serial port and the software-character-generator-based text-mode video output. Depending on screen resolution, the video output might be quite slow to prin…

In which case, what on Earth is rsyslog doing that it takes 1ms to append a line of text to the log file? I could see it if it was outputting something to software-based video - except that in that case, why is it trying to output to software-based video?

> ...except that in that case, why is it trying to output to software-based video?

I don't have much experience with rsyslog, but syslog-ng can be configured to log to all consoles. [0] I know that consoles are usually configured to blank themselves after a period of inactivity. I wonder if a blanked console can be written to faster than an non-blanked one. [1]

Edit: The Fine Article supports my assumption about how rsyslog is configured on the RPi. Search for "printk". Also remember that -when syslog is disabled- printks are written to the active console.

[0] The default configuration appears to log everything to /dev/tty12. I see no reason why you couldn't log everything to -I guess- /dev/console. (Or, failing that, /dev/tty[0-12].)

[1] I know that screen-scrolling updates that happen in a console that is not the current one (or that happen in a screen window that's not the current one) are substantially faster than on-screen screen-scrolling updates.

Re: The mystery of the fifteen-millisecond breakpoint instruction

#44

Post by simoncion » Sat, Sep 05, 2015, 10:54 PM UTC

TheLoneWolfling wrote: post #40

tjoff wrote: post #38

Earlier quoted context omitted.

I'm going to guess that the Pi uses ext3/4 which I'm assuming does pretty well with appending files on a spinning disk. Yes, we need something tailored for flash and the advent of SSDs have put some effort into that. However, the controller of a SD-card is for understandable reasons not as advanced as one on an SSD. One of the issues that seemed prevalent of the Pi was that SD-cards, when issuing a TRIM command, just…

Or you have one filesystem that is geared towards dumb flash, and another that is geared towards smart flash. I don't see why people try to design filesystems that do everything. As usual for hybrids, they try to do everything and as a result don't work well anywhere. As long as you keep the limitations of dumb flash in mind (namely, that flipping bits in one direction is slow (things are a little more complex on MLC…

FSs designed to run on dumb flash exist: https://encrypted.google.com/#q=site:lwn.net+nand+flash+file...

> I don't see why people try to design filesystems that do everything.

For general use, [0] I would rather use a FS that works on pretty much every media type [1] -and has reasonable perf- than to have to worry about whether $SPECIALTY_FS is actually tuned for $WEIRDO_BLOCK_DEVICE, or if I've failed to understand the particulars of either the FS, device, or both.

[0] That is, when getting the absolute best perf isn't a requirement.

[1] Except for like, tape, and optical RAM/WORM drives, natch.

Re: The mystery of the fifteen-millisecond breakpoint instruction

#45

Post by TheLoneWolfling » Sat, Sep 05, 2015, 11:10 PM UTC

simoncion wrote: post #44

TheLoneWolfling wrote: post #40

Earlier quoted context omitted.

Or you have one filesystem that is geared towards dumb flash, and another that is geared towards smart flash. I don't see why people try to design filesystems that do everything. As usual for hybrids, they try to do everything and as a result don't work well anywhere. As long as you keep the limitations of dumb flash in mind (namely, that flipping bits in one direction is slow (things are a little more complex on MLC…

FSs designed to run on dumb flash exist: https://encrypted.google.com/#q=site:lwn.net+nand+flash+file... > I don't see why people try to design filesystems that do everything. For general use, [0] I would rather use a FS that works on pretty much every media type [1] -and has reasonable perf- than to have to worry about whether $SPECIALTY_FS is actually tuned for $WEIRDO_BLOCK_DEVICE, or if I've failed to understand…

I don't mind having a filesystem that works reasonably well on most devices. But I would consider "being able to append to a file reasonably quickly" as part of that "reasonably well".

Note that it's entirely possible to have a filesystem that's actually several different filesystems with a magic selector. (I mean, block sizes and reserved block percentages and inline inodes already are along those lines)

Re: The mystery of the fifteen-millisecond breakpoint instruction

#46

Post by simoncion » Sat, Sep 05, 2015, 11:33 PM UTC

TheLoneWolfling wrote: post #45

simoncion wrote: post #44

Earlier quoted context omitted.

FSs designed to run on dumb flash exist: https://encrypted.google.com/#q=site:lwn.net+nand+flash+file... > I don't see why people try to design filesystems that do everything. For general use, [0] I would rather use a FS that works on pretty much every media type [1] -and has reasonable perf- than to have to worry about whether $SPECIALTY_FS is actually tuned for $WEIRDO_BLOCK_DEVICE, or if I've failed to understand…

I don't mind having a filesystem that works reasonably well on most devices. But I would consider "being able to append to a file reasonably quickly" as part of that "reasonably well". Note that it's entirely possible to have a filesystem that's actually several different filesystems with a magic selector. (I mean, block sizes and reserved block percentages and inline inodes already are along those lines)

If your underlying device is slow -as SD cards are notorious for being for any use case other than picture or video storage and retrieval-, no amount of FS juju can help you.

Re: The mystery of the fifteen-millisecond breakpoint instruction

#47

Post by spc476 » Sun, Sep 06, 2015, 1:26 AM UTC

derf_ wrote: post #25

evmar wrote: post #24

He wrote: > Think about the common debugging scenario where the user sets a conditional breakpoint: "break if x > y". Testing that condition is going to take 15ms each time. Wouldn't the implementation of a software breakpoint look like: if x > y: bkpt so it'd only be slow on the case where it needed to break? I guess I can imagine you could implement it the other way (always break, check the condition after breaking…

I don't know what gdb does, but even on x86 a conditional breakpoint can be ridiculously slow. On a loop that executes a few hundred thousand times before meeting the condition, it can take minutes to evaluate something that executed instantly before. I often go back, edit the code to add something like if (condition) { foo = foo; } recompile, restart the debugger, and break on the foo = foo line. Even with all of th…

Worse case scenario---the CPU only supports a breakpoint instruction. Doing a conditional breakpoint involves the debugger setting a breakpoint instruction at the next instruction, resuming, then resetting the breakpoint and setting a breakpoint at the next instruction (with special care taken for calls, jumps and conditional jumps) and doing this, for every instruction until the condition is met.

Second worst case---the CPU has both breakpoints and single stepping mode (the 8086 supports this). In this case, the debugger no longer needs to set a breakpoint on every instruction, but the CPU still endures an interrupt per instruction until the condition is met.

Best case---the CPU has a way to specify a condition in hardware (the 80386 and above can do this to varying degrees) but often you have a limited number of conditions that can be handled and if you exceed this, then you either have to disallow it, or fall back to one of the previous methods (or, perhaps depending upon the condition, you can play around with page settings to capture reads or writes but that has its own overhead).

Re: The mystery of the fifteen-millisecond breakpoint instruction

#48

Post by ars » Sun, Sep 06, 2015, 2:02 AM UTC

TheLoneWolfling wrote: post #35

JoachimSchipper wrote: post #33

Earlier quoted context omitted.

You need quite a few kernel userland switches in this case, maybe that's enough?

Potentially, although in that case it's still dsyslog's fault.

It's classed as an ALERT level message, so rsyslog will sync after outputting it. That might account for the time.

Re: The mystery of the fifteen-millisecond breakpoint instruction

#49

Post by lultimouomo » Sun, Sep 06, 2015, 11:43 AM UTC

madez wrote: post #9

userbinator wrote: post #7

Earlier quoted context omitted.

I meant that you can already do these if you want to consume a lot of CPU time: for(;;) printf("Hello world!"); for(;;) ;

But that won't slow down the whole system, if you didn't mess with the process priorities.

sync() will most definitely do.

Re: The mystery of the fifteen-millisecond breakpoint instruction

#50

Post by anon4 » Sun, Sep 06, 2015, 8:48 PM UTC

moyix wrote: post #28

evmar wrote: post #24

He wrote: > Think about the common debugging scenario where the user sets a conditional breakpoint: "break if x > y". Testing that condition is going to take 15ms each time. Wouldn't the implementation of a software breakpoint look like: if x > y: bkpt so it'd only be slow on the case where it needed to break? I guess I can imagine you could implement it the other way (always break, check the condition after breaking…

Typically the condition testing for a conditional is done inside the debugger, not inside the patched code. The reason for this is that the conditions can be quite complex, and require debug symbol resolution. And this takes up a lot of space. The issue of space is actually a bit subtle. Imagine you have something like (assuming 4 byte fixed width instructions, which is the case on ARM): 0000 instruction 1 0004 instr…

The very latest versions of gdb actually can use gcc to compile code and allow you to hotpatch code into the executable. https://sourceware.org/gdb/current/onlinedocs/gdb/Compiling-... so I'm guessing maybe in the future this can be used for faster conditional breakpoints