Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bug #915: Please help

nedbatchelder.com

Post reply on HN
Print view View original

81–90 of 97 posts

Re: Bug #915: Please help

#81

Post by lostmsu »

mbreese wrote: post #61
lostmsu wrote: post #42

Earlier quoted context omitted.

Does anybody know if this can be done in Windows (not WSL)? E.g. is it possible to open a named pipe (?) through its name, and work with it as a sequential file?

This isn’t a Linux thing, it’s a bash thing. The A quick search says that Windows has support for named pipes as an IPC mechanism [1], but I’ve never used them. But this is a Windows API. I am not aware for any way to do this with plain cmd.exe. You’re not the first to ask though... [2] [1] https://docs.microsoft.com/en-us/windows/win32/ipc/named-pip... [2] https://superuser.com/questions/430466/in-windows-can-i-redi…

Are you sure? How does diff know to read from those two pipes?

Even the echo example hints, that the created pipe has a name, and that name is then simply passed as the argument to the command.

Re: Bug #915: Please help

#82

Post by avip »

nneonneo wrote: post #69
marcan_42 wrote: post #66

Earlier quoted context omitted.

That would do it. Calling _os.close() on an fd that is now owned by a file object is a big no-no. Both pieces of code are broken. "except BaseException" is wrong because it'll catch KeyboardInterrupt and such, after the file object exists. The mock is wrong because it's introducing an exception that the original code can never raise and was written to assume wouldn't be raised, thus turning a corner case rare bug (fd…

It's still like that in CPython: https://github.com/python/cpython/blob/master/Lib/tempfile.p... If there's a strong enough case that this is broken (and the KeyboardInterrupt example is a good one), I think we should file a bug about it. EDIT: Created issue39318: https://bugs.python.org/issue39318 . I think the possibility of this failing due to a KeyboardInterrupt, even if remote, is sufficient to consider this a b…

Impressive debugging expertise!

relevant history:

https://bugs.python.org/issue21058

https://bugs.python.org/issue26385

Re: Bug #915: Please help

#83

Post by thaumasiotes »

lostmsu wrote: post #81
mbreese wrote: post #61

Earlier quoted context omitted.

This isn’t a Linux thing, it’s a bash thing. The A quick search says that Windows has support for named pipes as an IPC mechanism [1], but I’ve never used them. But this is a Windows API. I am not aware for any way to do this with plain cmd.exe. You’re not the first to ask though... [2] [1] https://docs.microsoft.com/en-us/windows/win32/ipc/named-pip... [2] https://superuser.com/questions/430466/in-windows-can-i-redi…

Are you sure? How does diff know to read from those two pipes? Even the echo example hints, that the created pipe has a name, and that name is then simply passed as the argument to the command.

> Even the echo example hints, that the created pipe has a name

Well, the echo example shows that the pipe is reified at /dev/fd/63 (or /dev/fd/62 for the other pipe).

That's not a named pipe. It's a file descriptor identified by the integer that defines it, accessed under the /dev/fd listing of all file descriptors.

So, to sort of answer your questions:

> How does diff know to read from those two pipes?

They're two different pipes. Diff takes two arguments. Each argument is one of the pipes.

> (implied) Is there such a thing as an anonymous pipe, in any context?

Not if you consider /dev/fd/id-of-pipe to be a name. The operating system lists them all there. But usually you would use a named pipe if you wanted to be able to know the name, so that you could find it again if you didn't already have a reference to it. If you jammed your finger into my chest, I would be there, and my physical form would block the passage of your finger analogously to how an anonymous pipe nevertheless exists as an entry (two entries?) in /dev/fd. But that wouldn't tell you my name in the conventional sense, even though it would be a valid and unambiguous way to refer to me.

Re: Bug #915: Please help

#84

Post by albertzeyer »

nneonneo wrote: post #68
maxmcd wrote: post #56

Earlier quoted context omitted.

Very curious about this wonderful magic. How were you able to get the call stack from the strace line?

I basically set up my GDB with commands to stop on a specific pattern of "lseek, then close", and if the pattern isn't met it just automatically continues the program. This is what the gdb script looks like: set height 0 catch syscall close catch syscall read catch syscall lseek disable 1 2 commands 2 disable 1 2 continue end commands 3 if $rdi == 31 enable 1 2 continue else continue end end The lseek catchpoint (3)…

This is very nice and interesting. Does that also work for lldb? Would sth like this also be possible with dtrace?

Can you recommend any good resources to learn more about this? Should I just study the LLDB/GDB references? Or maybe if you think there are not much good resources on this, maybe it would be a good idea if you could write a blog post about this?

Re: Bug #915: Please help

#85

Post by alehander42 »

nneonneo wrote: post #68
maxmcd wrote: post #56

Earlier quoted context omitted.

Very curious about this wonderful magic. How were you able to get the call stack from the strace line?

I basically set up my GDB with commands to stop on a specific pattern of "lseek, then close", and if the pattern isn't met it just automatically continues the program. This is what the gdb script looks like: set height 0 catch syscall close catch syscall read catch syscall lseek disable 1 2 commands 2 disable 1 2 continue end commands 3 if $rdi == 31 enable 1 2 continue else continue end end The lseek catchpoint (3)…

and you can do it in a rr trace on linux I think if its hard to reproduce which sometimes happen with similar programs

Re: Bug #915: Please help

#86

Post by nneonneo »

avip wrote: post #82
nneonneo wrote: post #69

Earlier quoted context omitted.

It's still like that in CPython: https://github.com/python/cpython/blob/master/Lib/tempfile.p... If there's a strong enough case that this is broken (and the KeyboardInterrupt example is a good one), I think we should file a bug about it. EDIT: Created issue39318: https://bugs.python.org/issue39318 . I think the possibility of this failing due to a KeyboardInterrupt, even if remote, is sufficient to consider this a b…

Impressive debugging expertise! relevant history: https://bugs.python.org/issue21058 https://bugs.python.org/issue26385

Wacky: the patch given in the initial submission of issue21058 is correct, but the patch that was actually committed incorrectly placed the wrapper inside the try block.

Re: Bug #915: Please help

#87

Post by nneonneo »

rustybolt wrote: post #79
nneonneo wrote: post #48

This doesn't quite get all the way there, but the failure is being caused by the closure of a _io.FileIO due to GC, which happens to have the same fd as the sqlite database. The closure happens in the middle of a SQLite operation, which causes a subsequent flock() call to fail. strace log from the failure: open("/home/travis/apprise-api/.coverage", O_RDWR|O_CREAT|O_CLOEXEC, 0644) = 31 fstat(31, {st_mode=S_IFREG|0644,…

Nice find. Is there a good way to find out which line of code does a particular system call in the strace log? E.g. how did you find out that the close(31) is caused by the GC of the mock?

See my answer above (https://news.ycombinator.com/item?id=22031581) - the gist is that I set up GDB to break on a syscall matching the pattern of the offending close(31) (without also breaking on the zillions of legitimate close(31) calls generated by SQLite). Then I backtraced at that point and observed that this particular close wasn't from SQLite (which I'd already suspected), and the backtrace had several lines indicating that the close was from the GC finalization of a FileIO object.

The second step was to figure out which FileIO object was being GCed. I ran the program again and broke inside the FileIO __init__ function (which is written in C), conditioned on the fd being 31. From that breakpoint I did a Python backtrace (using the tools from https://github.com/python/cpython/blob/master/Misc/gdbinit) to figure out the Python code that was responsible for the allocation.

The third step was futzing around with the NamedTemporaryFile code until I could determine why it was leaking a file. This took a little while because I did not initially realize that it had been mocked. Once I did, the solution presented itself.

Re: Bug #915: Please help

#88

Post by saagarjha »

albertzeyer wrote: post #84
nneonneo wrote: post #68

Earlier quoted context omitted.

I basically set up my GDB with commands to stop on a specific pattern of "lseek, then close", and if the pattern isn't met it just automatically continues the program. This is what the gdb script looks like: set height 0 catch syscall close catch syscall read catch syscall lseek disable 1 2 commands 2 disable 1 2 continue end commands 3 if $rdi == 31 enable 1 2 continue else continue end end The lseek catchpoint (3)…

This is very nice and interesting. Does that also work for lldb? Would sth like this also be possible with dtrace? Can you recommend any good resources to learn more about this? Should I just study the LLDB/GDB references? Or maybe if you think there are not much good resources on this, maybe it would be a good idea if you could write a blog post about this?

> Does that also work for lldb?

LLDB doesn't appear to have a way to set a breakpoint at syscalls, unfortunately. If it's dynamically linked, however, it will probably go through your platform's libc so you can probably achieve the same effect by breaking on all of the syscall wrapper functions.

Re: Bug #915: Please help

#89

Post by saagarjha »

jnwatson wrote: post #77
nneonneo wrote: post #68

Earlier quoted context omitted.

I basically set up my GDB with commands to stop on a specific pattern of "lseek, then close", and if the pattern isn't met it just automatically continues the program. This is what the gdb script looks like: set height 0 catch syscall close catch syscall read catch syscall lseek disable 1 2 commands 2 disable 1 2 continue end commands 3 if $rdi == 31 enable 1 2 continue else continue end end The lseek catchpoint (3)…

This is fantastic. I thought I was pretty good at gdb, but this is next level (and it isn't complicated). I've never seen any resource on "how to gdb a really gnarly bug"; this is a great example.

Debuggers are great! They're printf debugging, except that your printf can respond to the program state almost arbitrarily at runtime; not only that, they can modify the program state and can do so from a lot more lenient environment than your compiler would usually allow you to do. An advanced user can end up doing more programming in the debugger than outside of it.

Re: Bug #915: Please help

#90

Post by avemilia »

albertzeyer wrote: post #84
nneonneo wrote: post #68

Earlier quoted context omitted.

I basically set up my GDB with commands to stop on a specific pattern of "lseek, then close", and if the pattern isn't met it just automatically continues the program. This is what the gdb script looks like: set height 0 catch syscall close catch syscall read catch syscall lseek disable 1 2 commands 2 disable 1 2 continue end commands 3 if $rdi == 31 enable 1 2 continue else continue end end The lseek catchpoint (3)…

This is very nice and interesting. Does that also work for lldb? Would sth like this also be possible with dtrace? Can you recommend any good resources to learn more about this? Should I just study the LLDB/GDB references? Or maybe if you think there are not much good resources on this, maybe it would be a good idea if you could write a blog post about this?

I am also interested in this. So far this video [0] from CPPCon'18 is the most complete and fresh I managed to find. Other than that, GDB manual is okay (LLDB's is not complete, as seems to be the case with most LLVM-based tools).

[0] " rel="nofollow">https://youtu.be/V1t6faOKjuQ>

Post reply on HN