Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bug in reader/writer locks in Windows API

old.reddit.com

Post reply on HN
Print view View original

91–100 of 142 posts

Re: Bug in reader/writer locks in Windows API

#91

Post by amluto »

Avamander wrote: post #38
amluto wrote: post #23

Earlier quoted context omitted.

Is there any vendor of actually good email-as-a-service? O365/Exchange/Outlook/Hotmail is a mess. Google has a support problem. Fastmail has an offline email problem. iCloud is not obviously suitable for professional use. What’s left?

There are multiple providers out there that have their own stacks (Google, Microsoft, Tutanota, Protonmail to some extent and others) and there are those that use more common combinations (basically just managed mail-in-a-box). Nothing perfect though, so pick your poison. Some issues are also due to the ecosystem itself. Avoiding POP3 goes a long way for example.

Avoiding POP3 is great. Some people like to say the future is JMAP, but those same people provide such a weak mobile email app that I find it hard to believe. Maybe the protocol can do better than the app by the same vendor?

Re: Bug in reader/writer locks in Windows API

#92

Post by secondcoming »

AndrewStephens wrote: post #85
secondcoming wrote: post #83

Earlier quoted context omitted.

If you have many readers and a single writer there's no point in the readers blocking each other.

True, but in my experience the overhead of std::shared_mutex outweighs the benefits. Other approaches include: * breaking up the lock so different threads can access different parts of your data structure concurrently. * double-buffering (also called ping-pong buffers) where you effectively keep two copies of your data structure. The readers can access one without blocking, a single writer can modify the other and th…

The problem with double-buffering is that you still need to know when all the readers are no longer using one of the copies.

   T1: writer populates copy1
   T2: readers access copy1
   T3: writer populates copy2, and swaps
   T4: writer populates copy1, and swaps
At T4 a reader from T2 could still be accessing the old data structure.

Unless I'm overthinking it.

Re: Bug in reader/writer locks in Windows API

#93

Post by NeillC »

zozbot234 wrote: post #52
davekilian wrote: post #28

I was wondering how something so basic could go unnoticed for so long. Halfway down the page on OP's link, a user u/rbmm provides a compelling answer: that there are (possibly expected?) cases where a thread trying to acquire the lock in shared mode can accidentally get it in exclusive mode instead. This is due to interleaving of atomic bit test-and-[re]set operations between the (shared mode acquire) thread and the…

Why does it have to use bit test and set and interleave with other threads, though. AIUI you can use a CAS loop to implement any RMW atomically over word-sized (or double word sized, on many platforms) data. That seems like a no-brainer. For comparison, the Rust implementation for lightweight RWLocks on futex-capable *nix platforms is here: https://doc.rust-lang.org/stable/src/std/sys/unix/locks/fute... It sets the "…

We use bit test and set as it causes us to get the cache line exclusive. This avoids using the prefetch write before fetching if we were to use CompareExchange. somebody was trying to talk to me about this stuff this morning but I didn't understand him. You can't expect a reader to always be compatible with other readers otherwise you livelock with a constant stream of readers. So we become incompatible. I am unsure if there is something beyond this.

Re: Bug in reader/writer locks in Windows API

#94

Post by NeillC »

tialaramex wrote: post #39
AndrewStephens wrote: post #22

Subtle bugs in Reader/Writer locks do not surprise me. I worked on an in-house implementation based on Win32 (before C++11 and std::shared_mutex) and my recollection is that although the implementation sounds simple it is exceedingly easy to make subtle mistakes. The experience left me with such a bad feeling for shared locks that I tend to avoid them unless absolutely required. When I last tested std::shared_mutex,…

SRWLock seems like a situation where it's small enough for you to construct proofs that what you did is correct, and important enough that the enormous expense of such proof is justified. The C++ std::mutex provided in Microsoft's STL is ludicrously big, whereas SRWLocks are the same size as a pointer. In one sense being ludicrously big is great - less risk the std::mutex will mistakenly share a cache line with your…

Maybe my memory is faulty but I believe it was analyzed by Leslie Lamports stuff. Ofcourse your building a model of how it should work and you might have faults in that.

Re: Bug in reader/writer locks in Windows API

#95

Post by mangamadaiyan »

nightowl_games wrote: post #55
userbinator wrote: post #54

I'm curious if this also occurs in WINE's implementation. I also want to test this on my highly customised XP install which has been patched to add the SRW API among other extensions, and where I had also patched the kernel to fix a race condition causing a deadlock in the keyed event API that the SRW implementation is based on (maybe it's this same one, although in Vista+ they changed it significantly; but the same…

How did you patch the kernel? Like how is that possible?

Here's an example showing how to patch a userspace binary:

http://www.malsmith.net/blog/patching-closed-software/

Note: not my blog. (Edit: removed a probably unnecessary, and likely inaccurate, detail).

Patching the kernel would involve a similar (but slightly more complicated) process.

Re: Bug in reader/writer locks in Windows API

#96

Post by mangamadaiyan »

userbinator wrote: post #64
nightowl_games wrote: post #55

Earlier quoted context omitted.

How did you patch the kernel? Like how is that possible?

With a hex editor, debugger, and skills that most developers these days seem to lack. I patched the kernel in memory first, using a kernel debugger, to verify my fix worked before editing the file on disk.

Windbg or SoftICE? :)

Re: Bug in reader/writer locks in Windows API

#97

Post by indrora »

1over137 wrote: post #7
mastax wrote: post #2

I understand why this is the case but it’s also extremely frustrating: > It is extremely difficult for programmer-users to report bugs against the Windows API (we're supposed to direct you to Feedback Hub, but you may as well transmit your message into deep space). I've filed OS-49268777 "SRWLOCK can deadlock after an exclusive owner has released ownership and several reader threads are attempting to acquire shared o…

Nice to know that it's not just Apple to whom reporting bugs in hopeless. :)

Former AWS here.

My literal job for the last part of my time at AWS was "help triage bugs in the AWS SDK." This is by far the best repro I've ever seen for such an in-depth event.

Most of the tickets you get in open ticket trackers are incomplete [ https://github.com/boto/boto3/issues/4011 ] nonsensical [ https://github.com/boto/boto3/issues/4018 ] or weird [ https://github.com/boto/boto3/issues/358 ].

Re: Bug in reader/writer locks in Windows API

#98

Post by AndrewStephens »

secondcoming wrote: post #92
AndrewStephens wrote: post #85

Earlier quoted context omitted.

True, but in my experience the overhead of std::shared_mutex outweighs the benefits. Other approaches include: * breaking up the lock so different threads can access different parts of your data structure concurrently. * double-buffering (also called ping-pong buffers) where you effectively keep two copies of your data structure. The readers can access one without blocking, a single writer can modify the other and th…

The problem with double-buffering is that you still need to know when all the readers are no longer using one of the copies. T1: writer populates copy1 T2: readers access copy1 T3: writer populates copy2, and swaps T4: writer populates copy1, and swaps At T4 a reader from T2 could still be accessing the old data structure. Unless I'm overthinking it.

No, you are right. One solution is that the readers access the buffer through a shared_ptr and can hold onto the old version for as long as they need it while the writer makes changes and creates a wholly new data structure. It is also possible for the writer to block new readers while doing the swap. Tradeoffs everywhere, depending on if you need readers to see changes that occurred after they started accessing the data.

Re: Bug in reader/writer locks in Windows API

#99

Post by to11mtm »

secondcoming wrote: post #92
AndrewStephens wrote: post #85

Earlier quoted context omitted.

True, but in my experience the overhead of std::shared_mutex outweighs the benefits. Other approaches include: * breaking up the lock so different threads can access different parts of your data structure concurrently. * double-buffering (also called ping-pong buffers) where you effectively keep two copies of your data structure. The readers can access one without blocking, a single writer can modify the other and th…

The problem with double-buffering is that you still need to know when all the readers are no longer using one of the copies. T1: writer populates copy1 T2: readers access copy1 T3: writer populates copy2, and swaps T4: writer populates copy1, and swaps At T4 a reader from T2 could still be accessing the old data structure. Unless I'm overthinking it.

Nope, Ironically ran into this category of issue today with some buffer-reuse in a multi-threaded system.

Re: Bug in reader/writer locks in Windows API

#100

Post by tom_ »

cesarb wrote: post #84
ww520 wrote: post #81

Earlier quoted context omitted.

Based on the Windows CreateThread API [1], it doesn't say anything about memory synchronization guarantee. Does it do internally? [1] https://learn.microsoft.com/en-us/windows/win32/api/processt...

That MSDN documentation is unfortunately silent on this, but the example in the documentation (at https://learn.microsoft.com/en-us/windows/win32/procthread/c... ) only makes sense if the operating system guarantees the ordering. The C++ standard (at least a draft of it I found on a quick web search) is more explicit: it says ( https://eel.is/c++draft/thread.thread.constr ) "The completion of the invocation of the co…

Any time somebody tells you how simple C is by comparison, point them at https://port70.net/~nsz/c/c11/n1570.html#6.2.4p5:

> An object whose identifier is declared with no linkage and without the storage-class specifier static has automatic storage duration, as do some compound literals. The result of attempting to indirectly access an object with automatic storage duration from a thread other than the one with which the object is associated is implementation-defined

(I bet in practice almost all implementations behave as an equivalent C++ would, as per your notes, so only the ordering is relevant. But people maintaining C implementations have on occasion shown themselves to be their users' enemies, so don't quote me on this!)

Post reply on HN