Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DLL that was not present in memory despite not being formally unloaded

devblogs.microsoft.com

Post reply on HN
Print view View original

41–49 of 49 posts

Re: DLL that was not present in memory despite not being formally unloaded

#41

Post by xmodem »

rwmj wrote: post #17

What MSFT support policy do you need to have the legendary Raymond Chen take a look at it? I say this because we've reported a bunch of Windows bugs (mainly running Windows under virtualization) and getting them to pay attention at all is an up-hill battle.

If you can reproduce it reliably and doing so generates some form of telemetry, then just set up some automation to keep doing that in a loop. From as many machines as possible.

No comment is offered on if I have ever gotten a bug noticed this way.

Re: DLL that was not present in memory despite not being formally unloaded

#42

Post by toast0 »

Chu4eeno wrote: post #38
FartyMcFarter wrote: post #20

Earlier quoted context omitted.

As someone who has debugged his fair share of tricky low-level issues, the parts that I find impressive in his blog posts are things such as "then we look at the bytes in memory and oh yeah, this looks like an exception record". I would usually not think to do that (or be able to recognise it as easily as I presume he did).

I assume it's mostly just something you learn to recognize after decades of poking at the same things. I remember being impressed with Thiago (Qt developer) being able to immediately tell if a pointer was heap allocated, invalid/unaligned, etc. until I spent more time poring over /proc/*/maps and in gdb. Never figured out how he could tell someone's Qt version just from an strace excerpt, though.

> Never figured out how he could tell someone's Qt version just from an strace excerpt, though.

Sonames might be a big clue? Otherwise, initialization order changes maybe? Sometimes there's enough file content in an strace to be able to see a strong indicator?

Those are just guesses, I do a lot more debugging with pcaps rather than straces. Although you do often want to determine which side of the syscall caused whatever you're seeing in the pcap.

Re: DLL that was not present in memory despite not being formally unloaded

#43

Post by garaetjjte »

londons_explore wrote: post #22
Someone1234 wrote: post #8

Earlier quoted context omitted.

Part 1 was interesting; it isn't clear why he split that into a Part 2 since it adds little to the story and is a paragraph long.

I assume the fact it is a third party application means debugging gets harder, and the business case for doing so is weaker/none. But I would hope that some kind of reverse debugger triggered on one of these crashes would make it pretty simple to say "who wrote this 01".

You usually hope that TTD points to the culprit in such situations. But once I encountered single-byte corruption that didn't make any sense in TTD trace, there was good value at write and next read was garbage. I never discovered whether that was CPU bug, corruption by GPU shaders, stray kernel writes, or whatever.(I think it's unlikely that CPU bug would manifest with both native and TTD-instrumented runs. Corrupted byte was inside heap allocated memory so it shouldn't be in GPU pagetables at all. Kernel writes wouldn't appear in TTD trace, so really I think that was most likely issue, but how to debug that...)

Re: DLL that was not present in memory despite not being formally unloaded

#44

Post by algorithmsRcool »

forestry wrote: post #34
hackrmn wrote: post #27

The fact that Raymond Chen is debugging these kind of issues, tells me Microsoft is short on staff that has his particular set of skills, handing him the hairiest issues from the annals of Windows. The new hires are probably all about .NET and JavaScript and what have you -- whatever Microsoft is about these days. I doubt it's C/C++. Chen is probably on standby and is paid handsomely as a de-facto VIP consultant. He…

Managed dump analysis in windbg was a thing. It’s been many years since I’ve needed it, though. Service telemetry improved quickly thereafter.

It is still a thing, just not a very common one since the debugger in VS has become more ergonomic and powerful. But windbg is still the king here, for the most advanced analysis of both managed and unmanaged code and it isn't even close to be honest once you learn the arcane commands and incatations

Re: DLL that was not present in memory despite not being formally unloaded

#45

Post by kjellsbells »

hackrmn wrote: post #27

The fact that Raymond Chen is debugging these kind of issues, tells me Microsoft is short on staff that has his particular set of skills, handing him the hairiest issues from the annals of Windows. The new hires are probably all about .NET and JavaScript and what have you -- whatever Microsoft is about these days. I doubt it's C/C++. Chen is probably on standby and is paid handsomely as a de-facto VIP consultant. He…

I do wonder how Microsoft will manage the transition of the NT generation. Raymond Chen has been doing this kind of work for thirty years. He probably has, what, another ten years, max? Who are the next generation of Windows gurus that will take up the mantle?

Hanselman is good on the blog part, but not in Chen's class re Windows domain expertise. And Windows is not, despite all that is said, going away anytime soon. I think this could be a real problem.

Hopefully there is a set of 25 year old developers in the Windows team who have deep and growing skills in Winternals, and Microsoft have the good sense to encourage them in their career.

Re: DLL that was not present in memory despite not being formally unloaded

#46

Post by jamesfinlayson »

taneq wrote: post #14
brookst wrote: post #9

Earlier quoted context omitted.

When you’ve eliminated all possible explanation, it’s time to pack it in.

Oh man, my journey from idealistic “there is always an explanation” youth to “some days it do be like that, and we may never know why” in a nutshell.

Yeah I'm nearly 15 years into my career and still once or twice a year I have a moment where I think - there is no way that should ever have worked and I really don't know why it ever worked and didn't cause any issues.

Re: DLL that was not present in memory despite not being formally unloaded

#48

Post by nianderwallace »

garaetjjte wrote: post #43
londons_explore wrote: post #22

Earlier quoted context omitted.

I assume the fact it is a third party application means debugging gets harder, and the business case for doing so is weaker/none. But I would hope that some kind of reverse debugger triggered on one of these crashes would make it pretty simple to say "who wrote this 01".

You usually hope that TTD points to the culprit in such situations. But once I encountered single-byte corruption that didn't make any sense in TTD trace, there was good value at write and next read was garbage. I never discovered whether that was CPU bug, corruption by GPU shaders, stray kernel writes, or whatever.(I think it's unlikely that CPU bug would manifest with both native and TTD-instrumented runs. Corrupte…

For specific cases, you'd convert your memory allocator - hopefully you could reduce the need to just certain mem allocs - and write-protect (aka read-only memory) those mem allocs except for the situations where your code is purposely writing to those areas of memory.

Yes, it'll be slow and use p lots of memory pages, unless you can reduce the mem allocs to a certain small set of allocs. And you'll have to have code to write-enable/write-disable those mem allocs. But if it catches the culprit writing bytes where they shouldn't, it'll be worth it.

The one time I did this for a buffer passed to a HW device, I could prove that the hardware was doing DMA-writes where it shouldn't. Had to bring a HW logic analyzer to verify.

Re: DLL that was not present in memory despite not being formally unloaded

#49

Post by hackrmn »

kjellsbells wrote: post #45
hackrmn wrote: post #27

The fact that Raymond Chen is debugging these kind of issues, tells me Microsoft is short on staff that has his particular set of skills, handing him the hairiest issues from the annals of Windows. The new hires are probably all about .NET and JavaScript and what have you -- whatever Microsoft is about these days. I doubt it's C/C++. Chen is probably on standby and is paid handsomely as a de-facto VIP consultant. He…

I do wonder how Microsoft will manage the transition of the NT generation. Raymond Chen has been doing this kind of work for thirty years. He probably has, what, another ten years, max? Who are the next generation of Windows gurus that will take up the mantle? Hanselman is good on the blog part, but not in Chen's class re Windows domain expertise. And Windows is not, despite all that is said, going away anytime soon.…

These were my thoughts more or less, but how many Chens does the world's largest software company need to at least maintain the now "legacy" code before they find a way not to need them any longer. Maybe they're betting on large language models having trained on the cumulative output of experts like Chen, but I think that's fit for contingency situations -- maintaining existing code and making sure it doesn't break the bottom-line -- but for all the praise of AI I have my doubts Microsoft are crazy enough to just leave it to AI to develop Windows containing hundreds of thousands of SLOC of the kind only Chen can effectively debug.

As for 25 year olds -- I am sure there will always be those around, people have been pretty consistently varied in their pursuits, but it going to be so niche they'll be like COBOL/FORTRAN developers hired by the banks today -- far and in between and paid so handsomely they can pick and choose consultancy any day of the week anywhere in the world. So I guess good for them, but again -- can Microsoft depend on that form of provision of labour they still need (and will need for a few years ahead at least)?

Post reply on HN