Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Windows malloc() implementation from MSVCRT is slow

erikmcclure.com

Post reply on HN
Print view View original

81–90 of 187 posts

Re: The Windows malloc() implementation from MSVCRT is slow

#81

Post by FreakLegion »

leajkinUnk wrote: post #68
FreakLegion wrote: post #35

Earlier quoted context omitted.

There's also UCRT, which ships with the OS since Windows 10. The logic of this rant was a real head-scratcher. If you must blame one side, it's LLVM. Fragmentation of C runtimes is annoying but inescapable. Glibc for example isn't any better.

Could you elaborate why Glibc isn't any better? I remember some funny problems with Glibc, like, 20 years ago, but it's been invisible to me (as a user) since then. You get a new Glibc, old binaries still work, it's fine.

Just like with Windows the challenges affect developers rather than users.

> You get a new Glibc, old binaries still work, it's fine.

Indeed, but when you need to build for an older glibc it's not so simple. This is a common use case, since e.g. AWS's environments are on glibc 2.26.

Ideally you'd like to build for all targets, including older systems, from a single, modern environment (this is trivial in Windows) -- and you can do some gymnastics to make that happen[1] -- but in practice it's easier to just manage different build environments for different targets. This is partly why building Linux wheels is so convoluted for Python[2].

Hardly a world-ending problem, but my point is simply that C runtimes are a pain everywhere.

1. https://stackoverflow.com/questions/2856438/how-can-i-link-t...

2. https://github.com/pypa/manylinux

Re: The Windows malloc() implementation from MSVCRT is slow

#82

Post by stagger87 »

lkfjasdlkjfsad wrote: post #59
sterlind wrote: post #37

Earlier quoted context omitted.

(I work for MS, though in core Azure rather than Office or Windows.) I think that PowerShell story was how old MS worked, back in the days of stack ranking, hatred of Linux and the Longhorn fiasco. things inside the company are a lot more functional now. I saw internal politics drama at my first position, but once I moved everything was chill, and experimentation and contributing across team boundaries was actively e…

any idea what the hell is going on with Teams? why can't i simply scroll up in my own conversations? let alone search them. the sticky sludge of communication in something as simple as chat has cost me hours since i was forced to use teams. outlook search is so superior to teams i'd easily prefer to have lync back. this one thing absolutely cripples communication. there are a list of other very basic issues that make…

PEBKAC, you can scroll and search chats in teams.

Re: The Windows malloc() implementation from MSVCRT is slow

#83

Post by jrockway »

londons_explore wrote: post #53
sterlind wrote: post #40

Earlier quoted context omitted.

! I disagree. CMD badly needed replacing. MS needed a new shell language. A functional company would connect people with a passion for X with the resources to achieve X, if X has a chance of helping the company. Windows Terminal and WSL show how far MS has come from the PS days. (Disclaimer: I work for MS)

I think the smart move would have been to make an official port of bash...

Some experimentation in that space is probably a good thing. Bash is familiar, but it's far from perfect. In terms of executing programs, it gets the critical functionality down pretty well; readline-based editing, variables, aliases, pipes, IO redirects, background process management, etc.

In terms of being a programming language, I always regret when I use it. I recently had a minor heart attack when I realized that our CI was green when tests were failing; deep down in some shell script, someone forgot to turn on "pipefail" and the shell linter to check for that was misconfigured, and so the CI command that piped output to a logfile failed, but the log file was written OK, so "exit 0".

In terms of interactive poking around at a computer, I never really liked the UNIX philosophy here. I run a terminal emulator, which connects to a remote server over SSH, which is running a terminal multiplexer, which is running bash, which then runs my programs. None of these things know anything about each other. The UNIX way!!! The end result is total jank and some useful features are completely impossible to implement. The various shells running under the multiplexer overwrite each other's history. The terminal multiplexer can't give the terminal emulator its scrollback buffer. The shell history is specific to the machine that the shell is running on, not the machine that the terminal is running on. Echoing the character you just typed involves two TCP round trips! It's so bad, guys.

For that reason, I totally see the desire to re-engineer this space. There are a lot of improvements to be mad. Powershell is an interesting attempt. It doesn't solve any of my problems, though, and I personally don't enjoy using it. It's verbose and unergonomic, and still not a good programming language for making stuff happen. Windows is missing the glue ecosystem of things like "grep", "sed", "curl", etc., which make matters worse. (Powershell provides some of those things as cmdlets, but the "curl" one opens up IE to make you click something, and weird stuff like that.) It's nice that someone tried to make a new thing. I personally think it's worse than everything else out there.

TL;DR: "we've always used bash" leaves a lot to be desired. It's fine. But if someone says they can do better, I completely agree.

Re: The Windows malloc() implementation from MSVCRT is slow

#84

Post by mananaysiempre »

FreakLegion wrote: post #35
TonyTrapp wrote: post #7

Not that it helps here, but Microsoft never considered the MSVCRT that ships with Windows to be public API. This is not the "Windows allocator", this is the (very) old MSVC runtime library's allocator. Of course that doesn't keep anyone from using this library because it's present on any Windows system, unlike the newer MSVC versions' runtime library. Using the allocator from a later MSVC's runtime library would prov…

There's also UCRT, which ships with the OS since Windows 10. The logic of this rant was a real head-scratcher. If you must blame one side, it's LLVM. Fragmentation of C runtimes is annoying but inescapable. Glibc for example isn't any better.

> Fragmentation of C runtimes is annoying but inescapable. Glibc for example isn't any better.

Glibc very much is better.

There cannot be more than a single version of a (tightly coupled) ld.so+libc.so pair in a given address space any more than there can be more than a single KERNEL32 version in a given address space, and given that some system services are exclusively accessible via dynamic linking (accelerated graphics, for one), this forces essentially everything on a conventional Linux desktop to use a single version of a single C runtime (whether Glibc or Musl). No need to guess whether you need CRTDLL or MSVCRT or MSVCR70 or MSVCR71 or ... or UCRT (plus which compiler-version-specific overlay?): you need a sufficiently recent libc.so.6.

I cannot say I like this design (in some respects I actually like the Windows one more), but it does force Glibc to have very strong back-compat mechanisms and guarantees; e.g. it includes a bug-compatible version of memcpy() for old binaries that depended on it working byte by byte. As far as I’m aware, this applies from the point GNU stopped pretending Linux did not exist and the “Linux libc” fork died, that is 1998 or thereabouts.

(There are a myriad reasons why old Linux binaries won’t run on a new machine, but Glibc isn’t one of them.)

This is not to say that Glibc is perfect. Actually building a backwards-compatible binary that references old symbol versions is a gigantic pain: you either have to use a whole old environment, apply symbol versioning hacks on top of a new one and pray the resulting chimera works, or patch antique Glibc sources for new compilers and build a cross toolchain. But if you already have an old binary, it is better.

Re: The Windows malloc() implementation from MSVCRT is slow

#85

Post by cesarb »

kazinator wrote: post #65
jart wrote: post #33

Earlier quoted context omitted.

It's effectively mandatory. Microsoft provides about twelve different C Runtimes. But if you're building something like an open source library, you can't link two different C runtimes where you might accidentally malloc() memory with one and then free() with the other. If you want to be able to pass pointers around your dynamic link libraries, you have to link the one C runtime everyone else uses, which is MSVCRT. Al…

C applications targeting Windows must provide their own C library with malloc and free (if they are using the "hosted implementation" features of C). MSVCRT.DLL isn't the library "everyone" uses; just Microsoft programs, and some misguided freeware built with MinGW. Even if ADVAPI32.DLL uses MSVCRT.DLL, it's not going to mistakenly call the malloc that you provide in your application; Windows DLL's don't even have th…

> MSVCRT.DLL isn't the library "everyone" uses; just Microsoft programs, and some misguided freeware built with MinGW.

There's a third set of users: programs built with old enough versions of the Microsoft compiler. Before Microsoft decided that every single version of its C compiler should use a different C runtime (and much later they changed their mind again), all Microsoft C and C++ compilers linked their output with MSVCRT.DLL. In fact, that's probably the reason MinGW chose to use MSVCRT.DLL: to increase compatibility with the Microsoft compilers, by using the same C runtime.

Re: The Windows malloc() implementation from MSVCRT is slow

#86

Post by remram »

tick_tock_tick wrote: post #62
moonchild wrote: post #18

> it basically represents control flow as a gigantic DAG Control flow is not a DAG.

For it to be a DAG you'd have the solve the halting program wouldn't you?

No. Knowing whether programs end in finite time (dependent on input) doesn't mean all programs end or that all programs end in constant time.

The halting problem is also not considered unsolved (though P=NP is unsolved).

Re: The Windows malloc() implementation from MSVCRT is slow

#88

Post by mananaysiempre »

cesarb wrote: post #85
kazinator wrote: post #65

Earlier quoted context omitted.

C applications targeting Windows must provide their own C library with malloc and free (if they are using the "hosted implementation" features of C). MSVCRT.DLL isn't the library "everyone" uses; just Microsoft programs, and some misguided freeware built with MinGW. Even if ADVAPI32.DLL uses MSVCRT.DLL, it's not going to mistakenly call the malloc that you provide in your application; Windows DLL's don't even have th…

> MSVCRT.DLL isn't the library "everyone" uses; just Microsoft programs, and some misguided freeware built with MinGW. There's a third set of users: programs built with old enough versions of the Microsoft compiler. Before Microsoft decided that every single version of its C compiler should use a different C runtime (and much later they changed their mind again), all Microsoft C and C++ compilers linked their output…

> Before Microsoft decided that every single version of its C compiler should use a different C runtime (and much later they changed their mind again).

It is... not as simple as that[1]. MSVCRT itself is the fifth version of a Win32 C runtime, after CRTDLL (used to ship with the OS but no longer), MSVCRT10, MSVCRT20, and MSVCRT40. It’s just that both the toolchains linking with MSVCRT and the OSes shipping it (and nothing newer) endured for a very long time while Microsoft went on a collective .NET piligrimage.

Of course, “NT OS/2” started the same year ANSI C was ratified (and Win32 only a couple of years later), so some degree of flailing around was inevitable. But the MSVCRT Garden of Eden story is not true.

[1] http://www.malsmith.net/blog/visual-c-visual-history/

Re: The Windows malloc() implementation from MSVCRT is slow

#89

Post by spc476 »

pjmlp wrote: post #23

There is no Windows malloc(). Only UNIXes have the C API as part of the OS API.

malloc() is defined by the C Standard. If you want to claim your compiler is ANSI or ISO certified, you need to support malloc() (as well as the rest of the C Standard library).

Re: The Windows malloc() implementation from MSVCRT is slow

#90

Post by lkfjasdlkjfsad »

stagger87 wrote: post #82
lkfjasdlkjfsad wrote: post #59

Earlier quoted context omitted.

any idea what the hell is going on with Teams? why can't i simply scroll up in my own conversations? let alone search them. the sticky sludge of communication in something as simple as chat has cost me hours since i was forced to use teams. outlook search is so superior to teams i'd easily prefer to have lync back. this one thing absolutely cripples communication. there are a list of other very basic issues that make…

PEBKAC, you can scroll and search chats in teams.

> you can scroll

outrageously slowly, and i'm not talking about a couple HTTP requests and a database query slow

> and search

search is not practical in my opinion. i'd go as far as saying it's unusable. i can _find individual messages_, but there are times (often, i might add!) where context or jumping to that message aren't even options. context is often the reason you search for messages in the first place. if i'm alone here, sure, PEBKAC

Post reply on HN