41–50 of 218 posts
Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Is there any chance that merging and rebasing via drag-n-drop is coming to Sublime Merge? For me that's the one big feature which keeps me from switching from Gitkraken to Sublime Merge.
I feel like this is kind of a dumb design. You want to abstract two different kinds of file reader: an mmap reader and a regular reader. (And I would add a gz reader, personally). Then by inspecting the properties of the file, you can determine if it is local when opening, and if so, mmap the file. I say this because if the file is coming via the network or a FAT32 partition you’re not going to save much time with mm…
Seems like the worst choice in a situation like this!
I feel like this is kind of a dumb design. You want to abstract two different kinds of file reader: an mmap reader and a regular reader. (And I would add a gz reader, personally). Then by inspecting the properties of the file, you can determine if it is local when opening, and if so, mmap the file. I say this because if the file is coming via the network or a FAT32 partition you’re not going to save much time with mm…
But now you have two completely independent code paths. Both of which will need to go through the same maturation phase that the ST folks evidently went through with mmap. And if the code needs to evolve for other reasons, potentially both of these paths will need some love too. Seems like the worst choice in a situation like this!
There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first. That said , there's nothing wrong with mmap or SIGB…
Can you link to the libc maintainer responses? I would like to know how they countered you exactly.
I feel like this is kind of a dumb design. You want to abstract two different kinds of file reader: an mmap reader and a regular reader. (And I would add a gz reader, personally). Then by inspecting the properties of the file, you can determine if it is local when opening, and if so, mmap the file. I say this because if the file is coming via the network or a FAT32 partition you’re not going to save much time with mm…
We already do this. Small files aren't mmap'd in Sublime Merge and instead copied into memory.
> I say this because if the file is coming via the network or a FAT32 partition you’re not going to save much time with mmap relative to the read speed anyways.
Speed in this case was the absolute least of our concerns. mmap was used to deal with large files without refactoring large parts of the codebase. Not using it for networked files doesn't fix the memory usage issue.
There's also the matter of taking an implicit "system call" (via page fault) the first time your program touches a page that hasn't yet been faulted. This old myth that mmap is the fast and efficient way to do IO just won't die. mmap does have perfectly legitimate use cases (e.g., reducing anonymous commit charge) but you should try to make regular reads work first. That said , there's nothing wrong with mmap or SIGB…
Well... because it's not a myth in all cases?
$ time rg zqzqzqzq OpenSubtitles2016.raw.en --mmap
real 1.167
user 0.815
sys 0.349
maxmem 9473 MB
faults 0
$ time rg zqzqzqzq OpenSubtitles2016.raw.en --no-mmap
real 1.748
user 0.506
sys 1.239
maxmem 9 MB
faults 0
> ripgrep may abort unexpectedly when using
> default settings if it searches a file that
> is simultaneously truncated. This behavior
> can be avoided by passing the --no-mmap flag
> which will forcefully disable the use of
> memory maps in all cases.
Changing the workload can dramatically alter these conclusions. For example, on a checkout of the Linux kernel:
$ time rg zqzqzqzq --mmap
real 1.661
user 1.603
sys 3.128
maxmem 41 MB
faults 0
$ time rg zqzqzqzq --no-mmap
real 0.126
user 0.702
sys 0.586
maxmem 20 MB
faults 0
FWIW, I do generally disagree with your broader point, but it's important to understand that there's actually good reason to believe that using mmaps can be faster in some circumstances.
Things are so much better if you are not writing apps for general public (mine are trading-related). You can tell your few clients — make sure that the access to mmapped file is exclusive — and get away with it. And yes, mmap is the awesomest thing out there.
While mmap is fast, the combination of factors that can make it decide to stall your thread while it commits to disk is difficult to manage from an operational standpoint. A slight misconfiguration is all it takes to introduce a rare and hard to notice multi-millisecond delay.
Whereas with a spinlocked sized-reserved vector, the fail state performance is however long it takes to allocate more space which is on the level of microseconds. You do pay 50-150ns for that spinlock though.
Oh I see you didn't get to caveat 5: you can't read anything more complicated than raw bytes, i.e. chars, because of unaligned memory access errors. Let's say you mmap a file and do something like this: char *fileContents=...mmap etc...; int headerOffset=*(int*)fileContents; int *someListOfNumbers=(int*)(fileContents+headerOffset); int importantSum= someListOfNumbers[0] + someListOfNumbers[1] + someListOfNumbers[2] +…
> On x86 you'll get away with it...
Guess we'll have to wait for the proliferation of a different architecture to encounter this one :)
And that's just for reading. Writing, especially if you want to be sure when the writes hit the backing file, or in what order, or if you run out of disk space, is another kettle of problems. I wonder how Multics dealt with all this, since AIUI in that system everything was effectively an mmapped file.
Earlier quoted context omitted.
If you roughly know your access patterns in advance you can reduce the page fault costs with madvise.
Sure, but you still end up with double-caching and frequent entry in the kernel. madvise will never give you as much control and flexibility as just getting the system out of the way and doing the work yourself. (DB systems can also use fun tricks like compressing their cached pages.)