Why not simply use mmap judiciously in a program managed shared memory ring buffer? Then you can copy at roughly memory speed.
This post is an excuse to explain VM concepts, rather than a tutorial, something I maybe could have made clearer.
How fast are Linux pipes anyway? (2022)
71–80 of 116 posts
Re: How fast are Linux pipes anyway? (2022)
#72Are there good data handling libraries that provide abstractions over pipes, sockets, files, and memory and implement optimizations like these? I'd be interested in knowing if there are such libraries in C, C++, Rust, or other systems languages. I wasn't familiar with some of the APIs mentioned in the article like splice() and vmsplice(), so I wondered if there are libraries that I might use when building ~low-level…
This may go against the grain but this isn't really worth abstracting over since it's not portable. You'll probably want to implement it by hand everywhere you need it. Higher level code only uses them rarely because they're pretty special purpose and they have to be specialized for Linux. If you're shuffling data around without looking at it only on Linux, splice is useful. There's not that many applications that ha…
I think Linus generally considers splice a failed experiment. It works fine is some simple scenarios, but the generalized support for it needed to make it work failed to materialize.
Having said that, these days sendfile is implemented in term of splice, so in a way many HTTP servers use it.
Re: How fast are Linux pipes anyway? (2022)
#73So if I understand correctly, vmsplice is more of a mini shared memory mechanism between two processes, if used on both the reader and writer end simultaneously? Meaning both processes need to be exceptionally careful in when they read and write to the buffers and how it is returned after use. Hot, yet scary at the same time. Other main takeaway, it’s a bit sad that the naive implementation everybody will write, is 2…
>vmsplice is more of a mini shared memory mechanism between two processes
Doesn't seem to be the case, as it only supports zero-copy from user-memory to the pipe. The other way around results in a copy - see https://mazzo.li/posts/fast-pipes.html#fn10
Re: How fast are Linux pipes anyway? (2022)
#74You can probably still make it faster by avoiding libc and using syscalls directly. From looking at the final perf output, it looks like there's some overhead from using libc functions
Re: How fast are Linux pipes anyway? (2022)
#75Earlier quoted context omitted.
Pipes are FIFO data buffers implemented in the kernel. For communication between threads of the same process, you can replace any pipe object by a userspace queue implementation protected by e.g. mutex + condition variable. It is functionally equivalent and has potential to be faster. And if you wrap all accesses in lock/unlock pairs (without locking any other objects in between) there is no danger of introducing any…
Incidentally you can use the exact same setup (plush mmap) for interprocess queues. The advantage of threads is that you can pass pointers to your data through the queue, while that's harder to do between processes and you have to resort to copying data in the queue instead.
>while that's harder to do between processes and you have to resort to copying data in the queue instead.
I could be wrong - I've never done it, but I understood that you can even store POSIX mutexes and condition vars in shared mem so that 2 processes (or more?) can process data without copying, so long as they use the both use the same locks stored in the shared memory.
Re: How fast are Linux pipes anyway? (2022)
#76Earlier quoted context omitted.
And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.
Not necessarily. Good comments go a long way.
I have a very long list of things that were good, worked well, and ended up rejected because the team didn't want to put in effort to learn how they work.
My conclusion so far is that if you want to make things work well, you shouldn't be working on a commercial project, use a unpopular language with a steep learning curve to filter out those who'd be a drag on your project. Maybe you don't have to be a jerk, but being blunt helps.
Below are some examples of initiatives that were meant to improve things and how they failed due to other programmers being lazy and / or ignorant.
When ActionScript was a thing it competed with HaXe. A similar (also ECMAScript-related) language with a small but dedicated community, a compiler that was hugely superior to to MXMLC (official Adobe compiler for AS3) and a bunch of features intended to improve code correctness and performance.
I was hired by a company making a "PowerPoint online" kind of product. The main system component was a large Flex (AS3) applet that was hugely inefficient especially in terms of how it utilized network. It had to load huge shared libraries with assets (mostly clip art) every time users wanted to either edit or watch a presentation. The AWS bill was growing dangerously big. My mission was to find a solution to reduce the network activity.
My idea was to create a separate player component that would extract the relevant assets from the libraries server-side, compile them into individual SWFs. The reason was that the final presentations were loaded a lot more often and by first-time users (i.e. no caching). HaXe was the ideal language because it already had a library that could generate a large subset of SWF, and it could compile both to AS3 and to C++, so that generation could also be done on a server using a more efficient implementation.
After several month of work, I produced a set of programs that could generate SWFs both server-side and client side and showed how this would improve the network activity. The other programmers on the AS3 team, who earlier promised to get familiar with HaXe, since they had to incorporate the new player component into the existing Flex applet... didn't hold their part of the bargain. No matter the amount of help I provided, they simply wouldn't do anything to incorporate the new component, instead making claims that grew more bizarre and more untrue as time went by.
Having spent more time trying to convince the team to adopt my code rather than writing it, I decided to look for a different place to work at. In the end, this entire effort went down the drain.
----
In a very similar way, I had to solve a problem created by using Google's Protobuf Python bindings which required generating Python modules in order to function. We needed to have an API server that could simultaneously serve multiple versions of the same Protobuf API from similarly named modules. Since Google's implementation didn't allow this, I wrote my own (while my manager was on maternity leave). I improved parsing speed, network load, reduced the amount of maintenance the component needed by making it possible to add new Protobuf message definitions at run time...
The problem was I wrote the parser in C. This is what enabled good performance. When my manager came back to work, she realized she declared that she doesn't know C and will never learn (even though she wasn't related directly to the project), and the project was thrown to the dogs.
----
I have a similar story about extracting and aggregating Web interface from a RoR app, producing a Swagger definition... written in Prolog, which was also thrown away because Prolog. Similarly, had written an I/O tester for distributed filesystem in Prolog, which was thrown away for the same reason... And this answer will eventually hit the character limit if I keep listing things that were discarded simply because programmers didn't want to learn how to do their job.
Re: How fast are Linux pipes anyway? (2022)
#77Earlier quoted context omitted.
Not necessarily. Good comments go a long way.
I have a very long list of things that were good, worked well, and ended up rejected because the team didn't want to put in effort to learn how they work. My conclusion so far is that if you want to make things work well, you shouldn't be working on a commercial project, use a unpopular language with a steep learning curve to filter out those who'd be a drag on your project. Maybe you don't have to be a jerk, but bei…
wouldnt it have been easier to port the Haxe code to AS3? afaik both language are very similar. Then you wouldn't have a problem and you wouldnt have to rely on them to learn your codebase. Likewise I doubt that the parser being written in C was the problem, the problem was probably that (I assume?) you didnt also provide the convenient python bindings. Otherwise it's hard to believe that someone would throw away finished code. I myself was in a similar situation once though, I bound to an external java code via JNI and it worked fine. But they ended up rewriting via HTTP calls anyway. Sure it's easier that way but I mean damn.. not even keeping it around as an option or as a benchmark seems a bit too lazy. But I still understood why they did it, it's just a source of possible errors for them and they don't want to deal with it. Also I wouldn't want to inherit Prolog code either because it's an ancient niche language, IDE and docs and everything else is probably terrible.
Re: How fast are Linux pipes anyway? (2022)
#78Earlier quoted context omitted.
And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.
Your coworkers would prefer you splitting the thing into two microservices communicating over a REST api, aka the 200x slower version.
Yep. And then wonder why they're getting pages in the middle of the night because CustomerPaymentsProcessingService became inconsistent with CustomerMembershipProcessingService and CustomerOrderStatusService and people's credit cards are charged without their memberships showing as active, or whatever.
Or maybe not wonder, because it makes them feel important?
Re: How fast are Linux pipes anyway? (2022)
#79Earlier quoted context omitted.
And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.
it's time to change your workplace, not everyone is meant to be petty/incompetent.
My experience is that even the most committed and smart people degrade as teams into patterns which produce seemingly unnecessary complexity because that's where our industry trends and tools push us. People create boxes to isolate perceived complexity, that in turn create new complexity.
I share the other commenter's frustrations. I want out of the tarpit.
Re: How fast are Linux pipes anyway? (2022)
#80So if I understand correctly, vmsplice is more of a mini shared memory mechanism between two processes, if used on both the reader and writer end simultaneously? Meaning both processes need to be exceptionally careful in when they read and write to the buffers and how it is returned after use. Hot, yet scary at the same time. Other main takeaway, it’s a bit sad that the naive implementation everybody will write, is 2…
And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.
> And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.
Hear hear!
Why is it like this?