[flagged]
You crossed into personal attack here, and that's not allowed on HN. It's not what this site is for, and destroys what it is for.
If you'd please review the site guidelines and stick to them when posting, we'd appreciate it.
Task Failed Successfully: Saturating NIC and Disk Bandwidth
11–20 of 23 posts
Re: Task Failed Successfully: Saturating NIC and Disk Bandwidth
#12[flagged]
[flagged]
We've banned this account for repeatedly breaking the site guidelines and ignoring our request to stop.
If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
Re: Task Failed Successfully: Saturating NIC and Disk Bandwidth
#13[flagged]
You crossed into personal attack here, and that's not allowed on HN. It's not what this site is for, and destroys what it is for. If you'd please review the site guidelines and stick to them when posting, we'd appreciate it. https://news.ycombinator.com/newsguidelines.html
Lies lies lies
Mods routinely break their own policies. You delete accounts that do not break any guidelines, you delete accounts because they say things you don't agree with, lies lies lies.
You allow government agents to do whatever they want on this site. Lies lies lies. You're so full of yourself too it's gross. Liar.
Re: Task Failed Successfully: Saturating NIC and Disk Bandwidth
#14Earlier quoted context omitted.
You crossed into personal attack here, and that's not allowed on HN. It's not what this site is for, and destroys what it is for. If you'd please review the site guidelines and stick to them when posting, we'd appreciate it. https://news.ycombinator.com/newsguidelines.html
Lies lies lies Mods routinely break their own policies. You delete accounts that do not break any guidelines, you delete accounts because they say things you don't agree with, lies lies lies. You allow government agents to do whatever they want on this site. Lies lies lies. You're so full of yourself too it's gross. Liar.
None of this is true as far as I know (ok, other than being full of myself). But why not supply links so readers can make up their own minds?
Re: Task Failed Successfully: Saturating NIC and Disk Bandwidth
#15This might be orthogonal to the TLB miss overhead you found, but have you looked at using P2PDMA to transfer directly from the NVMe SSDs to the NIC? Not sure how the CRC calculation would play into that.
Thank you for your reply. This is a long-running service. Without CRC validation, errors caused by partial writes could accumulate over time and affect correctness. Therefore, we adopted this approach.
Re: Task Failed Successfully: Saturating NIC and Disk Bandwidth
#16It seems that you could have reached this conclusion faster by elaborating on your use of the profiler. Don't assume that cycles are spent on instructions. Look at your IPC and drill down into what CPU-bound means for your workload. In your case I think a standard top down analysis would have made the virtual memory management cost jump right out.
Thank you for your suggestion. You are absolutely right, XD. In fact, the order of events in this blog post does not match the actual order in which I debugged and analyzed the issue.
After I identified the TLB misses and confirmed that huge pages were effective, I noticed that there were still many suspicious points in the flame graph. Interestingly, my agent attributed the effectiveness of huge pages to those suspicious points, which turned out to be unrelated to the bottleneck at the time. That sparked my curiosity.
The structure of this blog post was mainly chosen to make the story easier to follow, while also covering the various issues I investigated in depth along the way.
In fact, I recently switched from my previous job building data infrastructure on cloud to an HPC-related role, so I am still not very familiar with some of the mature practices and established conclusions in the HPC world.
So thank you very much for your suggestions. I also hope to learn about more and better methods that can help people identify root causes more quickly and accurately in complex scenarios.
Re: Task Failed Successfully: Saturating NIC and Disk Bandwidth
#17Author here. This post is a write-up of a performance-debugging rabbit hole I hit while trying to saturate NICs with NVMe reads using io_uring and RDMA. The short version: READ_FIXED fixed the obvious per-I/O GUP overhead in a small demo, but the larger deployment still got stuck at roughly half of line rate. After ruling out io-wq backlog, request splitting, fd lookup, and CRC arithmetic, the actual wall turned out…
I disagree with the AI part. Because hugepages is one of the things that can be guessed to improve performance when doing something with substantial amount of data. So anyone familiar with the space could have suggested something like that without knowing the details of the problem. Hence it is not useful advice IMO. That aside, the blog post was really cool to read and a instant favorite, wish there were more englis…
Thank you for liking this blog. I agree with your point. Actually, I’ve just recently transitioned from building data infrastructure on the cloud to taking on a high-performance computing role that truly handles massive amounts of data. So, although I’d heard about the benefits of hugepages before, I had never actually reproduced these issues in my own environment. This time, even though I initially suspected the problems were related to hugepages and the TLB, I didn’t write this blog from a seasoned perspective. Instead, I wanted to methodically investigate and eliminate all other possible issues I could think of. (Interestingly, my agent attributed the effectiveness of hugepages to the root causes of these bugs, which piqued my curiosity and drove my deeper exploration.)
Finally, thank you very much for your appreciation, which means a lot to me. Previously, I was working on open-source projects, but now that I’ve changed jobs, I may not have the same amount of energy to contribute to open-source code as before. However, I think blogging might be a new way for me to contribute. I hope I can keep it up.
(My English writing skills are poor, so I wrote in Chinese and used AI to translate it; I hope you don’t mind.)
Re: Task Failed Successfully: Saturating NIC and Disk Bandwidth
#18Earlier quoted context omitted.
I disagree with the AI part. Because hugepages is one of the things that can be guessed to improve performance when doing something with substantial amount of data. So anyone familiar with the space could have suggested something like that without knowing the details of the problem. Hence it is not useful advice IMO. That aside, the blog post was really cool to read and a instant favorite, wish there were more englis…
Thank you for liking this blog. I agree with your point. Actually, I’ve just recently transitioned from building data infrastructure on the cloud to taking on a high-performance computing role that truly handles massive amounts of data. So, although I’d heard about the benefits of hugepages before, I had never actually reproduced these issues in my own environment. This time, even though I initially suspected the pro…
But I'm trying my best to practice it. Hope one day I can produce some solid posts in English directly. qwq
Re: Task Failed Successfully: Saturating NIC and Disk Bandwidth
#19This might be orthogonal to the TLB miss overhead you found, but have you looked at using P2PDMA to transfer directly from the NVMe SSDs to the NIC? Not sure how the CRC calculation would play into that.
Thank you for your reply. This is a long-running service. Without CRC validation, errors caused by partial writes could accumulate over time and affect correctness. Therefore, we adopted this approach.
Depending on your hardware you may be able to do the CRC validation in the NIC. If you're using Intel you could use DSA also, but then you're still copying the data through DRAM.
Just throwing out some ideas, obviously the best solution is the one that you already have working :)
Re: Task Failed Successfully: Saturating NIC and Disk Bandwidth
#20[flagged]
You crossed into personal attack here, and that's not allowed on HN. It's not what this site is for, and destroys what it is for. If you'd please review the site guidelines and stick to them when posting, we'd appreciate it. https://news.ycombinator.com/newsguidelines.html
There's no personal attack here. There's a message addressed to a person.
That is, your statement has no sense. If you believe it is, please report it. Oh, please put your public defamation and devaluation of someone's commentary together, considering it the same "personal attack" towards me.
If you are so afraid of receiving messages that address your name, please reconsider your presence on the Internet.
If you will invest your time and effort addressing me on my projects published, where I assume responsibility and my authorship of them, being accountable for the works I publish on public.
That is, it is a work of someone who published online, and that is a responsibility to be taken, hence licenses exist, and your name on it. You are free to be "personal" on my work, too. Please do! I would appreciate it, if that would help us discover more and educate ourselves on subjects involved.
One of my GitHub accounts: https://github.com/serious-angel