Earlier quoted context omitted.
Optane was cancelled because manufacturer sold the fab
Oh? Complete coincidence they got caught not paying ECDL royalties?
?
wdym
21 GB/s CSV Parsing Using SIMD on AMD 9950X
161–170 of 176 posts
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#162Earlier quoted context omitted.
Not an issue when you control both ends of the pipe. CSV is a great interchange format for tabular data, especially so if it's only/mostly numeric. If you need to pass tabular data from internal service X to internal service Y it's great. And it's really fast.
Hmmm if they're just internal tools, why not just an array of structs? No parsing needed. Can have optionals. Can't go faster than nothing.
Mostly because dependencies are hard and extra so when you need another team using a different language to also neeeds support the same format.
I’d love to pass parquet data around, or SQLite dbs, or something else, but that requires dedicated support from other teams upstream/downstream.
Everyone and everything supports CSV, and when they don’t they can hack a simple parser quickly. I know that getting a CSV parser right for all the edge cases is very hard, but they don’t need to. They just need to support the features we use. That’s simple and quick and everyone quickly moves on to the actual work of processing the data.
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#163Earlier quoted context omitted.
Not an issue when you control both ends of the pipe. CSV is a great interchange format for tabular data, especially so if it's only/mostly numeric. If you need to pass tabular data from internal service X to internal service Y it's great. And it's really fast.
yep use it a lot for internal stuff and I can't recall the last time we had an issue with parsing or using it. It just works for us as a data interchange file format for tabular data. Of course our character set is basically just ascii letter and numbers, we don't even need commas or quotation marks.
Precisely. And if things get a bit more complicated slap a ‘|’ as a separator and you are almost guaranteed to never need to quote anything.
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#164Earlier quoted context omitted.
Oh? Complete coincidence they got caught not paying ECDL royalties?
? wdym
When Energy Conversion Devices went bankrupt, it appears Intel pirated the technology, and never bothered to pay the royalties for the PCM memory in Optane.
Case No. 12-43166 is what finally killed Optane.
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#165Earlier quoted context omitted.
Hmmm if they're just internal tools, why not just an array of structs? No parsing needed. Can have optionals. Can't go faster than nothing.
Mostly because dependencies are hard and extra so when you need another team using a different language to also neeeds support the same format. I’d love to pass parquet data around, or SQLite dbs, or something else, but that requires dedicated support from other teams upstream/downstream. Everyone and everything supports CSV, and when they don’t they can hack a simple parser quickly. I know that getting a CSV parser…
Yeah there's no format to support that way. Maybe I'm more biased towards numeric data (sensor readings, etc), but I never have to worry about libraries and dependencies to say
data = (uint32_t *)read(f);
Or
data = struct.unpack...
Sounds like you're dealing with more heavily formatted or variably formatted data that benefits from more structure to it
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#166Earlier quoted context omitted.
That is what Intel does, they build up a market (Optane) and then do a rug pull (Depth Cameras). They continue to do this thing where they do a huge push into a new technology, then don't see the uptake and let it die. Instead of building slowly and then at the right time, doing a big push. Optane support was just getting mature in the Linux kernel when they pulled it. And they focused on some weird cost cutting move…
> They continue to do this thing where they do a huge push into a new technology, then don't see the uptake and let it die. Except Intel deliberately made AVX 512 a feature exclusively available to Xeon and enterprise processors in future generations. This backward step artificially limits its availability, forcing enterprises to invest in more expensive hardware. I wonder if Intel has taken a similar approach with A…
They've changed that decision. All upcoming cores (even e-cores) will have AVX10 (-512)
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#167Earlier quoted context omitted.
That is what Intel does, they build up a market (Optane) and then do a rug pull (Depth Cameras). They continue to do this thing where they do a huge push into a new technology, then don't see the uptake and let it die. Instead of building slowly and then at the right time, doing a big push. Optane support was just getting mature in the Linux kernel when they pulled it. And they focused on some weird cost cutting move…
The rugpull on Optane was incredibly frustrating. Intel developed a technology which made really meaningful improvements to workloads in an industry that is full of sticky late adopters (RDBMSes). They kept investing until the point where they had unequivocally made their point and the late adopters were just about getting it... and then killed it! It's hard to understand how they could have played that particular ha…
> Even a few years on, I'm missing Optane drives because there is still no functional alternative...... they would have created a set of enterprise customers who would still be buying the things in 2040.
I guess many on HN are software developers looking at Optane.
In reality Optane was simply not cost effective. Optane came at a time when DRAM cost / GB was at its peak, the idea to developers could have slower DRAM that is non-volatile is great, until they realise slower DRAM causes CPU performance regression. Optane Memory, even on its roadmap in future product will always effectively be another layer between DRAM and NAND ( Storage ). And they could barely make profits when DRAM was at its peak. I dont think people realise there is near "4x" price difference between the height of DRAM price in ~2016 ish to ~2023.
In terms of Optane Storage, it was again at NAND's cost /GB peak and it was barely completing or making profits. Most would immediately point out it has lower latency and better QD1 performance. But Samsung showcased with Z-NAND, which is specifically tuned SLC NAND you can get close enough performance, far higher bandwidth and QD32 results, while using much lower power. And has a reliable Roadmap that is alongside the NAND development. Even Samsung stopped development of Z-NAND in 2023.
The truth is the market isn't interested in Optane enough at the price/ performance and feature it was offering. And Intel's execution for Optane, they have either over promised ( as they do in that era ) and fail to deliver on time or they are basically lying about the potential. And fail to bring down cost of fabbing it, which they blame Micron but in reality it is all on Intel.
The industry have also repeatedly stated they are not interested in a technology that is single sourced by either Intel or Micron. Unlike NAND and DRAM.
Intel was giving away Optane and pushing to Facebook and other Hyperscaler. But even then they couldn't even fill the minimum order for Micron and had to pay hundreds of millions per year for empty fabs.
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#168Earlier quoted context omitted.
Isn’t NVMe disks basically same value as optane? Comments saying optane was amazing doesn’t make sense if NVMe is basically as good and there are other NVMe disk manufacturers
They are not really the same. First, NVMe is a protocol to access block storage. It can be used to access any kind of block device, Optane, SSD, NVDIMM, virtual storage on EC2, etc. So it's true that the protocol is the same (well, not quite - more on this in a bit), but that's like saying a server is the same as an iPhone because they can both speak TCP/IP. What was the "more in a bit" bit? Persistent memory (PMEM)…
Thanks for the great explanation
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#169Earlier quoted context omitted.
Mostly because dependencies are hard and extra so when you need another team using a different language to also neeeds support the same format. I’d love to pass parquet data around, or SQLite dbs, or something else, but that requires dedicated support from other teams upstream/downstream. Everyone and everything supports CSV, and when they don’t they can hack a simple parser quickly. I know that getting a CSV parser…
Yeah there's no format to support that way. Maybe I'm more biased towards numeric data (sensor readings, etc), but I never have to worry about libraries and dependencies to say data = (uint32_t *)read(f); Or data = struct.unpack... Sounds like you're dealing with more heavily formatted or variably formatted data that benefits from more structure to it
Lucky you! We produce/consume several files from other teams, and those teams use Python, Java, Go, and C++ internally. I can try to bend those teams to my will by pushing my own custom serialization library (and would that even be a fair thing to do?) or I can just pass them a CSV.
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#170I shudder to think who needs to process a million lines of csv that fast...
I have. I think it's a pretty easy situation for certain kinds of startups to find themselves in: - Someone decides on CSV because it's easy to produce and you don't have that much data. Plus it's easier for the to read so they quit asking you to give them Excel sheets. Here is anyone who has a legit need to see your data and knows Excel really well. It can range from business types to lab scientists. - Your internal…
We use parquet extensively at work, and it's really slow to ingest. Slower than a hand rolled binary column oriented format.
Sometimes using something standardized is just worth it though.