Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What makes Intel Optane stand out (2023)

blog.zuthof.nl

Post reply on HN
Print view View original

101–110 of 164 posts

Re: What makes Intel Optane stand out (2023)

#101

Post by dmayle »

rkagerer wrote: post #32

My understanding is Optane is still unbeaten when it comes to latency. Has anyone examined its use as an OS volume, compared to today's leading SSD's? I know the throughput won't be as high, but in my experience that's not as important to how responsive your machine feels as latency.

I run two 1.5TB Optanes in raid-0 with XFS (I picked them up for $300 each on sale about two years ago). These are limited to PCIE 3.0 x4 (about 4GB/s max each). I also have a 64GB optane drive I use as my boot drive.

It's hard to tell you, because it's subjective, I don't swap back and forth between an SSD and the optane drives. I have my old system, which has a 2TB Samsung 980 Pro NVME drive (PCIE 4.0 x4, or 8GB/s max) as root, and a Sabrent rocket 4 plus 4TB drive secondary (also PCIE 4.0), so I ran sysbench on both systems, so I could share the differences. (Old system 5950X, new system 9950X3D).

It feels snappier, especially when doing compilations...

Sequential reads: I started with a 150GB fileset, but it was being served by the kernel cache on my newer system (256GB RAM vs 128GB on the old), so I switched to use 300GB of data, and the optanes gave me 5000 MiB/s for sequential read as opposed to 2800 MiB/s for the 980 Pro, and 4340 MiB/s for the Rocket 4 Plus.

Random writes alone (no read workload) The optane system gets 2184 MiB/s, the 980 Pro gets 32 MiB/s, and the Rocket 4 Plus gets 53 MiB/s.

Mixed workload (random read/write) The optanes get 725/483 as opposed to 9/6 for the 980 Pro, and 42/28 for the Rocket 4 Plus.

2x1.5TB Optane Raid0: Prep time: `sysbench fileio --file-total-size=150G prepare` 161061273600 bytes written in 50.41 seconds (3047.27 MiB/sec).

    Benchmark:
    `sysbench fileio --file-total-size=150G --file-test-mode=rndrw --max-time=60 --max-requests=0 run`
    WARNING: --max-time is deprecated, use --time instead
    sysbench 1.0.20 (using system LuaJIT 2.1.1741730670)

    Running the test with following options:
    Number of threads: 1
    Initializing random number generator from current time

    Extra file open flags: (none)
    128 files, 1.1719GiB each
    150GiB total file size
    Block size 16KiB
    Number of IO requests: 0
    Read/Write ratio for combined random IO test: 1.50
    Periodic FSYNC enabled, calling fsync() each 100 requests.
    Calling fsync() at the end of test, Enabled.
    Using synchronous I/O mode
    Doing random r/w test
    Initializing worker threads...

    Threads started!

    File operations:
        reads/s:                      46421.95
        writes/s:                     30947.96
        fsyncs/s:                     99034.84

    Throughput:
        read, MiB/s:                  725.34
        written, MiB/s:               483.56

    General statistics:
        total time:                          60.0005s
        total number of events:              10584397

    Latency (ms):
             min:                                    0.00
             avg:                                    0.01
             max:                                    1.32
             95th percentile:                        0.03
             sum:                                58687.09

    Threads fairness:
        events (avg/stddev):           10584397.0000/0.00
        execution time (avg/stddev):   58.6871/0.00
2TB Nand Samsung 980 Pro: Prep time: `sysbench fileio --file-total-size=150G prepare` 161061273600 bytes written in 87.15 seconds (1762.53 MiB/sec).

    Benchmark:
    `sysbench fileio --file-total-size=150G --file-test-mode=rndrw --max-time=60 --max-requests=0 run`
    WARNING: --max-time is deprecated, use --time instead
    sysbench 1.0.20 (using system LuaJIT 2.1.1741730670)

    Running the test with following options:
    Number of threads: 1
    Initializing random number generator from current time

    Extra file open flags: (none)
    128 files, 1.1719GiB each
    150GiB total file size
    Block size 16KiB
    Number of IO requests: 0
    Read/Write ratio for combined random IO test: 1.50
    Periodic FSYNC enabled, calling fsync() each 100 requests.
    Calling fsync() at the end of test, Enabled.
    Using synchronous I/O mode
    Doing random r/w test
    Initializing worker threads...

    Threads started!

    File operations:
        reads/s:                      594.34
        writes/s:                     396.23
        fsyncs/s:                     1268.87

    Throughput:
        read, MiB/s:                  9.29
        written, MiB/s:               6.19

    General statistics:
        total time:                          60.0662s
        total number of events:              135589

    Latency (ms):
             min:                                    0.00
             avg:                                    0.44
             max:                                   15.35
             95th percentile:                        1.73
             sum:                                59972.76

    Threads fairness:
        events (avg/stddev):           135589.0000/0.00
        execution time (avg/stddev):   59.9728/0.00
4TB Sabrent Rocket 4 Plus: Prep time: `sysbench fileio --file-total-size=300G prepare` 322122547200 bytes written in 152.39 seconds (2015.92 MiB/sec).

    Benchmark:
    `sysbench fileio --file-total-size=300G --file-test-mode=rndrw --max-time=60 --max-requests=0 run`
    WARNING: --max-time is deprecated, use --time instead
    sysbench 1.0.20 (using system LuaJIT 2.1.1741730670)

    Running the test with following options:
    Number of threads: 1
    Initializing random number generator from current time

    Extra file open flags: (none)
    128 files, 2.3438GiB each
    300GiB total file size
    Block size 16KiB
    Number of IO requests: 0
    Read/Write ratio for combined random IO test: 1.50
    Periodic FSYNC enabled, calling fsync() each 100 requests.
    Calling fsync() at the end of test, Enabled.
    Using synchronous I/O mode
    Doing random r/w test
    Initializing worker threads...

    Threads started!

    File operations:
        reads/s:                      2690.28
        writes/s:                     1793.52
        fsyncs/s:                     5740.92

    Throughput:
        read, MiB/s:                  42.04
        written, MiB/s:               28.02

    General statistics:
        total time:                          60.0155s
        total number of events:              613520

    Latency (ms):
             min:                                    0.00
             avg:                                    0.10
             max:                                    8.22
             95th percentile:                        0.32
             sum:                                59887.69

    Threads fairness:
        events (avg/stddev):           613520.0000/0.00
        execution time (avg/stddev):   59.8877/0.00

Re: What makes Intel Optane stand out (2023)

#102

Post by jauntywundrkind »

mapt wrote: post #100
jauntywundrkind wrote: post #27

Earlier quoted context omitted.

Cost was fantastically cheap, if you take into account that Optane is going to live >>10x longer than a SSD. For a lot of bulk storage, yes, you don't have frequently changing data. But for databases or caches, that are under heavy load, optane was not only far faster, but if looking at life-cycle costs, way way less.

Write endurance of the drive would be measured in TBW, and TLC flash kept adding enough 3D layers to stay cheap enough, quickly enough, that Optane never really beat their pricing per TBW to make a practical product. I have to wonder if it isn't usable for some kind of specialized AI workflow that would benefit from extremely low latency reads but which is isn't written often, at this point. Perhaps integrated in a G…

The extra capacity of modern SSD is a good point, especially now that we have 100TB+ SSD.

But Optane still offered 100 DWPD (drive writes per day), up to 3.2TB. Thats still just so many more DWPD than flash ssd. A Kioxia CM8V for example will do 12TB at 3 DWPD. The net TBW is still 10x apart.

You can get back to high endurance with SLC drives like the Solidigm p7-p5810, but you're back down to 1.6TB and 50 DWPD, so, 1/4 the Intel P5800X endurance, and worse latencies. I highly suspect the drive model here is a homage, and in spite of being much newer and very expensive, the original is still so much better in so many ways. https://www.solidigm.com/content/solidigm/us/en/products/dat...

You also end up paying for what I assume is a circa six figure drive, if you are substituting DWPD with more capacity than you need. There's something elegant about being able to keep using your cells, versus overbuying on cells with the intent to be able to rip through them relatively quickly.

Re: What makes Intel Optane stand out (2023)

#103

Post by zozbot234 »

mapt wrote: post #100
jauntywundrkind wrote: post #27

Earlier quoted context omitted.

Cost was fantastically cheap, if you take into account that Optane is going to live >>10x longer than a SSD. For a lot of bulk storage, yes, you don't have frequently changing data. But for databases or caches, that are under heavy load, optane was not only far faster, but if looking at life-cycle costs, way way less.

Write endurance of the drive would be measured in TBW, and TLC flash kept adding enough 3D layers to stay cheap enough, quickly enough, that Optane never really beat their pricing per TBW to make a practical product. I have to wonder if it isn't usable for some kind of specialized AI workflow that would benefit from extremely low latency reads but which is isn't written often, at this point. Perhaps integrated in a G…

Optane practical TBW endurance is way higher than that of even TLC flash, never mind QLC or PLC which is the current standard for consumer NAND hardware. It even seems to go way beyond what's stated on the spec sheet. However, while Optane excels for write-heavy workloads (not read-heavy, where NAND actually performs very well) these are also power-hungry which is a limitation for modern AI workflow.

Re: What makes Intel Optane stand out (2023)

#104

Post by downrightmike »

ece wrote: post #18

Fabs are expensive and all, but maybe running a right-sized fab could have still been profitable at making optane for low-latency work that it was so good at. Even moreso with RAM prices as they are.

PE popped already, now Private credit is already popping, having been in too many bubbles, Datacenter bubble popped them. The whole system just isn't talking about it because the media isn't doing their job to report. There are no more good loans to make, the ROI is nil and the interest rate is spot plus 5.5%, and businesses don't want to pay 10% interest on a loan. We're ina zombie cycle

Re: What makes Intel Optane stand out (2023)

#105

Post by PunchyHamster »

jauntywundrkind wrote: post #27
ksec wrote: post #17

Earlier quoted context omitted.

>Which is weird.... It isn't weird at all. I would be surprised if it ever succeed in the first place. Cost was way too high. Intel not sharing the tech with others other than Micron. Micron wasn't committed to it either, and since unused capacity at the Fab was paid by Intel regardless they dont care. No long term solution or strategy to bring cost down. Neither Intel or Micron have a vision on this. No one wanted a…

Cost was fantastically cheap, if you take into account that Optane is going to live >>10x longer than a SSD. For a lot of bulk storage, yes, you don't have frequently changing data. But for databases or caches, that are under heavy load, optane was not only far faster, but if looking at life-cycle costs, way way less.

So instead of replacing every 5 years you replace every 5 years because if you need that level of performance you're replacing servers every 5 years anyway

Re: What makes Intel Optane stand out (2023)

#106

Post by PunchyHamster »

thesz wrote: post #84
hbogert wrote: post #3

It stands out, because it didn't sell. Which is weird because there were some pretty big pros about using them. The latency for updating 1 byte was crazy good. Some databases or journals for something like zfs really benefited from this.

In "databases and journals" you rarely update just one byte, you do a transaction that updates data, several indexes and metadata. All of that needs to be atomic. Power failure can happen in between any of "1 byte updates with crazy latencies." However small latency is, power failure is still faster. Usually, there is a write ahead or some other log that alleviates the problem, this log is usually written in streamin…

It's not. You won't be writing one byte, ever (even if you had layers that actually supported less-than-block writes), because the overhead of instruction would be massive and you'd be murdering both latency and bandwidth for anything non-trivial

Re: What makes Intel Optane stand out (2023)

#107

Post by PunchyHamster »

hrmtst93837 wrote: post #86
zozbot234 wrote: post #14

Earlier quoted context omitted.

Optane didn't sell because they focused on their weird persistent DIMM sticks, which are a nightmare for enterprise where for many ordinary purposes you want ephemeral data that disappears as soon as you cut power. Thet should have focused on making ordinary storage and solving the interconnect bandwidth and latency problems differently, such as with more up-to-date PCIe standards.

[flagged]

> and file systems built around NAND assumptions, a lot of the upside got shaved off before users ever saw it.

What file systems ? Most common one you'd find would be ext4 or XFS and neither of them are

Re: What makes Intel Optane stand out (2023)

#109

Post by PunchyHamster »

exmadscientist wrote: post #78
georgeburdell wrote: post #67

Earlier quoted context omitted.

Flash has the same shrink problem. And the solution for Optane was the same: go 3D

I don't think the shrink problem is at all the same for the two technologies. There are some really weird materials and production steps in Optane that are simply not present when making Flash cells.

durability drops quickly with shrinking flash, we won't see much smaller cells, the growth has been MLC-TLC-> QLC and stacking
Post reply on HN