Seagate Announces 16TB HDD • Hacker News Board

Re: Seagate Announces 16TB HDD

#201

Post by bunnycorn » Thu, Dec 06, 2018, 1:36 PM UTC

notacoward wrote: post #144

bunnycorn wrote: post #137

Earlier quoted context omitted.

I think it's a question of supply and demand. If professionals need throughput, they will buy a couple of hard drives and array them together. Latency is generally not requested.

> a couple of hard drives > Latency is generally not requested. That might be your experience, but it's not mine. I'm on a team that runs one of the three or four largest storage infrastructures in the world. In aggregate, those few account for a significant fraction of all the disk drives sold - enough that we have to account for possible market distortions in our planning. So I think our experience is relevant too.…

And you lack reading comprehension.

I didn't say professionals don't care about latency. I said that professionals don't ask for latency for hard drives, they go straight to SSDs when latency is important.

Re: Seagate Announces 16TB HDD

#202

Post by notacoward » Thu, Dec 06, 2018, 1:42 PM UTC

bunnycorn wrote: post #201

notacoward wrote: post #144

Earlier quoted context omitted.

> a couple of hard drives > Latency is generally not requested. That might be your experience, but it's not mine. I'm on a team that runs one of the three or four largest storage infrastructures in the world. In aggregate, those few account for a significant fraction of all the disk drives sold - enough that we have to account for possible market distortions in our planning. So I think our experience is relevant too.…

And you lack reading comprehension. I didn't say professionals don't care about latency. I said that professionals don't ask for latency for hard drives, they go straight to SSDs when latency is important.

Reading comprehension doesn't cover you retconning what you said. You didn't mention SSDs at all in your previous comment. What you actually said did imply that professionals don't care about latency. Full stop. Perhaps that's not what you meant to say, but that has more to do with your writing than anyone else's reading.

More to the point, your gaslight version is still wrong. Professionals do have latency expectations even with hard disks. Those requirements might not be as stringent for hard disks as for flash (or for distributed vs. local) but they're very much still there. If a significant fraction of users' requests are taking too long, they do complain. Loudly. Data scientists don't care if the system is simultaneously delivering dozens of gigabytes per second to other users, and doing ten kinds of background maintenance stuff besides. They care that their job is slow. When you're dealing with truly large amounts of data being processed by thousands of machines, and ad hoc queries none of which are likely to be repeated, "just use flash" isn't an answer.

Maybe you're dealing with a different kind of professional than I am. That's fine, but you shouldn't keep making these super-general statements that are wrong for one of the largest classes of data professionals. "Couple of hard drives" was so far off the mark I literally laughed out loud.

Re: Seagate Announces 16TB HDD

#203

Post by StillBored » Thu, Dec 06, 2018, 2:39 PM UTC

kalleboo wrote: post #172

notacoward wrote: post #134

As a professional storage-system developer, my reaction to every announcement of larger disks is something close to horror. Capacity keeps increasing while performance remains darn near constant, so the gap between the two keeps getting wider. Yes, we can use flash to absorb the I/O demand, treat disk more like tape, yadda yadda, but all of that takes significant effort. You do want your storage system to be correct…

Even as a home user, expanding a RAID on a consumer NAS with a 10 TB harddisk takes days due to the size. It would take weeks with a 100 TB drive! (then again, that's not something you'd need to do often with 100 TB to play with :)

The bandwidth of these drives are increasing due to the fact that the rotational speeds are tending to remain the same while the density is increasing. So, as the linear in track BPI increases so does the throughput. Tracks per inch, which is where most of the density has been doesn't improve bandwidth. This is where a lot of the density improvements a few years ago was coming from (higher TPI). At one point the TPI was actually higher than the linear BPI, which was something completely non-obvious to me when I discovered it.

HAMR is an increase in linear BPI, so expect these drives to get faster.

Re: Seagate Announces 16TB HDD

#204

Post by StillBored » Thu, Dec 06, 2018, 2:49 PM UTC

notacoward wrote: post #134

As a professional storage-system developer, my reaction to every announcement of larger disks is something close to horror. Capacity keeps increasing while performance remains darn near constant, so the gap between the two keeps getting wider. Yes, we can use flash to absorb the I/O demand, treat disk more like tape, yadda yadda, but all of that takes significant effort. You do want your storage system to be correct…

OTOH, I was in the storage industry until recently, and HD capacity increases were always welcome. That is because we were a bandwidth play, sold on the basis of density and capacity. Hello backup and archive. That is frankly where most of HD storage is today, 40 drives sitting in 4U is still a few ten-thousand ops per second, but it has a few tends of GB/sec of bandwidth too. So, if your access patterns are long linear read/writes then HDD's work just fine. Even for midrange IOP applications, if you burst a couple GB from disk to a SSD cache, its possible to absorb a lot of IOP's.

Also, rebuild/scrub operations aren't really an issue either if your running RAID6, and your system is sufficiently designed/over provisioned to deal with rebuild/scrub operations happening in the background. If your IOP limited OTOH, you likely have a problem.

Re: Seagate Announces 16TB HDD

#205

Post by StillBored » Thu, Dec 06, 2018, 3:19 PM UTC

objektif wrote: post #73

rixrax wrote: post #29

Earlier quoted context omitted.

As you say you stream your videos. Now imagine getting that shiny new 4k video camera and start storing raw footage even after you have edited it. Rough ballbark of 120Mbits/s ==> 15MB/s ==> 900MB/minute ==> 54GB/hour. This is why I keep adding more and more disks to my NAS.

What type of NAS do you have? I am researching alternatives would appreciate your advice.

On the build your on basis, there are a number of nice atom/xeon-D micro ATX motherboards with 10G Ethernet and 8+ sata/sas ports. Combined with something like the old silverstone DS380B (8 hotswap 3.5" drives, + couple internal) makes for a nice small form factor NAS.

Why would you build your own? First, because the NAS vendors charge on a non-linear scale for drive slots and 10G Ethernet. The latter is basically a $50 additional charge on a motherboard, or a $100 add in card, but will easily add $500+ to a off the shelf NAS. Then there are the drives, today you might be happy with a two or three drive NAS, but in 5 years adding an additional 8-12TB drive (or three) will be just a hundred dollars or so, vs buying a whole new NAS.

Then there is actually using it. Running a NAS a 400-600MB/sec is a far cry from using one at 80-120MB/sec which is where you will peak out with 1Gbit ethernet. This becomes really noticeable when your copying one of the 50GB 4k video's you record during the kids birthday, or just running a machine backup to the NAS.

Frankly, at the end of the day, I find the flexibility to be the largest advantage. When I got tired of my old NAS's plex encoding perf, I swapped the motherboard for $300, without having to copy any data because i'm running a stock linux distro. Much of the old advantage on NAS's were the 3rd party marketplaces to get things like plex/crashplan/etc. Now those things are frequently found in docker containers, so a bit of setup and your application installs tend to stay continuously updated, rather than hoping your NAS vendors market place is remembered.

Re: Seagate Announces 16TB HDD

#206

Post by notacoward » Thu, Dec 06, 2018, 3:27 PM UTC

StillBored wrote: post #204

notacoward wrote: post #134

As a professional storage-system developer, my reaction to every announcement of larger disks is something close to horror. Capacity keeps increasing while performance remains darn near constant, so the gap between the two keeps getting wider. Yes, we can use flash to absorb the I/O demand, treat disk more like tape, yadda yadda, but all of that takes significant effort. You do want your storage system to be correct…

OTOH, I was in the storage industry until recently, and HD capacity increases were always welcome. That is because we were a bandwidth play, sold on the basis of density and capacity. Hello backup and archive. That is frankly where most of HD storage is today, 40 drives sitting in 4U is still a few ten-thousand ops per second, but it has a few tends of GB/sec of bandwidth too. So, if your access patterns are long lin…

> Hello backup and archive. That is frankly where most of HD storage is today

I'm not sure whether that's true or not. At one end, tape still exists. At the other end, a lot of data that's already stored with high levels of redundancy doesn't get backed up anywhere else. Do you have a good source for a definitive answer?

> rebuild/scrub operations aren't really an issue either if your running RAID6

Rebuild/scrub operations still happen, and are still an issue, with RAID6. They might not be quite as visible because they're not soaking up host cycles, but they are soaking up disk IOPS. In any large or even medium-sized storage infrastructure (by today's standards) you'll have some going on and occasionally you'll have some that overlap because the first one took too long. Before long you'll hit a case where you get a third failure while the first two are still going on, and you stop relying on insufficient RAID6. Then you're into erasure codes and your own kind of scrubbing. Those aren't exotic situations or responses any more.

> If your IOP limited OTOH, you likely have a problem.

Yep, sure do. Can't wish it away, or make it all better with a magical free SSD caching layer. Have to solve it, which requires effort and expense. What I'm saying is that bigger disks make that harder. I'm not complaining, it's what I and others choose to do, but it's a fact.

Re: Seagate Announces 16TB HDD

#207

Post by StillBored » Thu, Dec 06, 2018, 3:27 PM UTC

wmf wrote: post #95

PHGamer wrote: post #60

when are we getting 8 tb ssds. its been awhile. ssds should have caught up to spinning rust by now in capacity

Enterprise 15 TB SSDs are already available. It would be easy to create 8 TB consumer SSDs but they don't see demand. There's tons of empty space inside 2.5" SSDs: https://www.anandtech.com/Gallery/Album/6783#3

Right, but a huge part of the consumer SSD space has moved to m.2, where there isn't a lot of space to just pack in another couple dies. While conceivably you might be able to make a 8TB drive in m.2 its going to be easier in 2.5", and people with the money to buy that large of a SSD probably don't want it on SATA. Which is why, as you point out they exist, here is a 8TB u.2

https://www.newegg.com/Product/Product.aspx?Item=1E4-006U-00...

for only $2600.

Re: Seagate Announces 16TB HDD

#208

Post by bunnycorn » Thu, Dec 06, 2018, 3:55 PM UTC

notacoward wrote: post #202

bunnycorn wrote: post #201

Earlier quoted context omitted.

And you lack reading comprehension. I didn't say professionals don't care about latency. I said that professionals don't ask for latency for hard drives, they go straight to SSDs when latency is important.

Reading comprehension doesn't cover you retconning what you said. You didn't mention SSDs at all in your previous comment. What you actually said did imply that professionals don't care about latency. Full stop. Perhaps that's not what you meant to say, but that has more to do with your writing than anyone else's reading. More to the point, your gaslight version is still wrong. Professionals do have latency expectati…

No, I didn't. It was your problem you misinterpreted what I said.

Nobody else misunderstood me but you.

Re: Seagate Announces 16TB HDD

#209

Post by StillBored » Thu, Dec 06, 2018, 3:56 PM UTC

notacoward wrote: post #206

StillBored wrote: post #204

Earlier quoted context omitted.

OTOH, I was in the storage industry until recently, and HD capacity increases were always welcome. That is because we were a bandwidth play, sold on the basis of density and capacity. Hello backup and archive. That is frankly where most of HD storage is today, 40 drives sitting in 4U is still a few ten-thousand ops per second, but it has a few tends of GB/sec of bandwidth too. So, if your access patterns are long lin…

> Hello backup and archive. That is frankly where most of HD storage is today I'm not sure whether that's true or not. At one end, tape still exists. At the other end, a lot of data that's already stored with high levels of redundancy doesn't get backed up anywhere else. Do you have a good source for a definitive answer? > rebuild/scrub operations aren't really an issue either if your running RAID6 Rebuild/scrub oper…

I don't have a direct summary link, but if you read random storage revenue/capacity reports that seems to be the case.

There are a few large buyers of bulk storage which appear to be using it for some form of nearline. At least in my experience (which is obviously warped by the part of the industry I was in), enterprise applications needing IOP's have been overwhelmingly moving to pure flash. There remains a large amount of revenue in hybrid arrays, but the volume is shrinking (although maybe not the raw capacity, similar to the mainframe which is selling record amounts of capacity in fewer and fewer machines). A few years ago I stopped being surprised to see a couple racks of infortrend's or supermicro storage chassis (or any number of other 2nd tier vendors products) sitting at one end of random datacenters, where the local storage admins were running some huge snapshot repo, or Ceph, or whatever on them and the resulting capacities were frequently powers of ten greater than the online storage.

Also, I've seen plenty of tape arrays too, but they don't seem nearly as common these days. Partially, because tape has the same problems as disk (lots of bandwidth, but it never seems to be enough). It seems everyone still has one, but you have to hunt for it, and it might only be getting the most critical data (or stuff that is required for compliance with some law), which turns out to be limited by the 2-4-8 drives they have constantly spinning, where an operator walks in every few days and swaps a couple dozen tapes offsite. Sure there are larger libraries but it seems most admin's start any conversation about tape with a groan and eye rolling, which tends to be an attitude that keeps finite resources from being heavily invested in them. So, people will buy 10+Gbit links between their data centers (which sometimes tend to be shockingly inexpensive) rather than spend $10k on a tape library and a fedex account.

Re: Seagate Announces 16TB HDD

#210

Post by notacoward » Thu, Dec 06, 2018, 4:31 PM UTC

bunnycorn wrote: post #208

notacoward wrote: post #202

Earlier quoted context omitted.

Reading comprehension doesn't cover you retconning what you said. You didn't mention SSDs at all in your previous comment. What you actually said did imply that professionals don't care about latency. Full stop. Perhaps that's not what you meant to say, but that has more to do with your writing than anyone else's reading. More to the point, your gaslight version is still wrong. Professionals do have latency expectati…

No, I didn't. It was your problem you misinterpreted what I said. Nobody else misunderstood me but you.

> Nobody else misunderstood me but you.

I think it would be more correct to say that nobody at all misunderstood you, and nobody else bothered to correct your claims. I do see there were some downvotes, though, which suggests what others thought of them.