Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Digital clutter: Learning to let go and stop hoarding terabytes

paulstamatiou.com

Post reply on HN
Print view View original

131–140 of 142 posts

Re: Digital clutter: Learning to let go and stop hoarding terabytes

#131

Post by megous »

jeffbee wrote: post #51
megous wrote: post #4

Not sure how are these nvme enclosures supposed to work. The problem is right in the picture. Samsung SSD bla bla 3.3V/3.3A = 11W and that's just for the SSD itself, the controller chip in the enclosure will also consume a lot of power at the advertised speeds. So you're looking at say 13W or so peak power consumption of the whole deal. Normal USB 3.0 port can do 4.5W. So sticking that contraption into such port will…

The thing in the picture won't work at all if you plug it into a USB 3.0 port so that problem solves itself. I believe Thunderbolt ports are required to provide a minimum of 15W. But overall you might be right, there is not an end-to-end negotiation that would avoid a worst-case brownout of you had a very power-hungry SSD. The SSD in the article has a high-power active mode of 7.5W max (I am not sure why it says 3.3v…

> It's possible to configure NVMe devices with power caps, but the 970 Evo in the article loses 95% of its performance if you cap it to 3.6W, which is its lowest operational power state.

Heh, yeah I saw about 4 MiB/s max read speed with similar enclosure plugged into USB 3.0 port (I guess it self limited for a while). Meanwhile much less power hungry SATA SSD + USB-SATA bridge worked at 400 MiB/s there.

Re: Digital clutter: Learning to let go and stop hoarding terabytes

#132

Post by megous »

rektide wrote: post #13
megous wrote: post #4

Not sure how are these nvme enclosures supposed to work. The problem is right in the picture. Samsung SSD bla bla 3.3V/3.3A = 11W and that's just for the SSD itself, the controller chip in the enclosure will also consume a lot of power at the advertised speeds. So you're looking at say 13W or so peak power consumption of the whole deal. Normal USB 3.0 port can do 4.5W. So sticking that contraption into such port will…

Peak power on these recent-ish high performance recent drive reviews shows 9w, but most drives peak consumption is 5W. https://www.tomshardware.com/reviews/samsung-990-pro-ssd-rev... Absolutely some drives won't qualify, especially PCIe 5. But many will be fine. Also, PCIe 3 x4 is 4GBps or 32Gbps which isn't that far from saturating usb4 anyways, so it's not like we even can get drives running full tilt over USB at p…

5W is still a lot if you add controller power requirements, and 5V->3.3V DC-DC conversion losses.

Also I have one of those Power-Z Type-C Type-C power meters, and even with less power hungry western digital nvme, it didn't fit into 4.5W. And it consumed the most power not even during initial write test, but during a minute after kernel reported sync() success so after everything was supposedly written to the device and no power hungry activity was needed. Then it jumped to 7-8W.

I guess these dram-less nvme's do some reshuffling from faster flash to slower flash locations during idle time. Which is probably terrible for portable use, where you may want to sync() and then unplug the device once the OS tells you everything is synced up. Creating regular power loss situations during these reshufflings just feels like putting too much trust into nvme firmware, IMO. :D

Re: Digital clutter: Learning to let go and stop hoarding terabytes

#133

Post by Baeocystin »

chaxor wrote: post #71
Baeocystin wrote: post #37

Earlier quoted context omitted.

I enjoy it, too. But I've got an active plan for cleaning up and managing my old folders- I'm going to wait until a computer can do it for me. The same plan with regards to tagging my thousands of photos worked out great! Without me needing to anything else, auto-tagging has gotten good enough that I can find images that had been lost under some IMG_xxxx.JPG filename for years. I'm looking forward to being able to ex…

https://github.com/jjuliano/aifiles

Thanks for the link. This is exactly the sort of thing I have in mind.

Re: Digital clutter: Learning to let go and stop hoarding terabytes

#135

Post by jeffbee »

megous wrote: post #131
jeffbee wrote: post #51

Earlier quoted context omitted.

The thing in the picture won't work at all if you plug it into a USB 3.0 port so that problem solves itself. I believe Thunderbolt ports are required to provide a minimum of 15W. But overall you might be right, there is not an end-to-end negotiation that would avoid a worst-case brownout of you had a very power-hungry SSD. The SSD in the article has a high-power active mode of 7.5W max (I am not sure why it says 3.3v…

> It's possible to configure NVMe devices with power caps, but the 970 Evo in the article loses 95% of its performance if you cap it to 3.6W, which is its lowest operational power state. Heh, yeah I saw about 4 MiB/s max read speed with similar enclosure plugged into USB 3.0 port (I guess it self limited for a while). Meanwhile much less power hungry SATA SSD + USB-SATA bridge worked at 400 MiB/s there.

I don't know why the Samsung is so bad at low power. A WD SN850X in the same machine has no noticeable change in random read performance with a low power state configured. It's half the speed of the Samsung at full power, but 10x as fast at low power.

Re: Digital clutter: Learning to let go and stop hoarding terabytes

#136

Post by danparsonson »

PStamatiou wrote: post #32
danparsonson wrote: post #2

If you choose SSDs, remember to power them up every few months or so to keep the data alive.

True though isn’t this closer to 2-5 years? (With manufacturers seemingly suggesting closer to 5-10)

Ah maybe modern drives are better, I don't know - I think the JEDEC standard mandates one year at 30 degrees C.

Re: Digital clutter: Learning to let go and stop hoarding terabytes

#137

Post by prawn »

ilyt wrote: post #109
prawn wrote: post #14

Earlier quoted context omitted.

Yes, usually shooting for a client: https://serio.com.au/projects/ I should probably just add a 10% line item for drives and archiving. But also a good amount is speculative - shoot a location and then try to sell the content to various parties. Sometimes sell stuff a year or more after shooting it. Keeping one copy isn’t onerous or expensive, but the mental baggage of shuffling around multiple copies gets a bit much…

Here is an idea: have "forever" directory and "for 3 years" directory. Stuff lands in "for 3 years" directory by default and gets removed automatically. "Forever" is only for stuff you personally think it's exceptionally well made.

The personal stuff is generally travel/outings with my children, and I naturally think everything about them is "exceptionally well made"! Har har.

Re: Digital clutter: Learning to let go and stop hoarding terabytes

#138

Post by AdrianB1 »

prawn wrote: post #7

I work with photos and videos and cannot comprehend downsizing to 2TB. Or even finding time to go through and selectively cull old footage. I shot a simple one-day drone-only vineyard job last week that generated 140GB, not including graded renders and photo edits afterwards. On any given day, I will have 5-7 external drives connected, plus a NAS box. There are another 15+ drives (2-5TB) in the drawer beside me. SSDs…

Why do you keep 5-7 external drives connected at all times instead of a NAS with all that storage, RAID5/6 or equivalent, and 10 Gbps network interface? Also the 2-5TB drivers in the drawer look like a pretty big risk of failure, the size suggests they are all old drives. I would rather build a NAS with 12TB drives (cost effective, larges is quite expensive) and plenty of redundancy, maybe a 0.5TB nVME or SATA SSD as cache for better speed. The only problem with such a setup is speed, unless you go for multiple RAID5/6 arrays the performance will be quite limited.

Re: Digital clutter: Learning to let go and stop hoarding terabytes

#139

Post by AdrianB1 »

noobcoder wrote: post #95

I work with a vast collection of unprocessed photographs, which takes up about 6 terabytes of storage space on my 4-bay network-attached storage (NAS) device - a DS418. Despite being limited to 1 GbE speeds, the process of loading each of the raw images, which are around 60-70 MB in size, only takes half a second. I occasionally use the smb multichannel flag to improve my bandwidth when connecting to my Mac through t…

For 10 Gbps equipment look at Mikrotik switches, a 4 port SFP+ one was around $130 when I bought it and the PCIe NICs were $50 a piece. I used DAC and AOC instead of UTP cables, they were $20-$50 each depending on the length, I paid for the whole thing a bit over $600 (6 SFP+ ports in total on 2 switches).

I use TrueNAS and the cache works only for reads, it helps with navigation of the folder structure (tens of thousands of photos, for example), but not with writes and not with most reads. Some other solutions can do write caching on SSD, that may improve the performance over 10 Gbps network a lot, but for 1 Gbps cache is not that useful.

Re: Digital clutter: Learning to let go and stop hoarding terabytes

#140

Post by prawn »

AdrianB1 wrote: post #138
prawn wrote: post #7

I work with photos and videos and cannot comprehend downsizing to 2TB. Or even finding time to go through and selectively cull old footage. I shot a simple one-day drone-only vineyard job last week that generated 140GB, not including graded renders and photo edits afterwards. On any given day, I will have 5-7 external drives connected, plus a NAS box. There are another 15+ drives (2-5TB) in the drawer beside me. SSDs…

Why do you keep 5-7 external drives connected at all times instead of a NAS with all that storage, RAID5/6 or equivalent, and 10 Gbps network interface? Also the 2-5TB drivers in the drawer look like a pretty big risk of failure, the size suggests they are all old drives. I would rather build a NAS with 12TB drives (cost effective, larges is quite expensive) and plenty of redundancy, maybe a 0.5TB nVME or SATA SSD as…

Because I work with those drives from multiple locations and don't want to cart around the NAS box (which I use as an extra backup instead). Home, office, location, while away, etc. Current method is survivable, it's just a lot to keep track of. Maybe there's an alternative I'm not considering though.

The 2-4 connected SSDs are usually projects I'm actively working intensely on that day or week, or they're annual Lightroom drives where I still need to access 2021-23 pretty regularly.

The 2-3 platter drives have slightly older projects, or edited/rendered files that I need to regularly send/shuffle/folio/etc but that don't need to be quite as fast. Or they're drives I'm assembling to post out.

Post reply on HN