Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SDcard wear leveling and translation layers (2014)

msreekan.com

Post reply on HN
Print view View original

31–40 of 72 posts

Re: SDcard wear leveling and translation layers (2014)

#31

Post by 01HNNWZ0MV43FF »

vardump wrote: post #23
billpg wrote: post #8

Earlier quoted context omitted.

I vaguely recall thinking the file was just some sort of serial number or stock tracking metadata when I deleted it. It might have instead failed for quite mundane reasons such as low quality manufacture. I don't know. Exposing the firmware in this fashion would be a bad idea for all of these reasons. It is a good thing that SD card manufacturers don't do this any more, if they ever did.

Firmware definitely would not be on the filesystem layer or even accessible on the block device.

You'd think yeah

Re: SDcard wear leveling and translation layers (2014)

#32

Post by 01HNNWZ0MV43FF »

st_goliath wrote: post #13
michaelt wrote: post #10

> For example, lets consider that we wrote to the sector 1024 on the SDCARD, and this address got initially mapped to physical sector 1024 itself on flash. Later we overwrite the contents, but this time write cannot go to the same physical address because it needs to be erased before programmed again. Maybe someone here will know the answer to something I've been wondering for a while: When appending to a file with w…

Pages that are already written cannot be overwritten, at least not without a full block erase, so it has to do a remap to append data. Directly logging to an SD card (i.e. lots of small writes) is the fastest way to destroy it.

That could explain why my dashcam is working fine after ten years whereas my previous job's IoT devices regularly ate SD cards. (Then again, the IoT devices were also overheating badly whereas the dashcam was designed for its purpose, kept cool while running, and runs on a low duty cycle)

What are you supposed to do, then? Buffer up writes and accept that you'll lose logs a minute or so before a power outage?

As much hell as we had with those SD cards I'd rather put tiny iPod spinners in them...

Re: SDcard wear leveling and translation layers (2014)

#33

Post by weweweoo »

goodburb wrote: post #28
londons_explore wrote: post #9

I am really disappointed by SD card reliability. It seems even a little wear and the whole thing slows down massively and eventually throws read/write errors. I would like to see a card design which, instead of failing when there is flash wear, instead just gets smaller. For compatibility with existing OS's, that would take the form of a self-partition-resizing sd card. It would understand fat32, ext3, NTFS, afs etc,…

Maybe there are more lemons with SD cards. The fact that they're used in dash cams, exposed to high temperature thermal cycles, and recording tens of terabytes of data which can be more than the typical consumer HDD/SSD in it's lifetime is quite impressive.

Recently I found a long lost go-pro style camera using metal detector, after it had fallen off an RC plane, which crashed into a tree in dense forest. Over nearly a decade the camera had been buried several centimeters into the ground, and the card inside the camera wasn't particularly well protected (just an open slot, not waterproof or anything), which made it exposed to lots of water and freezing winters for nearly a decade. After some cleaning the SD card worked perfectly, and all the data was intact. It even had the crash recorded in it. I was impressed by that. The camera didn't survive though.

Re: SDcard wear leveling and translation layers (2014)

#34

Post by goodburb »

st_goliath wrote: post #13
michaelt wrote: post #10

> For example, lets consider that we wrote to the sector 1024 on the SDCARD, and this address got initially mapped to physical sector 1024 itself on flash. Later we overwrite the contents, but this time write cannot go to the same physical address because it needs to be erased before programmed again. Maybe someone here will know the answer to something I've been wondering for a while: When appending to a file with w…

Pages that are already written cannot be overwritten, at least not without a full block erase, so it has to do a remap to append data. Directly logging to an SD card (i.e. lots of small writes) is the fastest way to destroy it.

I've seen some flash drives that prepare a pool of free AU blocks which is common with Phison controllers

They prepare and erase a new pool only when it's full causing bursts in speeds and long periods of unresponsiveness (10 sec) while writing, reading is blocked as well. You can see it as a zigzag in file transfer speed graphs. This is not to be confused with OS buffer flushing.

While I couldn't find any official info for the reasons behind it, it's assumed that the erase speed is slow on low quality flash and hence the controller has to pool in order to erase pages in parallel, increasing speed when copying small files between computers on plug and unplug.

In my case it was about 500MB before before freezing on a 128GB flash drive.

Re: SDcard wear leveling and translation layers (2014)

#35

Post by HankB99 »

account42 wrote: post #26
ssl-3 wrote: post #15

Earlier quoted context omitted.

ZFS can do this. Just set copies=2 on a dataset, and it'll always keep 2 copies of everything written thenceforth (with ~twice the storage usage). And since it's a dataset setting instead of a pool setting, it can be applied selectively on a storage device. "Oh, those photos are also backed up to Google Photos. It's maybe not ideal, but it's a lot of data and not worth keeping 2 copies those locally, so I'll leave th…

Is there even that much value to keeping multiple copies on the same device ? I fail to see what this adds over backups which can be done with any filesystem.

If you have a pool consisting of a single device (such as a drive in a laptop) the second copy could protect from loss of a file that includes storage that can not be read.

In a multiple device pool (with some redundancy) it would IMO be a extra and unneeded layer of protection.

Re: SDcard wear leveling and translation layers (2014)

#36

Post by goodburb »

faragon wrote: post #22

Why there's not generalized discard/trim capabilities on USB/SD devices? It should be "easy" to agree some standard on that.

Some A2 SD cards have command queueing for random read speed improvement and TRIM enabled. It requires kernel support and hardware acceleration being optional.

https://forums.raspberrypi.com/viewtopic.php?t=367459

Re: SDcard wear leveling and translation layers (2014)

#37

Post by [deleted] »

londons_explore wrote: post #9

I am really disappointed by SD card reliability. It seems even a little wear and the whole thing slows down massively and eventually throws read/write errors. I would like to see a card design which, instead of failing when there is flash wear, instead just gets smaller. For compatibility with existing OS's, that would take the form of a self-partition-resizing sd card. It would understand fat32, ext3, NTFS, afs etc,…

[deleted]

Re: SDcard wear leveling and translation layers (2014)

#38

Post by st_goliath »

SideQuark wrote: post #30
st_goliath wrote: post #18

Earlier quoted context omitted.

You cannot do single byte writes on NAND. You have erase blocks (typically several 100k to M range) that are divided into pages (typically a few to several dozen k), some devices have sub-pages but that's about it, that's the smallest unit of data you can write. If you attempt to clear bits from 1 to 0 on an already written page, it will generally not work. I say generally , because I have actually tried this using r…

>> Couldn't you perform a file append without the need for a block erase? > You cannot do single byte writes on NAND. This isn't true - that the OP wrote here certainly does work (at least for many places I've tested and used it). I've used it to do all sorts of low level logging in small devices (and I've thoroughly tested many, many chips to complete destruction to see how things degrade). I have not yet seen a pla…

I was talking about bare NAND flash, not eMMC or anything otherwise "fancy" with builtin FTL. What you say is certainly true for those.

None of the projects you describe sound "cheap" or like they involve devices first-and-foremost optimized for cost, and are unlikely to involve such dirt-cheap, raw NAND devices that are becoming increasingly rare even in the super-cheap embedded space.

The original context of the discussion (see: a few comments up) is on-die NAND flash, inside SD cards, which TFA is about.

The flash chips I brought up for comparison were from an extremely cost-optimized, embedded device that I worked on ~2016. The more expensive one already being on the upper end, regarding smarts & cost, but still exposing the flash directly (plus ECC & bad block management, but no FTL).

I mostly work with industrial devices, and also maintain the Linux user space for raw flash (mtd-utils). I have seen my fair share of broken flash devices over the years (often in-field) and also have a my own little pile of broken industrial SD cards on my desk :)

Re: SDcard wear leveling and translation layers (2014)

#39

Post by formerly_proven »

faragon wrote: post #22

Why there's not generalized discard/trim capabilities on USB/SD devices? It should be "easy" to agree some standard on that.

Even more strangely, proprietary SxS/XQD cards also don't support discard/trim, neither for devices nor from a PC. Only the more recent CFX cards support trim (most of them, but not with every reader, and formatting in a camera doesn't seem to issue them, perhaps to aid with data recovery in case of accidental formats).
Post reply on HN