Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SDcard wear leveling and translation layers (2014)

msreekan.com

Post reply on HN
Print view View original

61–70 of 72 posts

Re: SDcard wear leveling and translation layers (2014)

#61

Post by 01HNNWZ0MV43FF »

marcosdumay wrote: post #43
01HNNWZ0MV43FF wrote: post #32

Earlier quoted context omitted.

That could explain why my dashcam is working fine after ten years whereas my previous job's IoT devices regularly ate SD cards. (Then again, the IoT devices were also overheating badly whereas the dashcam was designed for its purpose, kept cool while running, and runs on a low duty cycle) What are you supposed to do, then? Buffer up writes and accept that you'll lose logs a minute or so before a power outage? As much…

Losing the logs on a power outage may be a good compromise. Alternatively, you can buffer on a battery-powered RAM unit, accept that your cards will be short-lived, immediately send the logs through some network, not log... When you are designing the hardware, you have lots and lots of choices.

Yeah. This company wasn't big enough to entertain battery backups, and the product was not expensive enough to add hardware that didn't bring in more sales.

And we logged too much to pull all the logs over WAN all the time, resulting in a clever hack for remotely browsing logs with (relative) security

Honestly we could have batched up the logs.

Re: SDcard wear leveling and translation layers (2014)

#62

Post by st_goliath »

SideQuark wrote: post #55
st_goliath wrote: post #38

Earlier quoted context omitted.

I was talking about bare NAND flash, not eMMC or anything otherwise "fancy" with builtin FTL. What you say is certainly true for those. None of the projects you describe sound "cheap" or like they involve devices first-and-foremost optimized for cost, and are unlikely to involve such dirt-cheap, raw NAND devices that are becoming increasingly rare even in the super-cheap embedded space. The original context of the di…

> I was talking about bare NAND flash Same here. > None of the projects you describe sound "cheap" or like they involve devices first-and-foremost optimized for cost A lot of them are low cost PIC microcontrollers, which are super cheap, and the lowest cost MCUs were chosen precisely because at large commercial sale scale saving pennies on HW is very important. The reason we went to such lengths to reuse bits at the…

> low cost PIC microcontrollers

ahhh, I think there's the source of our confusion: I suspect you are actually talking about NOR flash?

NOR flash is physically organized different from NAND. It typically does allow single word/byte or sometimes even bit level accesses, like you described. NOR flash is also much more reliable, often not needing any wear leveling at all. The downside is that NOR flash has orders of magnitudes lower data density.

Microcontrollers would use NOR flash internally for program memory, precisely because it allows direct, single-cycle access to individual opcodes. With NAND flash, program memory would have to be read page-wise and require some sort of instruction cache in between. Other EEPROM-like devices (e.g. AT28Fxxx) also use NOR flash for pretty much the same reasons.

On some embedded Linux devices, NOR flash is used for config/boot loader/kernel, while the rootfs lives on NAND.

Re: SDcard wear leveling and translation layers (2014)

#63

Post by goodburb »

robotnikman wrote: post #53
2Gkashmiri wrote: post #48

Earlier quoted context omitted.

Have an SD card that was copying file on to and in the middle of transfer the card went kaput and hasnt seen since. No machine I try it on even acknowledges it existence. The card is almost a decade old at this point but is there a way to recover data ?

Best to consult a data recovery specialist, will cost you an arm and a leg though. 10 years without power makes me think most of it wont be recoverable even if they can get to the data on it, but then again it depends on a lot of factors when it comes to flash memory.

IIRC SD cards and most SSDs don't go through error check routine when powered, it has to be specified. Some do use the power-on time and run a routine if left on for too long.

You usually you have to read the entire drive to maintain, in my case S.M.A.R.T. reported changes to reserve blocks after doing that on an old drive.

I have a 16 year old 4GB SDcard with all data still intact.

Re: SDcard wear leveling and translation layers (2014)

#64

Post by mystified5016 »

2Gkashmiri wrote: post #48
st_goliath wrote: post #4

At a conference, I once met somebody who said his team had the joy of dismantling broken SD cards and figuring out why they broke. One of the anecdotes I distinctly recall was about a batch of cards that (for obvious cost saving reasons) used the actual data flash to store its own firmware as well. IIRC due to a bug in the wear leveling accounting, once the card got sufficiently full, the wear leveling code ended up…

Have an SD card that was copying file on to and in the middle of transfer the card went kaput and hasnt seen since. No machine I try it on even acknowledges it existence. The card is almost a decade old at this point but is there a way to recover data ?

Sometimes you can recover data over the SPI interface instead of the proprietary SD interface. Essentially, plug it into an arduino with one of the many SD card libraries and slowly dump the data over the serial link.

I'm unsure if this is just urban legend or not, though. I've never tried myself.

Re: SDcard wear leveling and translation layers (2014)

#65

Post by mystified5016 »

michaelt wrote: post #10

> For example, lets consider that we wrote to the sector 1024 on the SDCARD, and this address got initially mapped to physical sector 1024 itself on flash. Later we overwrite the contents, but this time write cannot go to the same physical address because it needs to be erased before programmed again. Maybe someone here will know the answer to something I've been wondering for a while: When appending to a file with w…

It sort of depends, but in general flash cannot write a page without erasing it first. Sometimes you can set, but not un-set bits (can be a 0 or 1 depending, usually 0 is 'set') without erasing a page.

At the lowest level this means to update a page you have to first read it all back out, apply your modifications, then write back in a single operation. You must always write the full page size.

In high level contexts like SD cards or SSDs being driven by an operating system, all of this is handled transparently in the background. If you're using raw flash in an embedded context, you usually have to manage this yourself (or let a library do it for you).

Re: SDcard wear leveling and translation layers (2014)

#66

Post by goodburb »

mystified5016 wrote: post #64
2Gkashmiri wrote: post #48

Earlier quoted context omitted.

Have an SD card that was copying file on to and in the middle of transfer the card went kaput and hasnt seen since. No machine I try it on even acknowledges it existence. The card is almost a decade old at this point but is there a way to recover data ?

Sometimes you can recover data over the SPI interface instead of the proprietary SD interface. Essentially, plug it into an arduino with one of the many SD card libraries and slowly dump the data over the serial link. I'm unsure if this is just urban legend or not, though. I've never tried myself.

You can also tape over the pins to force 1-bit SDIO mode on most readers.

Re: SDcard wear leveling and translation layers (2014)

#67

Post by anthk »

st_goliath wrote: post #38
SideQuark wrote: post #30

Earlier quoted context omitted.

>> Couldn't you perform a file append without the need for a block erase? > You cannot do single byte writes on NAND. This isn't true - that the OP wrote here certainly does work (at least for many places I've tested and used it). I've used it to do all sorts of low level logging in small devices (and I've thoroughly tested many, many chips to complete destruction to see how things degrade). I have not yet seen a pla…

I was talking about bare NAND flash, not eMMC or anything otherwise "fancy" with builtin FTL. What you say is certainly true for those. None of the projects you describe sound "cheap" or like they involve devices first-and-foremost optimized for cost, and are unlikely to involve such dirt-cheap, raw NAND devices that are becoming increasingly rare even in the super-cheap embedded space. The original context of the di…

I remember the change from SLC to MLC at the PocketCHIP's UBIFS. Then it soft-bricked itself a lot.

Re: SDcard wear leveling and translation layers (2014)

#68

Post by Sakos »

robotnikman wrote: post #54
Sakos wrote: post #7

Earlier quoted context omitted.

Man, I'd love to work in that team for a year. I've had so many SD cards fail on me (most recently a 1TB SanDisk Extreme which was used in my Steam Deck) and it feels like most of them are junk.

IIRC the denser the memory on the cards, the more likely problems are to arise. I forget the specifics but it has to do with the physics involving electronics on such a small scale. That's why industrial grade cards tend to be in much smaller capacities, but they are much longer lasting.

I wish we'd move away from microSD cards to a larger format if it meant more durable storage. I hate these things. I just end up keeping them in an SDcard adapter when they're not in active use so I don't lose them.

USB sticks seem to have the same issue with durability though. I've also had multiple of those fail on me. I'm currently eyeing those SSD USB sticks and I wonder how much more durable they might be.

Re: SDcard wear leveling and translation layers (2014)

#69

Post by 2Gkashmiri »

robotnikman wrote: post #53
2Gkashmiri wrote: post #48

Earlier quoted context omitted.

Have an SD card that was copying file on to and in the middle of transfer the card went kaput and hasnt seen since. No machine I try it on even acknowledges it existence. The card is almost a decade old at this point but is there a way to recover data ?

Best to consult a data recovery specialist, will cost you an arm and a leg though. 10 years without power makes me think most of it wont be recoverable even if they can get to the data on it, but then again it depends on a lot of factors when it comes to flash memory.

10 years in use. On a phone.

Re: SDcard wear leveling and translation layers (2014)

#70

Post by robotnikman »

Sakos wrote: post #68
robotnikman wrote: post #54

Earlier quoted context omitted.

IIRC the denser the memory on the cards, the more likely problems are to arise. I forget the specifics but it has to do with the physics involving electronics on such a small scale. That's why industrial grade cards tend to be in much smaller capacities, but they are much longer lasting.

I wish we'd move away from microSD cards to a larger format if it meant more durable storage. I hate these things. I just end up keeping them in an SDcard adapter when they're not in active use so I don't lose them. USB sticks seem to have the same issue with durability though. I've also had multiple of those fail on me. I'm currently eyeing those SSD USB sticks and I wonder how much more durable they might be.

The regular SD card form factor seems great when it comes to more durable storage, but obviously would not be ideal for smaller devices.

I've heard that many USB sticks use lower quality binned chips so that may be why they usually fail more often despite their size, but this is just anecdotal from what I've heard.

Post reply on HN