Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The design of littlefs: A fail-safe filesystem designed for microcontrollers

github.com

Post reply on HN
Print view View original

21–30 of 74 posts

Re: The design of littlefs: A fail-safe filesystem designed for microcontrollers

#21

Post by delfinom »

kartoffelwaffel wrote: post #7
technothrasher wrote: post #6

"Usually, microcontroller code is simple and reactive, with no concept of a shutdown routine." Really? Nobody uses brownout interrupts to lock down on power loss? I almost always do a few things as the power is fading, flagging the situation at the very least in order to clean up when power is restored.

Also why not chuck a capacitor in the mix to provide a little power buffer to shutdown the filesystem gracefully.

Many MCUs do not appreciate slow rise-times when using their internal regulators. Then you start oversizing your regulator or providing them replacement regulators. Which then opens more cans of worms, etc, etc.

Re: The design of littlefs: A fail-safe filesystem designed for microcontrollers

#22

Post by delfinom »

heyiforgotmypwd wrote: post #2

Why not TFAT or TexFAT?

This is meant to run even on low-end micros without need for memory allocation. And support small SPI memory chips where you need to roll your own wear leveling across different size blocks of memory depending on chip vendor.

Re: The design of littlefs: A fail-safe filesystem designed for microcontrollers

#23

Post by londons_explore »

delfinom wrote: post #20
technothrasher wrote: post #6

"Usually, microcontroller code is simple and reactive, with no concept of a shutdown routine." Really? Nobody uses brownout interrupts to lock down on power loss? I almost always do a few things as the power is fading, flagging the situation at the very least in order to clean up when power is restored.

Nope. Because brown-out detectors on micros are meant for momentary glitches and unreliable. The detectors can preserve a brown out detected bit in their reset cause registers. However, they don't guarantee the micro's core will get the interrupt in time before losing power. And if the brown-out goes down to POR levels, you won't even get the brown out cause bit and instead POR. The best you can do for file systems i…

You design the circuit... You can decide exactly how many microseconds of power you'll get between the brownout interrupt occurring and the lowest operating voltage. Your job as designer is to write the code to guarantee completion in that time.

Microcontrollers don't have to suffer the same issues as app/web developers. They can know exactly how many clock cycles something will take.

Re: The design of littlefs: A fail-safe filesystem designed for microcontrollers

#24

Post by geky »

geocar wrote: post #15

> and the unfortunate fact is you can't traverse a tree with constant RAM This is not true. See Knuth, The Art of Computer Programming Vol.1, § 2.3.5). Generally I wouldn't recommend naive SDW as a good strategy for filesystems, but if you have a fixed number of processes (common in microcontroller applications) it might be worth trying.

Reading through 2.3.5 (Lists and Garbage Collection), it looks like it's describing a tree where the leaf nodes also contain references to the root of the next tree.

This wouldn't work as a copy-on-write tree, as the root node is changed every write.

Copy-on-write behavior requires very strict trees, and this gets in the way of adding extra structures for traversal.

Re: The design of littlefs: A fail-safe filesystem designed for microcontrollers

#25

Post by Majromax »

the8472 wrote: post #5

Afaict checksums only cover metadata and it doesn't do any read integrity checking, isn't that bad when talking to raw flash devices since it relies on them accurately reporting errors?

That's mentioned towards the end of the link:

> littlefs by itself does not provide ECC. The block nature and relatively large footprint of ECC does not work well with the dynamically sized data of filesystems, correcting errors without RAM is complicated, and ECC fits better with the geometry of block devices.

That is, ECC and read-integrity checking is determined at the block level, not the filesystem level. The filesystem assumes that it's dealing with a block device that either succeeds or reports an error.

Re: The design of littlefs: A fail-safe filesystem designed for microcontrollers

#26

Post by geky »

_urga wrote: post #18

> Logging filesystem are beautifully elegant. With a checksum, we can easily detect power-loss and fall back to the previous state by ignoring failed appends. An edge case when designing a log-structured file system is that a corrupt checksum in a log entry could actually mean one of three things: power loss while writing the last transaction, a misdirected write or bitrot, or a misdirected read. If you simply read t…

I'm not sure I'm following this snap operation. What is a "log tail" in this context? Does this the snap operation still work if we are copying over entries of the log lazily and we don't have a definite end-of-log?

You are right though, a checksum used this way provides no protection over bit errors or misdirected write/reads. But, if you assume no bit errors, a checksum can provide power-loss protection as long as there's a fallback.

But doesn't this just move the problem somewhere else? Yes. In this case it moves error detection onto the block device. But it turns out performing error detection/correction at the block device level is simpler and more effective. Most NAND flash components even have built-in ECC hardware for this specific purpose.

Re: The design of littlefs: A fail-safe filesystem designed for microcontrollers

#27

Post by ianhowson »

londons_explore wrote: post #23
delfinom wrote: post #20

Earlier quoted context omitted.

Nope. Because brown-out detectors on micros are meant for momentary glitches and unreliable. The detectors can preserve a brown out detected bit in their reset cause registers. However, they don't guarantee the micro's core will get the interrupt in time before losing power. And if the brown-out goes down to POR levels, you won't even get the brown out cause bit and instead POR. The best you can do for file systems i…

You design the circuit... You can decide exactly how many microseconds of power you'll get between the brownout interrupt occurring and the lowest operating voltage. Your job as designer is to write the code to guarantee completion in that time. Microcontrollers don't have to suffer the same issues as app/web developers. They can know exactly how many clock cycles something will take.

Except that:

(a) you don't choose how long you get because the uC manufacturer already chose. (An external brownout warning is different, and useful, but still not a good solution to the problem of FS integrity.)

(b) those thresholds have nothing to do with the operating thresholds of the flash (which usually require a lot more energy to do work than the uC)

(c) writing to flash is a relatively high-power operation, and if you need to erase, you're doing the highest-power operation at a time when you have no power buffer

(d) warmup time to do an erase is measured in milliseconds, and you probably don't have that much power buffer

The best you can do at brownout is minimize the damage, because your uC is about to switch off and/or silently inject errors.

Re: The design of littlefs: A fail-safe filesystem designed for microcontrollers

#28

Post by craftinator »

ianhowson wrote: post #27
londons_explore wrote: post #23

Earlier quoted context omitted.

You design the circuit... You can decide exactly how many microseconds of power you'll get between the brownout interrupt occurring and the lowest operating voltage. Your job as designer is to write the code to guarantee completion in that time. Microcontrollers don't have to suffer the same issues as app/web developers. They can know exactly how many clock cycles something will take.

Except that: (a) you don't choose how long you get because the uC manufacturer already chose. (An external brownout warning is different, and useful, but still not a good solution to the problem of FS integrity.) (b) those thresholds have nothing to do with the operating thresholds of the flash (which usually require a lot more energy to do work than the uC) (c) writing to flash is a relatively high-power operation,…

But you can just... Decide how many clock cycles you'll need to finish shutdown, calculate that power usage, then solder a corresponding power source in place. There really isn't guesswork.

Re: The design of littlefs: A fail-safe filesystem designed for microcontrollers

#29

Post by geky »

the8472 wrote: post #19
evancox100 wrote: post #16

Earlier quoted context omitted.

Microcontrollers almost always use NOR flash, which is much less error prone than the NAND flash in your ssd. Still, if you need ECC (maybe it's a medical application), the company designing the MCU would build it directly in to the embedded flash. The foundry might have a standard 128-bit word size option, and an extended 144-bit word size option, giving you 16 extra bits per word for whatever ECC or secded code you…

I was talking about checksumming, not error correction. Since the meta-data blocks are already covered by CRC32 I wonder why the data blocks aren't.

Error correction provides the same value as a checksum, but better. (the tradeoff is ECC codes are much larger and more expensive to compute)

It's also worth noting the CRC is used for power-loss and doesn't actually provide error detection for metadata-blocks.

Checksumming data is a bit complicated in a file system, mostly because of random file writes. If you write to the middle of a file, you will need to update a CRC, but updating that CRC may require reading quite a bit of additional data to build the CRC back up.

To make random writes efficient, you could slice up the files, but then that raises the question of how large to make the slices. Too large and random file writes are expensive, too small and the overhead of the CRCs gets costly.

You could make these slices configurable, but at this point we've kinda recreated the concept of a block device.

The block device representing the underlying storage already has configuration for this type of geometry: erase size/program size/read size. If we checksum (or ECC) at the block device level we also get the added benefit of also protecting metadata blocks. Most NAND flash components already have hardware ECC for this specific purpose.

TLDR: It's simpler and more effective to checksum at the block device level.

And for MCU development, simpler == less code cost.

Re: The design of littlefs: A fail-safe filesystem designed for microcontrollers

#30

Post by neltnerb »

ianhowson wrote: post #27
londons_explore wrote: post #23

Earlier quoted context omitted.

You design the circuit... You can decide exactly how many microseconds of power you'll get between the brownout interrupt occurring and the lowest operating voltage. Your job as designer is to write the code to guarantee completion in that time. Microcontrollers don't have to suffer the same issues as app/web developers. They can know exactly how many clock cycles something will take.

Except that: (a) you don't choose how long you get because the uC manufacturer already chose. (An external brownout warning is different, and useful, but still not a good solution to the problem of FS integrity.) (b) those thresholds have nothing to do with the operating thresholds of the flash (which usually require a lot more energy to do work than the uC) (c) writing to flash is a relatively high-power operation,…

I don't personally usually bother, but an EE approach would be to put a diode between the 3.3V supply and the MCU/flash power supply pins with a large bulk capacitor, and use the voltage just before the diode as your brownout detection. It can't discharge back into the power supply or other parts on the board because of the diode so the capacitor just needs to store enough charge to run for a specified time after power loss.

Then you can configure how long the MCU runs after power shuts down. Put 470uF there and I imagine you'd have your milliseconds easily.

Post reply on HN