Considerations when building embedded databases
interrupt.memfault.com
Considerations when building embedded databases
1–10 of 18 posts
Re: Considerations when building embedded databases
#2Since it’s not clear from the title: embedded here does not mean sqlite-inside-your-app but small devices with microcontrollers
Re: Considerations when building embedded databases
#3Since it’s not clear from the title: embedded here does not mean sqlite-inside-your-app but small devices with microcontrollers
That's what I initially thought too, though having read and understood little of the article, is sqlite not a consideration too? Or does it take up too much memory?
Re: Considerations when building embedded databases
#4Since it’s not clear from the title: embedded here does not mean sqlite-inside-your-app but small devices with microcontrollers
That's what I initially thought too, though having read and understood little of the article, is sqlite not a consideration too? Or does it take up too much memory?
SQLite would require a file system. Embedded systems typically do not have this.
Re: Considerations when building embedded databases
#5The chosen solution isn’t ideal for all cases, especially those with many or large records. It doesn’t require parsing all the bytes of a payload but it does require reading them (because it’s a sequential scan of all content).
A better solution would separate the keys from the values (à la MFT) because reading a page of flash is going to be the slowest step. If you only have to sequentially scan the header table, you have to read an order of magnitude or more less data to find the record you are searching for.
Re: Considerations when building embedded databases
#6Earlier quoted context omitted.
That's what I initially thought too, though having read and understood little of the article, is sqlite not a consideration too? Or does it take up too much memory?
SQLite would require a file system. Embedded systems typically do not have this.
SQLite doesn’t require a file system per se, as you can create an in-memory db and then use vfs to load/write the actual data from/to your IO layer.
Re: Considerations when building embedded databases
#7Earlier quoted context omitted.
That's what I initially thought too, though having read and understood little of the article, is sqlite not a consideration too? Or does it take up too much memory?
SQLite would require a file system. Embedded systems typically do not have this.
Though note you can define your own "Virtual File System" (VFS) for SQLite to use[1]. It might be a bit of an undertaking, but I think you could use that to run on just about anything with storage you control. (I haven't done it myself, just some research back in the day)
Re: Considerations when building embedded databases
#8The chosen solution isn’t ideal for all cases, especially those with many or large records. It doesn’t require parsing all the bytes of a payload but it does require reading them (because it’s a sequential scan of all content). A better solution would separate the keys from the values (à la MFT) because reading a page of flash is going to be the slowest step. If you only have to sequentially scan the header table, yo…
A columnar database format might perform well out of the box in that case.
Reading more, it seems like a columnular database built on LSM trees would probably work great, since you could easily move the log around in flash to distribute write wear out, and only occasionally compact LSM tree files.
Re: Considerations when building embedded databases
#9The chosen solution isn’t ideal for all cases, especially those with many or large records. It doesn’t require parsing all the bytes of a payload but it does require reading them (because it’s a sequential scan of all content). A better solution would separate the keys from the values (à la MFT) because reading a page of flash is going to be the slowest step. If you only have to sequentially scan the header table, yo…
A columnar database format might perform well out of the box in that case. Reading more, it seems like a columnular database built on LSM trees would probably work great, since you could easily move the log around in flash to distribute write wear out, and only occasionally compact LSM tree files.
It would, but there are any small and simple enough to use in a standalone (no-OS, non-POSIX) environment?
Re: Considerations when building embedded databases
#10> However, if you compile on an AWS EC2 instance (as you may want to do if there is a cloud component to your fishtank), then you get a 16-byte structure because int there is 8 bytes
I'm not familiar with any OS that EC2 would offer that uses an 8-byte int. A better example might be that long is the size of a pointer on Unix and microcontroller systems, but always 4 bytes on Windows.