Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Theory crafting a system for 1000 simultaneous micro SD card ingests

forum.level1techs.com

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#31

Post by aa-jv »

At that scale, wouldn't it make sense to just have raw block i/o copies being done to a disk image, and then mount the image onboard to do the actual data transfers?

Seems to me a majority of the overhead with 1000 SD cards is in the filesystem translation layer, multiplied by the USB contention dealing with so many devices.

So if it were me, I'd have a big fat, fast mounted filesystem whose purpose is only to serve as a dd destination, get the dd done as quickly as possible with system bus buffer sizes, and then do the data transfer 'offline' once the SD has been mirrored...

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#32

Post by tetha »

PaulKeeble wrote: post #7

The physical interface itself is going to be a big deal, people are going to be just swapping cards constantly and you'll need multiple people just doing this. Status of read is a big part of this as well so some amount of custom LEDs and such for status of ingress will be required. What a horrible job being in a room with 100s of thousands of SD cards just feeding them into slots and then into an empty bin! Its poss…

In most cases I've seen scales similar to this - body cams in security, scanners at postal services and such, the physical part is offloaded to the individuals working with the thing: at the end of the day, you put your thing into a charger, or into a box and connect it up or something like that.

So from there, my thought is indeed: Why shouldn't we get 1000 micro sd card readers (or multi readers), attach those to a bunch of PIs streaming the cards to a central server, have people put their SD card in there at the end of a day? After that, you can kick off the reads in a batch, re-seat the cards that aren't set correctly and that's it.

One could reduce hardware cost by having some LED turn on or off if a card in a reader can be swapped, but with a thousand people doing this, complicated instructions tend to be a bad thing. Unless the transfer is fast enough to wait for it.

But it's always fun how complicated some of these physical problems become if you scale them up. Like, consider how many storm drains are out there in a city. If each needs to be touched once every 3 years... suddenly you've created a dozen jobs.

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#33

Post by mystified5016 »

I'd go in a completely different direction from this.

I'm assuming the SD cards come from a fleet of dashcams or similar, and that the driver is responsible for turning in their card at the end of a run or whatever.

I'd deploy a fleet of tiny WiFi card readers. Just an ESP32 and a card slot. Each individual unit is not fast, but if you have a few dozen units running at once you could easily saturate the dedicated ingest WiFi network at each hub location. The readers are dirt cheap and the only supporting infrastructure required is WiFi and a bunch of 5v power adapters.

You'd have a rack of these guys next to the key locker or timeclock. Pick up your keys and a blank card, drop off your keys and card at the end of the shift. Could even get extra fancy and use LEDs to indicate which card is yours for the day, or flip on an error light when the card is worn out.

You probably don't need each video to transfer as fast as possible, more likely you just want maximum overall throughput. So just build a bunch of very cheap and slow nodes. On aggregate you should get decent performance.

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#34

Post by reginald78 »

mystified5016 wrote: post #33

I'd go in a completely different direction from this. I'm assuming the SD cards come from a fleet of dashcams or similar, and that the driver is responsible for turning in their card at the end of a run or whatever. I'd deploy a fleet of tiny WiFi card readers. Just an ESP32 and a card slot. Each individual unit is not fast, but if you have a few dozen units running at once you could easily saturate the dedicated ing…

To bad they don't make micro Eye-Fi cards.

https://en.wikipedia.org/wiki/Eye-Fi

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#35

Post by imtringued »

PeterStuer wrote: post #30

I hope they have a robot for physically handeling the SD card insertions and retrieval. Even then, lets say you can load or swap 1 card per second, by the time you are dealing with the 200th card, would the first one not already be ingested? Seems like more of a scale out than a scale up problem, with a view on the whole operation from recording footage down to final archive to tackle the real bottleneck.

Swapping is dumb. Just wire pogo pins to the SD card reader and put the SD cards upside down.

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#36

Post by K0balt »

Lerc wrote: post #27
K0balt wrote: post #24

So, the 1000 card problem is pretty easy. What matters is how much is on each card, and how long the DL window is. For instance, if you can spend 2 hours downloading each card, (SPI to an esp32 can do about 32GB an hour) so you could Download 64GB in 2 hours via ESP32 boards with SPI uSD slots, pushed onto your server over wifi6. If that’s enough, that’s a simple solution. If you need more than 32GB an hour per card,…

I was thinking SPI access from a mcu would be the way to go. While the individual data rate would be fairly low, running a lot of them at the same time would probably add up. If they just need the slots to plug everything in and leave overnight you could probably just run off some PCBs with heaps of card slots per mcu and let it cycle through them one after another

With WiFi mcus below a dollar, no need to double up even.

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#37

Post by Lerc »

K0balt wrote: post #36
Lerc wrote: post #27

Earlier quoted context omitted.

I was thinking SPI access from a mcu would be the way to go. While the individual data rate would be fairly low, running a lot of them at the same time would probably add up. If they just need the slots to plug everything in and leave overnight you could probably just run off some PCBs with heaps of card slots per mcu and let it cycle through them one after another

With WiFi mcus below a dollar, no need to double up even.

I dunno, Have you ever operated 1000 WiFi devices in the one room?

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#38

Post by K0balt »

Lerc wrote: post #37
K0balt wrote: post #36

Earlier quoted context omitted.

With WiFi mcus below a dollar, no need to double up even.

I dunno, Have you ever operated 1000 WiFi devices in the one room?

It’s all 2.4, so you you can cook in there just fine.

But seriously that probably would be around 50w of tx power lol.

You could probably sort it with 10 good APs

But yeah, it’s going to be an RF star for sure.

Post reply on HN