Live data from Hacker News

Theory crafting a system for 1000 simultaneous micro SD card ingests

forum.level1techs.com

21–30 of 38 posts

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#21
Such a nerd bait request. I’m like “wait, how long between downloads of cards do you need? How many people will be placing these cards in? How will you know when one is downloaded??”

Only reading the comments did I think “Wait, who needs this and why?” I like the Mr. Beast angle - it’s fun to think about if nothing else.

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#22
One use case is to mount 360 cams on a bunch of motorcycles (fleet) and do a google streetview equivalent. You build a city level streetview pretty quickly.

I think at that point you'd just need to do 1 pc every 200 readers and pull the videos to a local server for processing. There's no magic here, just work.

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#23

I know it sounds boring, but did anyone suggest a second computer if this is an actual problem and not just a for fun challenge?

Or how about 999 additional computers? They could be really tiny ones, running some kind of operating system designed for putting inside small single purpose machines. You could also add a coloured light on the front of each one and a button that says "copy to NAS".

Or each SD-card-reader node could copy to its own hard drive (to free things up for another card) and asynchronously upload from there to the central unit... Or possibly send metadata/thumbnails first, and then alternate between sending specifically-requested content versus uploading the backlog.

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#24
So, the 1000 card problem is pretty easy. What matters is how much is on each card, and how long the DL window is.

For instance, if you can spend 2 hours downloading each card, (SPI to an esp32 can do about 32GB an hour) so you could Download 64GB in 2 hours via ESP32 boards with SPI uSD slots, pushed onto your server over wifi6.

If that’s enough, that’s a simple solution. If you need more than 32GB an hour per card, I’ve seen some adapters that hold 10? uSD cards and connect over SATA, so that might get you a better way into the bus than typical USB?

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#25
post #10

The physical interface itself is going to be a big deal, people are going to be just swapping cards constantly and you'll need multiple people just doing this. Status of read is a big part of this as well so some amount of custom LEDs and such for status of ingress will be required. What a horrible job being in a room with 100s of thousands of SD cards just feeding them into slots and then into an empty bin! Its poss…

Would be a straightforward job for a pick and place style robot, if you had cycle time to budget for it.

[deleted]

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#26

The physical interface itself is going to be a big deal, people are going to be just swapping cards constantly and you'll need multiple people just doing this. Status of read is a big part of this as well so some amount of custom LEDs and such for status of ingress will be required. What a horrible job being in a room with 100s of thousands of SD cards just feeding them into slots and then into an empty bin! Its poss…

Perhaps some kind of "SD-card multiplexer" arduino project, where you have 50 SD breakout boards all mounted in a tall stack, each with an LED, and maybe a "reset" button if it doesn't detect ejections. Some circuitry in the middle would set each LED status color, and switch which of the 50 is currently connected to a single set of access pins.

It still involves annoying work by humans, but it would be much more straightforward: See a bay that is showing the "idle" color, remove any finished card and put it in the finished-box, grab a card from the to-do box and put it in, then press the button to signal a new card is waiting.

If that process took 6 seconds, 1000 cards would mean 100 minutes of tedious work, but you can spread it over multiple people or (since the card-read takes time) do it in short bursts interspersed with other activities.

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#27
post #24

So, the 1000 card problem is pretty easy. What matters is how much is on each card, and how long the DL window is. For instance, if you can spend 2 hours downloading each card, (SPI to an esp32 can do about 32GB an hour) so you could Download 64GB in 2 hours via ESP32 boards with SPI uSD slots, pushed onto your server over wifi6. If that’s enough, that’s a simple solution. If you need more than 32GB an hour per card,…

I was thinking SPI access from a mcu would be the way to go. While the individual data rate would be fairly low, running a lot of them at the same time would probably add up.

If they just need the slots to plug everything in and leave overnight you could probably just run off some PCBs with heaps of card slots per mcu and let it cycle through them one after another

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#28
post #6

There is only one scenario that remotely makes sense for why this would be even necessary and that's fleet vehicles with dash cams. Even in that case, there are way better approaches because there are dashcams designed for fleet usage that upload to central servers over WiFi. Basically this scenario only makes sense if there is something horribly wrong elsewhere to the point that the only reasonable thing to do is fi…

Bodycams for any number of security orgs (military, police, security guard) in any number of places around the world? I mean, if done right you'd use some kind of wireless-while-they-charge at the end of the day. But given the wide world I could easily see someone somewhere going down this path.

This was my bet too. Sounds more like bodycam work than dashcam or "film production". NDA may be because they work for law enforcement.

Re: Theory crafting a system for 1000 simultaneous micro SD card ingests

#30
I hope they have a robot for physically handeling the SD card insertions and retrieval. Even then, lets say you can load or swap 1 card per second, by the time you are dealing with the 200th card, would the first one not already be ingested?

Seems like more of a scale out than a scale up problem, with a view on the whole operation from recording footage down to final archive to tackle the real bottleneck.

Post reply on HN