71–80 of 166 posts
Wow, so maybe instead of a node_modules folder, npm should use a node_modules.db?
This sounds like a fantastic idea. You would go from 4000 directories and 50000 files to a single file with b-tree indices, hashes, etc. You could probably get a 100x speedup over the current approach.
Next week on Show HN: `sqlitefs`, the SQLite-backed FUSE filesystem?
already exists
The fact that SQLite is much faster than file systems is not surprising at all. But there’s one surprising result there. > All machines use SSD except Win7 which has a hard-drive. And yet Win10 is much slower than Win7 in all tests (note they have latency on the graphs). How’s that possible? For a drive spinning at 5400 RPM, average random read latency can’t be smaller than 2.2ms, this is how long it takes to rotate…
There's the other obvious explanation that because the Win7 machine has slower storage it automatically uses a larger cache than the Win10 machine that more than offsets the difference.
I think I know what’s going on. They don’t publish absolute numbers, they only publish result compared to SQLight on the same machine. Windows has approximately fixed overhead when opening files. Most of that overhead goes to access checks, NTFS security descriptors are way more sophisticated than 9 bits of security in Linuxes. With a fast disk that overhead dominates in the cost, SQLight doesn’t open any files while running the test, the DB already opened at startup. With a slow disk, that overhead is masked by large IO cost which affects both SQLight and NTFS.
Earlier quoted context omitted.
Scheme could be something like; FileId, VersionId, Name, Data, UploadedTimestamp, UploadedBy, ... If you’re on AWS, pushing and pulling from an S3 bucket is probably a great solution, and then of course there’s nothing to worry about in terms of backups. Do you still need to keep an index of the files / metadata in a DB, or can you tag everything you need directly on the S3 objects and just pull the whole bucket?
Even if you’re not in AWS S3 works great, especially where you need to serve content direct to client’s browsers. S3 supports either making a file public and available to all, or using pre-signed URLs to provide short term authorisation to access a file to specific clients. No need to pull the bucket at all.
Earlier quoted context omitted.
Which is why for decades many dB offerings offered raw or cooked storage. Cooked being that they used the file system and raw being using the device directly. I'm not sure of the upsides and downsides today as I last worked in that area about 20 years ago, but was very much a case of raw a noticeable performance gain. But then you are equally tied into the vendors backup mechanisms.
I ran a large at the time Informix implementation in the late 90s/early 2000s. Raw disk vs file system (Solaris UFS) performance was 35% faster on our transaction systems.
Wow, so maybe instead of a node_modules folder, npm should use a node_modules.db?
For small- to mid-sized projects, I’ve always realized huge gains in simplicity by haves “Files” tables to store various assets. It means instances in a web-farm can pull the files down when they initialize easily, it means files are automatically versioned, it provides an obvious place to put the files when they are being uploaded on the Admin panel. It means all the files are getting backed up as part of the databa…
>Really the only downside I ever found — at the scale I was operating at — was that it bloats the database backups. Easily solved by having two separate databases. One for dynamic content, one for static files, which is probably good practice regardless.
Earlier quoted context omitted.
>Really the only downside I ever found — at the scale I was operating at — was that it bloats the database backups. Easily solved by having two separate databases. One for dynamic content, one for static files, which is probably good practice regardless.
Doesn't that introduce the desync problem again? Is there something that can enforce consistency across the two databases?
Earlier quoted context omitted.
Personal opinion: Those things (static assets) belong in a container with Nginx serving them (and that’s not even counting SAAS options like S3). Easy to integrate with CDNs, easy to update, easy to roll out and back. And you get to take advantage of layers to keep the network and disk usage minimal.
I don’t see how a container solves the backing store / source of truth problem? How are changes getting put into the container? Are you versioning the container each time a file is changed or added? When files change are all instances of the containers being restarted to get the new/changed files?