51–60 of 456 posts
I talked to people at AWS who work in RDS Aurora and they hinted they use S3 internally as a backend for MySQL and PostgreSQL.
Big if true. That was definitely not in the AWS cert I took lol.
* https://www.amazon.science/publications/amazon-aurora-design... * https://d1.awsstatic.com/events/reinvent/2019/REPEAT_Amazon_...
My big pet peeve is AWS adding buttons in the UI to make "folders". It is also a fiction! There are no folders in S3. > When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to suppor…
Is that really so different from how folders work on other systems? A directory inode is just an inode.
Many tools end up handling this by showing a folder named "a" containing a folder named "" (empty string). This confuses users quite a bit. It's more than the inodes, it's how the tooling handles the abstraction.
Earlier quoted context omitted.
Is that really so different from how folders work on other systems? A directory inode is just an inode.
Another challenge is directory flattening. On a file system "a/b" and "a//b" are usually considered the same path. But on S3 the slash isn't a directory separator, so the paths are distinct. You need to be extra careful when building paths not to include double slashes. Many tools end up handling this by showing a folder named "a" containing a folder named "" (empty string). This confuses users quite a bit. It's more…
This created a prefix named “/“ which confused the hell out of the application.
Hudi, Delta, iceberg bridge that gap now. Databricks built a company around it.
Don't try to do relational on object storage on your own. Use one of those libraries. It seems simple but it's not. Late arriving data, deletes, updates, primary key column values changing, etc.
Try to imagine your astonishment if a traditional storage vendor showed up and told you that their very expensive premium file system they had just sold you:
- can't store log files because it can't append
anything to an existing files
- can't copy files more than 5GB
- can't rename or move a file
> Filesystem software, especially databases, can't be ported to Amazon S3 Hudi, Delta, iceberg bridge that gap now. Databricks built a company around it. Don't try to do relational on object storage on your own. Use one of those libraries. It seems simple but it's not. Late arriving data, deletes, updates, primary key column values changing, etc.
Why would we need to do it on object storage which addresses a different type of storage need.
Nevertheless there are projects like EMRFS and S3 file system mount points that try to provide files stem interfaces to workloads that need to see S3 as a filesystem.
Earlier quoted context omitted.
Thank you, now I understand what the special 0-byte object refers to. It represents an empty folder. Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees.
> […] what the special 0-byte object refers to. It represents an empty folder. Alas, no. It represents a tag, e.g. «folder/», that points to a zero byte object. You can then upload two files, e.g. «folder/file1.txt» and «folder/file2.txt», delete the «folder/», being a tag , and still have the «folder/file1.txt» and «folder/file2.txt» file intact in the S3 bucket. Deleting «folder/» in a traditional file system, on t…
However, there are file managers, FTP clients, and S3 clients that will do that for you by deleting individual files.
Earlier quoted context omitted.
What exactly do you think a folder is? It’s just an abstraction for organising data.
S3 doesn’t have that abstraction. The console UI shows folders but they don’t actually exist in S3. They’re made up by the UI.
On the Mac, the Finder lets you have files with slashes in their names, even though it's a Unix file system underneath. Don't believe me? Go try to use the Finder to make a directory whose name is "Reports from 2024/03/10". See?
But as everyone knows, slash is the ONLY character you're not allowed to have in a file or directory name under Unix. It's enforced in the kernel at the system call inteface. There is absolutely no way to make a file with a slash in it. Yet there it is!
The original MacOS operating system used the ":" character to delimit directory names, instead of "/", so you could have files and directories with slashes in their names, justs not with colons in their names.
When Apple transitioned from MacOS to Unix, they did not want to freak out their users by reaming all their files.
So now try to use the Finder (or any app that uses the standard file dialog) to make a folder or file with a ":" in its name on a modern Mac. You still can't!
So now go into the shell and list out the parent directory containing the directory you made with a slash in its name. It's actually called "Reports from 2024:03:10"!
The Mac Finder and system file dialog user interfaces actually switche "/" and ":" when they show paths on the screen!
Try making a file in the shell with colons in it, then look at it in the finder to see the slashes.
However, back in the days of the old MacOS that permitted slashes in file names, there was a handy network gateway box called the "Gatorbox" that was a Localtalk-to-Ethernet AFP/NFS bridge, which took a subtly different approach.
https://en.wikipedia.org/wiki/GatorBox
It took advantage of the fact (or rather it triggered the bug) that the Unix NFS implementation boldly made an end-run around the kernel's safe system call interface that disallowed slashes in file names. So any NFS client could actually trick Unix into putting slashes into file names via the NFS protocol!
It appeared to work just fine, but then down the line the Unix "restore" command would totally shit itself! Of course "dump" worked just fine, never raising an error that it was writing corrupted dumps that you would not be able to read back in your time of need, so you'd only learn that you'd been screwed by the bug and lost all your files months or years later!
So not only does NFS stand for "No File Security", it also stands for "Nasty Forbidden Slashes"!
https://news.ycombinator.com/item?id=31820504
>NFS originally stood for "No File Security".
>The NFS protocol wasn't just stateless, but also securityless!
>Stewart, remember the open secret that almost everybody at Sun knew about, in which you could tftp a host's /etc/exports (because tftp was set up by default in a way that left it wide open to anyone from anywhere reading files in /etc) to learn the name of all the servers a host allowed to mount its file system, and then in a root shell simply go "hostname foo ; mount remote:/dir /mnt ; hostname `hostname`" to temporarily change the CLIENT's hostname to the name of a host that the SERVER allowed to mount the directory, then mount it (claiming to be an allowed client), then switch it back?
>That's right, the server didn't bother checking the client's IP address against the host name it claimed to be in the NFS mountd request. That's right: the protocol itself let the client tell the server what its host name was, and the server implementation didn't check that against the client's ip address. Nice professional protocol design and implementation, huh?
>Yes, that actually worked, because the NFS protocol laughably trusted the CLIENT to identify its host name for security purposes. That level of "trust" was built into the original NFS protocol and implementation from day one, by the geniuses at Sun who originally designed it. The network is the computer is insecure, indeed.
[...]
From the Unix-Haters Handbook:
https://archive.org/stream/TheUnixHatersHandbook/ugh_djvu.tx...
Don't Touch That Slash!
UFS allows any character in a filename except for the slash (/) and the ASCII NUL character. (Some versions of Unix allow ASCII characters with the high-bit, bit 8, set. Others don't.)
This feature is great — especially in versions of Unix based on Berkeley's Fast File System, which allows filenames longer than 14 characters. It means that you are free to construct informative, easy-to-understand filenames like these:
1992 Sales Report
Personnel File: Verne, Jules
rt005mfkbgkw0 . cp
Unfortunately, the rest of Unix isn't as tolerant. Of the filenames shown above, only rt005mfkbgkw0.cp will work with the majority of Unix utilities (which generally can't tolerate spaces in filenames).
However, don't fret: Unix will let you construct filenames that have control characters or graphics symbols in them. (Some versions will even let you build files that have no name at all.) This can be a great security feature — especially if you have control keys on your keyboard that other people don't have on theirs. That's right: you can literally create files with names that other people can't access. It sort of makes up for the lack of serious security access controls in the rest of Unix.
Recall that Unix does place one hard-and-fast restriction on filenames: they may never, ever contain the magic slash character (/), since the Unix kernel uses the slash to denote subdirectories. To enforce this requirement, the Unix kernel simply will never let you create a filename that has a slash in it. (However, you can have a filename with the 0200 bit set, which does list on some versions of Unix as a slash character.)
Never? Well, hardly ever.
Date: Mon, 8 Jan 90 18:41:57 PST
From: sun!wrs!yuba!steve@decwrl.dec.com (Steve Sekiguchi)
Subject: Info-Mac Digest V8 #3 5
I've got a rather difficult problem here. We've got a Gator Box run-
ning the NFS/AFP conversion. We use this to hook up Macs and
Suns. With the Sun as a AppleShare File server. All of this works
great!
Now here is the problem, Macs are allowed to create files on the Sun/
Unix fileserver with a "/" in the filename. This is great until you try
to restore one of these files from your "dump" tapes, "restore" core
dumps when it runs into a file with a "/" in the filename. As far as I
can tell the "dump" tape is fine.
Does anyone have a suggestion for getting the files off the backup
tape?
Thanks in Advance,
Steven Sekiguchi Wind River Systems
sun!wrs!steve, steve@wrs.com Emeryville CA, 94608