Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

35% Faster Than The Filesystem (2017)

sqlite.org

Post reply on HN
Print view View original

161–166 of 166 posts

Re: 35% Faster Than The Filesystem (2017)

#161

Post by mmis1000 »

majewsky wrote: post #105
mmis1000 wrote: post #60

Earlier quoted context omitted.

I though, one of the advantage here is, you are able to store/read metadata along the file content easy without worrying the metadata and file may desync in some way or corrupted. This makes serve something like Image (photo dimension) or video(length) metadata way easier. Some filesystem (like ntfs) do have place to save metadata along files, but most filesystem don't have the way to done that properly

You're referring to extended attributes, which are like a key/value database for each file. Wikipedia says: > In Linux, the ext2, ext3, ext4, JFS, Squashfs, Yaffs2, ReiserFS, Reiser4, XFS, Btrfs, OrangeFS, Lustre, OCFS2 1.6, ZFS, and F2FS[9] filesystems support extended attributes (abbreviated xattr) when enabled in the kernel configuration. > The Linux kernel allows extended attribute to have names of up to 255 byte…

Then you would encounter another problem, extended attributes arn't usually really portable between files system / operation systems (as the name 'extendded' suggests, it may vary between FSs or even not available at all). And may gone easily if any of the programs not operate on them correctly. Windows would show a big warning if your copy operation may cause the extra metadata along the file to gone, but I am not sure whether linux will do that.

Re: 35% Faster Than The Filesystem (2017)

#162

Post by notyourday »

ysleepy wrote: post #144
notyourday wrote: post #18

As someone who has spent time consulting for different kinds of "we do stuff on the internet" companies, I can confidently say that this is a premature optimization for 99.999% of the companies/projects. The companies simply don't have enough IO traffic to need it. Is it cool? Sure. Is it sexy? Maybe. Is it needed ? Nope. Do boring stuff. Use files. If you send your small files over the internet via web and you need…

While I agree with the sentiment, using the unix file API is wrought with peril if you try to do it properly. https://www.usenix.org/system/files/conference/osdi14/osdi14... sqlite does a good job to provide expected consistency even in the face of power loss. Half overwritten files or causal ordering violations can occur if you do not call fsync at the appropriate places.

Sorry, that's junk peddled by people who have no idea how often disks become "unhealthy" in the most spectacular way. Bitrot is real. Disks returning funky data is real. Sqlite does not even have online remote backup that would allow one to easily fix/tweak a portion of a database. When you start storing a few TB of data in every blob you are going to learn some very painful lessons that grey beards learned thirty years ago, except that you will learn them at a real misery producing scale.

Is it an efficient way of querying structured data that can easily be blown away/recreated? Absolutely. Storing bytestreams that should not change in SQLite? That's insanity. Edgy and hipster-esque insanity but insanity none the less.

Re: 35% Faster Than The Filesystem (2017)

#163

Post by bbmario »

zaroth wrote: post #6

For small- to mid-sized projects, I’ve always realized huge gains in simplicity by haves “Files” tables to store various assets. It means instances in a web-farm can pull the files down when they initialize easily, it means files are automatically versioned, it provides an obvious place to put the files when they are being uploaded on the Admin panel. It means all the files are getting backed up as part of the databa…

Have you seen https://icculus.org/physfs/ ?

You could put everything inside a zip and that's it.

Re: 35% Faster Than The Filesystem (2017)

#164

Post by d1zzy »

liability wrote: post #154
esmi wrote: post #136

Earlier quoted context omitted.

If that’s the case then the restrictions should’ve been given in the problem statement and the interviewer should’ve been able to answer why the candidate’s answer was rejected.

Consider it a test of your ability to infer the obvious (a valuable skill.)

I'm sorry but what is obvious in an interview setting? Is it obvious that you can or cannot use standard library? Is it obvious that you can or cannot use state of the art existing software to solve problems?

I do many interviews and I remember even when I started I had to be pretty clear what the candidate can and cannot use. I actually appreciate that they can find existing solutions that are likely better than what one can come up with in a few hours but if I need to test them on how they can code something like that I'll specifically state it.

Re: 35% Faster Than The Filesystem (2017)

#165

Post by d1zzy »

ImNotTheNSA wrote: post #135
notyourday wrote: post #126

Earlier quoted context omitted.

> But we're often not looking for software _users_ as much as we're looking for _engineers_. What the applicant illustrated is that he is a much better engineer than the interviewer and the interviewer did not like it. Good software engineering is about leveraging existing robust tools in a new way, not about re-inventing a wheel while making is square. For me that would have been an insta-hire.

I disagree. Applicant should have done more to understand the requirements for the task. That’s a massive part of software engineering, and you would be surprised how much time software engineers will spend solving the wrong problem . He didn’t understand that part of the problem included rolling a novel solution or restricted use of a database. Almost a daily problem for software engineers is extracting this informa…

I agree that the candidate should have maybe asked if they can just use third party software but I also think that was failure on the part of the interviewer. If there are constraints about the tools the candidate may use to solve the given problem then those constraints need to be specified. Similarly to how you specify which programming language you allow to solve the problem or if you allow standard library use.

Re: 35% Faster Than The Filesystem (2017)

#166

Post by ngirard »

ruuda wrote: post #51
OJFord wrote: post #15

Next week on Show HN: `sqlitefs`, the SQLite-backed FUSE filesystem?

Something similar exists: https://www.youtube.com/watch?v=wN6IwNriwHc

Interesting project, thanks.

For the record, their GitHub repository is https://github.com/samzyy/DB-based-replicated-filesystem

Post reply on HN