Really cool idea
Show HN: FSQL – Search through your file system with SQL-esque queries
81–90 of 132 posts
Re: Show HN: FSQL – Search through your file system with SQL-esque queries
#82Reminds me of osquery [0]. [0] - https://github.com/facebook/osquery
Doesn't Windows have something like this built-in? WMI or something?
yeah; wmi is incredibly powerful. before osquery linux and os x had nothing like it. it even has performance counters (albeit at slower intervals than win etl) at the ready.
Re: Show HN: FSQL – Search through your file system with SQL-esque queries
#83I never know when I have to use find vs grep. And linux grep is different from macOS grep so I google about it every day lol. I just never figure it out.
I think I'll be a heavy user of FSQL.
Re: Show HN: FSQL – Search through your file system with SQL-esque queries
#84Does anyone remember WinFS (1)? Bill Gates described it as his biggest product regret (2). I remember I thought it was brilliant. Too bad it was probably a little bit too futuristic for its time, as for a few other things they launched when it just was not the right time... the clunky Tablet PCs (3) were for sure another example. (1) https://en.m.wikipedia.org/wiki/WinFS (2) http://www.zdnet.com/article/bill-gates-bi…
I was researching WinFS recently and came across this series of posts explaining its vision:
* https://hal2020.com/2013/02/14/winfs-integratedunified-stora...
* https://hal2020.com/2013/02/14/winfs-integratedunified-stora...
* https://hal2020.com/2013/02/25/winfs-integratedunified-stora...
* https://hal2020.com/2013/03/10/winfs-integratedunified-stora...
Re: Show HN: FSQL – Search through your file system with SQL-esque queries
#85Earlier quoted context omitted.
Sounds dangerous!
No more than DELETE FROM ... WHERE ... wait, where is WHERE?
Even this can be made safe(r) if you only only only connect to your database nthrough a proxy that sanitizes queries. IIRC vitess adds an implicit LIMIT 10 to queries that don't have a limit.
Re: Show HN: FSQL – Search through your file system with SQL-esque queries
#86Didn't BeOS have some awesome database-like file system indexing and query system?
Sure; in general, a file system endowed with extended/extensible attributes can be naturally seen as a relational database (in which the files themselves are BLOBs).
This video talks in detail about the extended attributes in BeOS:
https://systemswe.love/archive/minneapolis-2017/ivan-richwal... - "Metadata Indexes & Queries in the BeOS Filesystem"
Re: Show HN: FSQL – Search through your file system with SQL-esque queries
#87Nice! I'm actually working on a similar project to push lsof and files from /proc into some postgres tables. Lets me do cool things like query log files across a ~6000 server infrastructure similar to: SELECT distinct(l.name) FROM lsof l, lsofer_runs r WHERE l.lsofer_id = r.id AND fd_type = 'REG' AND l.fd ~ '[0-9][uw]' AND l.name like '%log' GROUP BY l.name, r.hostname ORDER BY name Best of luck!
Is this something you could do with Presto? You'd need to write a custom connector and it doesn't look like there is support for dynamically adding/removing catalogs (https://github.com/prestodb/presto/issues/2445) but it would presumably handle the heavy lifting for you.
Re: Show HN: FSQL – Search through your file system with SQL-esque queries
#88Earlier quoted context omitted.
I'd really like to know why Microsoft can't do something like this in Windows itself. Everything is very useful.
My guess is that MS needs an excuse to index the contents of your files.
Everything does not index the content of your files, only the name and some attributes.
Also MS already index file contents by default, it just sucks at it. There have been several occasions where I use the find file by name syntax and Windows can't even find the file in the current folder.
Re: Show HN: FSQL – Search through your file system with SQL-esque queries
#89Does anyone remember WinFS (1)? Bill Gates described it as his biggest product regret (2). I remember I thought it was brilliant. Too bad it was probably a little bit too futuristic for its time, as for a few other things they launched when it just was not the right time... the clunky Tablet PCs (3) were for sure another example. (1) https://en.m.wikipedia.org/wiki/WinFS (2) http://www.zdnet.com/article/bill-gates-bi…
That was my first thought, too. I'd love a file system that was more like a relational database. Also see the Pick operating system.
BTRFS has had a bunch of problems trying to actually compete with traditional filesystems, though. In the distributed world, CalvinFS seems pretty promising to me.
Re: Show HN: FSQL – Search through your file system with SQL-esque queries
#90Earlier quoted context omitted.
Yep. I'm specifically writing it to find any log file that isn't being pushed into our third party logging service. It's a surprisingly difficult problem, especially considering the amount of tech sprawl that's accumulated. Since it's also a relatively low latency environment, it has to be written in a way that doesn't add too much load (without core isolation..).
You could use the audit subsystem. https://www.linux.com/learn/customized-file-monitoring-audit...
Definitely crossed my mind, but I'm working on hosts where installing auditd isn't really easy. Broken yum and apt all over the place makes installing new packages almost impossible. Same goes for lsof, but its installed in "enough" places. Kinda nightmarish, but it gives me a chance to write some fun code ;).
Also, thanks for the article! Super interesting. Think that'd be better than implementing something on top of sysdig?