Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fsearch, a fast file search utility for Unix-like systems

github.com

Post reply on HN
Print view View original

81–90 of 98 posts

Re: Fsearch, a fast file search utility for Unix-like systems

#81

Post by lelanthran »

CyberDildonics wrote: post #80
lelanthran wrote: post #79

Earlier quoted context omitted.

> Everything scans the filesystem very fast and there is no way it is just using 'stat' on every file then diving into the directories. The last time I checked, Everything worked by using the AV calls microsoft provides; anytime a file is written, the name (and other metadata) can be written to a log that Everything can check once every 5 seconds or so. If I thought there was any money at all to be made from providin…

What you are talking about is file change notifications. A huge part of Everything's speed comes from reading the master file table that other people mentioned, so you would need a way to quickly read file table entries on linux. https://learn.microsoft.com/en-us/windows/win32/devnotes/mas...

> What you are talking about is file change notifications. A huge part of Everything's speed comes from reading the master file table that other people mentioned, so you would need a way to quickly read file table entries on linux.

Not a problem. And no, I'm not talking about inotify either, and I'll additionally index the contents of (text) files as well with a negligible additional performance hit. It can be done as fast as, or faster than, `Everything`.

TBH, if I thought I could make even $100 in donations from this, I'd start it tomorrow, but absolutely no one misses ultra-fast searching when they don't have it.

Even on Windows, the number of users who go out and look for something that searches as fast as Everything is a rounding error - statistical noise. Now go and divide that fractional percentage of Everything users on Windows by 100 to get the number of Linux users who might use this.

Re: Fsearch, a fast file search utility for Unix-like systems

#82

Post by wander_homer »

lelanthran wrote: post #81
CyberDildonics wrote: post #80

Earlier quoted context omitted.

What you are talking about is file change notifications. A huge part of Everything's speed comes from reading the master file table that other people mentioned, so you would need a way to quickly read file table entries on linux. https://learn.microsoft.com/en-us/windows/win32/devnotes/mas...

> What you are talking about is file change notifications. A huge part of Everything's speed comes from reading the master file table that other people mentioned, so you would need a way to quickly read file table entries on linux. Not a problem. And no, I'm not talking about inotify either, and I'll additionally index the contents of (text) files as well with a negligible additional performance hit. It can be done a…

> Not a problem. And no, I'm not talking about inotify either, and I'll additionally index the contents of (text) files as well with a negligible additional performance hit. It can be done as fast as, or faster than, `Everything`.

Please enlighten us how that would work.

> TBH, if I thought I could make even $100 in donations from this, I'd start it tomorrow, but absolutely no one misses ultra-fast searching when they don't have it.

You can easily make $100 in donations with this. I did it with this piece of software while it was still less performant and powerful and without an official release and by only mentioning it on one or two forums.

If the software delivers what you're saying, I'll guarantee you, that this will lead to more than 100$ per month in donations.

Re: Fsearch, a fast file search utility for Unix-like systems

#83

Post by CyberDildonics »

lelanthran wrote: post #81
CyberDildonics wrote: post #80

Earlier quoted context omitted.

What you are talking about is file change notifications. A huge part of Everything's speed comes from reading the master file table that other people mentioned, so you would need a way to quickly read file table entries on linux. https://learn.microsoft.com/en-us/windows/win32/devnotes/mas...

> What you are talking about is file change notifications. A huge part of Everything's speed comes from reading the master file table that other people mentioned, so you would need a way to quickly read file table entries on linux. Not a problem. And no, I'm not talking about inotify either, and I'll additionally index the contents of (text) files as well with a negligible additional performance hit. It can be done a…

It can be done as fast as, or faster than, `Everything`.

Then how would you do it? That's what I'm asking, how would you get the file attributes off of the disk as fast as everything on linux? Once you get them off the disk any modern computer can burn through them, but getting that data into memory in the first place is the problem.

Re: Fsearch, a fast file search utility for Unix-like systems

#84

Post by gitaarik »

QuadrupleA wrote: post #38
ivanjermakov wrote: post #35

Earlier quoted context omitted.

What made you to give up?

Too much research and extremely low level troubleshooting required (i.e. source code reading) to get things working. Especially around wayland, multi-monitor, multi-GPU, Nvidia, etc. I hate what Microsoft has been doing with Windows, but Linux just isn't practical for my setup yet.

Nvidia is indeed not well supported, on which Nvidia is mostly to blame (also see Linus Torvalds finger to Nvidia). That is too bad because the community can't do much about it.

Multi monitor support should generally be fine in most desktop environments, at least for 2 screens. More than that can indeed be quirky, dependent on the desktop environment and window manager (X or Wayland) you're using.

Multi-GPU is probably a bit niche to have good driver coverage, partly probably because of the Nvidia issue.

Re: Fsearch, a fast file search utility for Unix-like systems

#85

Post by ametrau »

berkes wrote: post #56
QuadrupleA wrote: post #38

Earlier quoted context omitted.

Too much research and extremely low level troubleshooting required (i.e. source code reading) to get things working. Especially around wayland, multi-monitor, multi-GPU, Nvidia, etc. I hate what Microsoft has been doing with Windows, but Linux just isn't practical for my setup yet.

You are either extremely unlucky or chose a very strange setup. The problems you describe were common in early 2000s, but haven't been common in Linux desktop for a decade or so. For those reading above and thinking "I'll skip Linux, if that's the current status": it's not. Just pick Ubuntu LTS. Use it on common hardware (e.g not bleeding edge) and stick with the defaults. Don't try to make it exactly like your Mac o…

I have wasted so much time trouble shooting just basic things on Linux DEs. Example: mouse wheel scroll speed. In general it’s a mishmash of various egos, various low efforts, various high efforts, etc. Not usable imo if you just want to use a computer and have the OS out of your way.

Re: Fsearch, a fast file search utility for Unix-like systems

#86

Post by lelanthran »

wander_homer wrote: post #82
lelanthran wrote: post #81

Earlier quoted context omitted.

> What you are talking about is file change notifications. A huge part of Everything's speed comes from reading the master file table that other people mentioned, so you would need a way to quickly read file table entries on linux. Not a problem. And no, I'm not talking about inotify either, and I'll additionally index the contents of (text) files as well with a negligible additional performance hit. It can be done a…

> Not a problem. And no, I'm not talking about inotify either, and I'll additionally index the contents of (text) files as well with a negligible additional performance hit. It can be done as fast as, or faster than, `Everything`. Please enlighten us how that would work. > TBH, if I thought I could make even $100 in donations from this, I'd start it tomorrow, but absolutely no one misses ultra-fast searching when the…

Firstly, I appreciate you taking the time to engage with me. I hope that I didn't come off as dismissive of your hard work or of being disrespectful of what you have delivered.

My point was that the incentive to produce something like `Everything` on Linux just isn't aligned with what the target market wants or needs. I think that what you have produced satisfies what the target market wants.

> You can easily make $100 in donations with this.

Honestly, I'm still very skeptical that even a $100 target is possible. I have to also admit that I've looked at stuff in the past, gone "No one could possibly want that, at that price point" and been horribly wrong.

I feel like I should test the claim of how many people want an `Everything` equivalent on Linux: I'll make it, package it with a MVP GUI, and mention it on a few forums in addition to posting a show HN here.

For ideal reproducibility, let me know which forum(s) you initially got traction on. I'll try to mirror your marketing as closely as possible.

I'd also like to know how you went about benchmarking performance against existing stuff for your project; for comparison against `Everything` I was thinking that the metric to beat is delta between file creation/removal time and the time that the file shows up in the results set (or index).

Like the other responder here, I also think that once something is in the index, retrieval time should be almost instant, so there's not much point in benchmarking "How long does it take to update results after every keypress" once that metric falls below 100ms or so.

Re: Fsearch, a fast file search utility for Unix-like systems

#87

Post by wander_homer »

lelanthran wrote: post #86
wander_homer wrote: post #82

Earlier quoted context omitted.

> Not a problem. And no, I'm not talking about inotify either, and I'll additionally index the contents of (text) files as well with a negligible additional performance hit. It can be done as fast as, or faster than, `Everything`. Please enlighten us how that would work. > TBH, if I thought I could make even $100 in donations from this, I'd start it tomorrow, but absolutely no one misses ultra-fast searching when the…

Firstly, I appreciate you taking the time to engage with me. I hope that I didn't come off as dismissive of your hard work or of being disrespectful of what you have delivered. My point was that the incentive to produce something like `Everything` on Linux just isn't aligned with what the target market wants or needs. I think that what you have produced satisfies what the target market wants. > You can easily make $1…

> I hope that I didn't come off as dismissive of your hard work or of being disrespectful of what you have delivered.

Not at all, I'm just incredibly curious of how you'd solve the issue of creating an index of a filesystem as fast as Everything, because I've thought and read a lot about it in the last couple of years and haven't found any solution at all, nor did I find any other software which achieved something like that on Linux systems.

> For ideal reproducibility, let me know which forum(s) you initially got traction on. I'll try to mirror your marketing as closely as possible.

One post on the Arch Linux forum and one on the r/linux sub on Reddit. From there I got enough users to get more than 100$ in donations. Nowadays it's obviously more.

> I'd also like to know how you went about benchmarking performance against existing stuff for your project;

Everything has an extensive debug mode with detailed performance information about pretty much everything it's doing. That's how I know exactly how long it took to create the index, perform a specific search, update the index with x file creations, deletions or metadata changes etc.

> for comparison against `Everything` I was thinking that the metric to beat is delta between file creation/removal time and the time that the file shows up in the results set (or index).

That's not particularly interesting, because it's quite straight forward to achieve a similar performance.

The crucial metric is how long it initially takes to create the index and then update it when the application starts (i.e. finding all changes to the filesystem which happened while the application wasn't running). That's where Everything excels and to which I and others haven't found a solution on non-Windows systems (without making significant changes to the kernel of course). The best and pretty much only solution I'm aware of is the brute force method of walking the filesystem and calling stat, which obviously is much slower.

Re: Fsearch, a fast file search utility for Unix-like systems

#88

Post by jcul »

wander_homer wrote: post #16
jcul wrote: post #14

I was wondering what this adds over mlocate. It seems it's a GUI only / GUI first tool. The GitHub page recommends mlocate for a CLI version.

Hi, author here. Likely the most significant benefit is the more powerful query language. For example you can also search by file modification date or size and use boolean operators. https://github.com/cboxdoerfer/fsearch/wiki/Search-syntax

Ah thanks for that, I can see the benefit there alright.

Re: Fsearch, a fast file search utility for Unix-like systems

#89

Post by lelanthran »

wander_homer wrote: post #87
lelanthran wrote: post #86

Earlier quoted context omitted.

Firstly, I appreciate you taking the time to engage with me. I hope that I didn't come off as dismissive of your hard work or of being disrespectful of what you have delivered. My point was that the incentive to produce something like `Everything` on Linux just isn't aligned with what the target market wants or needs. I think that what you have produced satisfies what the target market wants. > You can easily make $1…

> I hope that I didn't come off as dismissive of your hard work or of being disrespectful of what you have delivered. Not at all, I'm just incredibly curious of how you'd solve the issue of creating an index of a filesystem as fast as Everything, because I've thought and read a lot about it in the last couple of years and haven't found any solution at all, nor did I find any other software which achieved something li…

> The crucial metric is how long it initially takes to create the index and then update it when the application starts (i.e. finding all changes to the filesystem which happened while the application wasn't running)

That's what I meant by " delta between file creation/removal time and the time that the file shows up in the results set (or index)."

Basically, how fast can we update the index?

> That's where Everything excels and to which I and others haven't found a solution on non-Windows systems (without making significant changes to the kernel of course).

I've got a couple of out-there ideas which may or may not pan out, one of which was, indeed, a kernel module.

Another idea is to deploy the indexer as a daemon with the applications all using IPC to query and update it. This will give the query applications a significant advantage on startup compared to Everything.

As for updating the index timeously, I've got a few ideas there as well. Walking the filesystem starting at `/` for each update will result in only performing index updates once a day or so (hence, the reason I expressed the metric as a delta) so I feel that that is no good.

I'll do an implementation and try to message you (if you want to check it out) because code talks louder than words :-)

Re: Fsearch, a fast file search utility for Unix-like systems

#90

Post by wander_homer »

lelanthran wrote: post #89
wander_homer wrote: post #87

Earlier quoted context omitted.

> I hope that I didn't come off as dismissive of your hard work or of being disrespectful of what you have delivered. Not at all, I'm just incredibly curious of how you'd solve the issue of creating an index of a filesystem as fast as Everything, because I've thought and read a lot about it in the last couple of years and haven't found any solution at all, nor did I find any other software which achieved something li…

> The crucial metric is how long it initially takes to create the index and then update it when the application starts (i.e. finding all changes to the filesystem which happened while the application wasn't running) That's what I meant by " delta between file creation/removal time and the time that the file shows up in the results set (or index)." Basically, how fast can we update the index? > That's where Everything…

> Basically, how fast can we update the index?

The two core issues are:

1) How do you quickly get a list of all files and their attributes from the filesystem, without recursively visiting all directories? The kernel has no such functionality and neither do most filesystems (except NTFS with the MFT, which is how Everything solves that).

2) How do you know which files have been modified on a filesystem since it was last mounted on the system or since your monitoring daemon/application was running the last time? This information also needs to be stored persistently on the filesystem (like the USN journal, which Everything is using) if you want to avoid slow recursive traversals.

> I've got a couple of out-there ideas which may or may not pan out, one of which was, indeed, a kernel module.

Well the problem is, my kernel isn't the only kernel who changes the filesystems I'm using. Hence a kernel module only works if your system is the only one whose modifying the data you're working with or most other systems need to be using the same kernel module, which isn't realistic.

> Another idea is to deploy the indexer as a daemon with the applications all using IPC to query and update it. This will give the query applications a significant advantage on startup compared to Everything.

Everything uses a daemon as well and it's not a solution to that issue, because somehow the daemon also has to get the list of files/folders and their attributes out of a filesystem without walking it. How else would the daemon know which files belong to the volume which was just mounted moments ago?

> As for updating the index timeously, I've got a few ideas there as well. Walking the filesystem starting at `/` for each update will result in only performing index updates once a day or so (hence, the reason I expressed the metric as a delta) so I feel that that is no good.

Walking the filesystem shouldn't be done at all, because it's just too slow.

> I'll do an implementation and try to message you (if you want to check it out) because code talks louder than words :-)

Of course, I'd appreciate that.

Post reply on HN