Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Taxonomy Is Hard

autodidacts.io

Post reply on HN
Print view View original

41–50 of 83 posts

Re: Taxonomy Is Hard

#41

Post by PaulHoule »

em500 wrote: post #17
bambax wrote: post #6

All true. Taxonomy is indeed hard. But, does it actually matter? It seems what matters is not how files are stored/organized, but how one can find the files one is looking for. Taxonomy is mostly a search problem. Yet, although one is usually capable of remembering specific or unique details about a file, it's still incredibly hard to search for a file or its contents effectively. Dropbox, to pick just one example, d…

MacOS and Windows got built in search around 15 years ago. Anything more advanced would be very niche, so there's probably no market left.

Windows search only finds things sporadically though. I turned off search indexing on my Mac because otherwise I'd have to go out for coffee when I open any application.

Re: Taxonomy Is Hard

#42

Post by bambax »

PaulHoule wrote: post #38
bambax wrote: post #32

Earlier quoted context omitted.

> because it conflicts with the major thing vendors want Maybe, but that doesn't explain why there are no startups on the case, that wouldn't have this conflict of interest.

You'd think it wouldn't be that hard to build a personal search engine on top of Lucene or Elasticsearch and on one level it isn't. But there are two very hard problems. (1) Performance. Back in the day people frequently turned full text indexing on Windows off because it would slow down their computer too much. People won't be happy with the overheard of a search engine that is always scanning 10's of GBs of documen…

> They told over and over again that customers didn't care about search quality, they just wanted to see a list of 350+ data sources that the product could index.

This seems to describe corporate buyers ticking boxes on a form, more than actual users.

Re: Taxonomy Is Hard

#43

Post by TeMPOraL »

bambax wrote: post #35
TeMPOraL wrote: post #34

Earlier quoted context omitted.

Who would fund those, and where's the potential for outsized growth?

I think the problem is there's no demand, not that incumbents are fighting it. But I also think it's weird and a mystery that nobody wants good search. The trust issue you mention may be a clue.

> I think the problem is there's no demand

Computing is a supply-driven market. The demand is there, it's just partially latent, partially ignored. Vast majority of technology users have no choice but to choose from what's being offered, and the minority of tech-savvy users with opinions are increasingly too small a niche to support "power user" tools.

Re: Taxonomy Is Hard

#44

Post by PaulHoule »

flanked-evergl wrote: post #28
nobrains wrote: post #24

Tagging, seems like a solution. But isn't. Specific problems with tagging: - Need to tag every file (whereas in folders, you just navigate to the folder and everything you store there is in that folder) - Takes too long - Too much thinking overhead (at the time of storing) - To be effective have to enter the name for all tag entries (e.g. project, type, etc.). If anything is missed for a file, that file will never be…

If you tag using RDF[1] and use OWL[2] to define both the tag name and values then it really works pretty well. And with some entailment rules[3] you can even define rules that tags on folders becomes tags on items inside them, or you can just solve this with a bit more complex SPARQL query [4][5]. For an example of where such a system is used, see: - WikiData[6]: They use RDF but they don't use OWL, they have a simi…

What I realized the other day is that the property restrictions in OWL are a lot like the options you see on a typical search interface, say,

https://ropercenter.cornell.edu/ipoll/

It is hidden in plain site, for instance if you read this

https://www.w3.org/TR/owl-ref/#Restriction

you might not realized that there are some other axioms documented elsewhere in the standard that let you query on "less than", "greater than", "matches a regex", etc. Instead of fighting with Protege or writing turtle you should be able to compose queries out of restrictions plus complements, unions and intersections the way you snap together a program in Scratch, but somehow the idea that OWL works as a query language (logically define a class in terms of its properties) had eluded people completely.

Re: Taxonomy Is Hard

#45

Post by TeMPOraL »

zozbot234 wrote: post #39
TeMPOraL wrote: post #30

Earlier quoted context omitted.

> it's still incredibly hard to search for a file or its contents effectively. Underappreciated aspect of this is trust . As much as "taxonomy is mostly a search problem", search is in large part a trust problem - trust that the search was done exhaustively, and if it returned no results, it means there aren't any. This is a problem that IMO most offerings are blind to. One big offender for me is Windows Explorer. It…

I think what you're describing is database query, not search. Heuristic search is going to be non-exhaustive by definition. Desktop-focused databases are of course widely available (Paradox, Access, LibreOffice Base etc.) but the volume of data you could manage with them is of course limited. I.e. non-toy datasets will naturally be hosted on some sort of "cloud".

> I think what you're describing is database query, not search.

What's the difference?

> Heuristic search is going to be non-exhaustive by definition.

To me, a non-exhaustive search is broken by design, because it will miss results. Making data accessible only through such search means there will be a hole in the system into which some of the data falls, never to be seen again.

Re: Taxonomy Is Hard

#46

Post by zozbot234 »

PaulHoule wrote: post #40
bambax wrote: post #35

Earlier quoted context omitted.

I think the problem is there's no demand, not that incumbents are fighting it. But I also think it's weird and a mystery that nobody wants good search. The trust issue you mention may be a clue.

There isn't even an open source product with traction.

ripgrep solves the "precise full-text search" problem quite nicely.

Re: Taxonomy Is Hard

#47

Post by Someone »

em500 wrote: post #17
bambax wrote: post #6

All true. Taxonomy is indeed hard. But, does it actually matter? It seems what matters is not how files are stored/organized, but how one can find the files one is looking for. Taxonomy is mostly a search problem. Yet, although one is usually capable of remembering specific or unique details about a file, it's still incredibly hard to search for a file or its contents effectively. Dropbox, to pick just one example, d…

MacOS and Windows got built in search around 15 years ago. Anything more advanced would be very niche, so there's probably no market left.

More like 25 years ago. https://en.wikipedia.org/wiki/Mac_OS_8 had it in 1998 (possibly earlier. It isn’t clear to me whether Sherlock became PowerPC native in 8.5 or was new and PowerPC native in 8.5), Windows NT in 1996/2000 (https://devblogs.microsoft.com/windows-search-platform/the-e...)

Re: Taxonomy Is Hard

#48

Post by bjhartin »

Can anyone recommend a paper or book that approaches this from first principles, e.g. are some limitations/abilities due to mathematical structures such as functions (folders), relations (tags), etc?

Re: Taxonomy Is Hard

#49

Post by PaulHoule »

zozbot234 wrote: post #46
PaulHoule wrote: post #40

Earlier quoted context omitted.

There isn't even an open source product with traction.

ripgrep solves the "precise full-text search" problem quite nicely.

There is that, and locate.

Most people have Word Documents, PDF files, and other things that need a more complex indexing strategy. Also a lot of people have lots of image and audio files which pose their own challenges, namely indexing textual metadata and possibly some indexing of the content.

Re: Taxonomy Is Hard

#50

Post by PaulHoule »

TeMPOraL wrote: post #45
zozbot234 wrote: post #39

Earlier quoted context omitted.

I think what you're describing is database query, not search. Heuristic search is going to be non-exhaustive by definition. Desktop-focused databases are of course widely available (Paradox, Access, LibreOffice Base etc.) but the volume of data you could manage with them is of course limited. I.e. non-toy datasets will naturally be hosted on some sort of "cloud".

> I think what you're describing is database query, not search. What's the difference? > Heuristic search is going to be non-exhaustive by definition. To me, a non-exhaustive search is broken by design, because it will miss results. Making data accessible only through such search means there will be a hole in the system into which some of the data falls, never to be seen again.

Elasticsearch in particular can do the kind of queries most people expect from a database, see

https://www.elastic.co/kibana/

At work we have a search interface powered by Postgres that uses the full text index but also uses GIN indexes on arrays to index things based on categories/tags.

Post reply on HN