Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Taxonomy Is Hard

autodidacts.io

Post reply on HN
Print view View original

51–60 of 83 posts

Re: Taxonomy Is Hard

#51

Post by tough »

bambax wrote: post #42
PaulHoule wrote: post #38

Earlier quoted context omitted.

You'd think it wouldn't be that hard to build a personal search engine on top of Lucene or Elasticsearch and on one level it isn't. But there are two very hard problems. (1) Performance. Back in the day people frequently turned full text indexing on Windows off because it would slow down their computer too much. People won't be happy with the overheard of a search engine that is always scanning 10's of GBs of documen…

> They told over and over again that customers didn't care about search quality, they just wanted to see a list of 350+ data sources that the product could index. This seems to describe corporate buyers ticking boxes on a form, more than actual users.

Sometimes those box tickers are your customers, as the ones paying you. That's how you get Enterprise Software

Re: Taxonomy Is Hard

#52

Post by jbverschoor »

bambax wrote: post #6

All true. Taxonomy is indeed hard. But, does it actually matter? It seems what matters is not how files are stored/organized, but how one can find the files one is looking for. Taxonomy is mostly a search problem. Yet, although one is usually capable of remembering specific or unique details about a file, it's still incredibly hard to search for a file or its contents effectively. Dropbox, to pick just one example, d…

True if you know what you're looking for. But apparently discovery is highly underrated by many.

Re: Taxonomy Is Hard

#53

Post by tsthename »

Separating functions helps us use the right tool for the job. Taxonomies are for semantics, and the file system is for retrievability. The comfort of hierarchies makes it easy to try and do both simultaneously.

- From computer science, we know graphs give us expressive modeling capabilities. I sometimes use mermaid ER diagrams as a concept map to capture complex relationships between files and concepts.

- From library science, faceted classification works well for extensive collections because inserting a new entry does not require thinking about existing entries. I maintain entries in a spreadsheet for extensive collections that matter to me. Note: Facets are meant for unchanging or infrequently changing properties. Creating a concept map and maintaining a faceted classification system take work, so I only use them for things that are very important to me.

90% of files I only care about for a short amount of time. I use the file system to co-locate the files I'm currently working on (so a project) but then archive all of it when I move on to something else.

The trade-off is that I give up on sharing files between projects. I don't want to deal with references. I copy from the archive when I need to. On the rare occasion when I need to reconcile the same file between projects, I do it manually. What helps is working on only a few projects at the same time.

TL;DR: Archive more. Use high-investment techniques only for the small percentage of files that really matter.

Re: Taxonomy Is Hard

#55

Post by PaulHoule »

MrPatan wrote: post #19

https://en.wikipedia.org/wiki/Celestial_Emporium_of_Benevole...

The trouble with that list is that those are supposed to be pigeonhole categories.

"Those that belong to the emperor" is an alright category as any animal could belong or not belong to the emperor. You get in trouble working in a system where that has to be disjoint from "bird".

"Those that look like flies from a long way away" is a category that contains all of them.

A while back I was interested in databases like DBpedia and Freebase where you don't really state that anything is disjoint from anything else as most of the categories overlap with other categories. For instance, a person can be an Emperor and an Actor and a Wrestler... See Nero Claudius. Wikipedia doesn't necessarily distinguish between a video game, a manga and an anime that have the same title, so unless you are going to split the topic you're going to have something that is all three even though somebody might think those classes are disjoint.

Lately I have been involved in OWL modelling of financial messages and there, as in some other domains, most classes are disjoint -- in some cases that's a deliberate decision of the modellers, in other cases it is fundamental to the platform I am sucking data out of.

Re: Taxonomy Is Hard

#56

Post by DeathArrow »

bambax wrote: post #6

All true. Taxonomy is indeed hard. But, does it actually matter? It seems what matters is not how files are stored/organized, but how one can find the files one is looking for. Taxonomy is mostly a search problem. Yet, although one is usually capable of remembering specific or unique details about a file, it's still incredibly hard to search for a file or its contents effectively. Dropbox, to pick just one example, d…

For Windows there's Everything a pretty good search tool.

https://www.voidtools.com/support/everything/

Re: Taxonomy Is Hard

#57

Post by Mezzie »

talkingtab wrote: post #37
jillesvangurp wrote: post #5

Tagging and categorizing are two subtly different things to do. Having dealt with a lot of real world data, all I can say is that getting your hands on consistently tagged or categorized data is hard and gets harder the more data sources you have. A real world example of how tagging can be both super useful and get out of hand is open street maps. The only meta data allowed in there are tags. The OSM community depend…

My thinking is that categorization is a dead horse. Here is a problem: take a car and a truck. Now slowly, bit by bit, morph the car into the truck. At what point is the car a truck? Or is it ever a truck? I, personally, personally cannot see this as a problem that can be solved. Maybe you are better. My conclusion is that composition is a better method. Has a. And composition by capability is even better. Not great,…

> I, personally, personally cannot see this as a problem that can be solved. Maybe you are better.

Both library science and cognitive linguistics have solutions for/discuss this problem.

Re: Taxonomy Is Hard

#58

Post by taeric »

talkingtab wrote: post #37
jillesvangurp wrote: post #5

Tagging and categorizing are two subtly different things to do. Having dealt with a lot of real world data, all I can say is that getting your hands on consistently tagged or categorized data is hard and gets harder the more data sources you have. A real world example of how tagging can be both super useful and get out of hand is open street maps. The only meta data allowed in there are tags. The OSM community depend…

My thinking is that categorization is a dead horse. Here is a problem: take a car and a truck. Now slowly, bit by bit, morph the car into the truck. At what point is the car a truck? Or is it ever a truck? I, personally, personally cannot see this as a problem that can be solved. Maybe you are better. My conclusion is that composition is a better method. Has a. And composition by capability is even better. Not great,…

This runs the very real risk of thinking you can overload all "tags" onto something that you want.

That is, embrace the restrictions that come from a taxonomy. It is a bit of a lie, but a lot of it is defined on contrasts and these contrasts help build things.

What distinguishes a car from a truck? One is more faceted to holding people than to holding cargo. But if you don't know either a truck or a car, you are unlikely to know from just that description. Indeed, you could wind up with a cargo van. Which is different from a passenger van, in much the same way.

To that end, we present the taxonomy as a heirarchy of descriptive properties. When, in many cases, it is a heirarchy of representative samples. In programming, we distinguish between class based and prototype based object oriented representation. In reality, most taxonomies are presenting class style based relationships, but using prototype based representation from a population.

Re: Taxonomy Is Hard

#59

Post by jillesvangurp »

talkingtab wrote: post #37
jillesvangurp wrote: post #5

Tagging and categorizing are two subtly different things to do. Having dealt with a lot of real world data, all I can say is that getting your hands on consistently tagged or categorized data is hard and gets harder the more data sources you have. A real world example of how tagging can be both super useful and get out of hand is open street maps. The only meta data allowed in there are tags. The OSM community depend…

My thinking is that categorization is a dead horse. Here is a problem: take a car and a truck. Now slowly, bit by bit, morph the car into the truck. At what point is the car a truck? Or is it ever a truck? I, personally, personally cannot see this as a problem that can be solved. Maybe you are better. My conclusion is that composition is a better method. Has a. And composition by capability is even better. Not great,…

Not disagreeing but unfortunately a lot of data sources have different opinions on this. And sometimes categorizations are actually useful indicators to tell what is what.

Composable and namespaced tags is exactly what we do in my company. It gives us a lot of flexibility and it mirrors what OSM does. You can do a lot of duck typing against such a system and it will work as good as your data is without breaking completely when you get bad data. You just loose some features.

Re: Taxonomy Is Hard

#60

Post by geephroh »

jbverschoor wrote: post #52
bambax wrote: post #6

All true. Taxonomy is indeed hard. But, does it actually matter? It seems what matters is not how files are stored/organized, but how one can find the files one is looking for. Taxonomy is mostly a search problem. Yet, although one is usually capable of remembering specific or unique details about a file, it's still incredibly hard to search for a file or its contents effectively. Dropbox, to pick just one example, d…

True if you know what you're looking for. But apparently discovery is highly underrated by many.

Yes! I think this is a key concept that is often overlooked. Taxonomies -- and ontologies for that matter -- can often serve as a guide to the corpus, particularly when a user has less familiarity with the knowledge space.

And yes, any taxonomy is by definition always a subjective, biased construct. But this is often useful in helping to reveal the preferences and motivations behind the creation and curation of the corpus itself.

Post reply on HN