Show HN: A database of everything (over 55M keys)
1–10 of 43 posts
Re: Show HN: A database of everything (over 55M keys)
#2What sort of backend storage does this use?
Re: Show HN: A database of everything (over 55M keys)
#3What sort of backend storage does this use?
Given that it's effectively key->key->value, I'm guessing Cassandra for the main backend. The data model fits very well, and it would give you the kind of scalability you would need for this sort of thing.
Re: Show HN: A database of everything (over 55M keys)
#4So searches like "Linux" "California" "lincoln" "Hitler" "Disney" and "red" return 0 results
Re: Show HN: A database of everything (over 55M keys)
#5As much as I like the idea, I can already see someone called "Drumpf" on the front page linking attributes about Hillary Clinton and Donald Trump to their Twitter accounts and not their names. I guess if you're after "everything" then curation makes no sense but you'll end up with a bunch of nonsense.
Re: Show HN: A database of everything (over 55M keys)
#6As much as I like the idea, I can already see someone called "Drumpf" on the front page linking attributes about Hillary Clinton and Donald Trump to their Twitter accounts and not their names. I guess if you're after "everything" then curation makes no sense but you'll end up with a bunch of nonsense.
Twitter url seems like a strong key since it is referenced in many other contexts.
As for curation, it is intended to provide examples of what key, attr and values are regardless of their content. This is an experimental feature and might be removed/tweaked...
Re: Show HN: A database of everything (over 55M keys)
#7So searches like "Linux" "California" "lincoln" "Hitler" "Disney" and "red" return 0 results
Up until last week, outpan was strictly used for gathering data on product barcodes.
We are in the process of adding data in other categories.
Re: Show HN: A database of everything (over 55M keys)
#8Signup page is blanking out for me.
Really curious to try it. It's a really neat idea. Has this been done before? I've never seen anything like it.
The full value of this would likely come from interesting, productive, insightful visualizations of the underlying graph that is being built.
Questions that come to my mind:
- What if you write bots that scrape Wikipedia, Twitter, etc. and output entries from semantic analysis performed on these sources?
- If many people write such bots, how similar would the graphs be? What are the parameters that determine graph overlap?
- Can you use this to tie in to the real world?
"A key can be any unique string such as product barcodes, book ISBNs, email addresses, URLs, domain names, names, phone numbers ..."
Interesting stuff there... A way to make this into an interesting social network is if people curated their own graphs, e.g. of books and webpages and favorite restaurants, and other people could browse these graphs in a read only mode. (perhaps they can clone it to their graphs or so and then start to edit them)
The temporal aspect might be interesting too. Would there be value in seeing how my graph has changed over time? When I was in academia, none of the tools to keep track of the papers I read suited me. I could see this working well for this use case scenario.
(if you could write a Prolog against this, would it have interesting properties?)
I see that a submitter just posted 7630028603780 -> volume -> 75ml (the first number being probably the bar code of some beverage), that's neat.
Someone else just posted the following entry:
5010438013621 -> ingredients -> Water, sugar, mixed fruit juices from concentrate 10% (Grape, blackcurrant, raspberry), acid (Citric acid), Vimto flavouring (Includes natural extracts of fruits, herbs, barley malt and spices), colouring food (Concentrate of carrot, hibiscus), acidity regulator (Sodium citrate), preservatives (Sodium benzoate, Potassium sorbate), Vitamin C, Sweeteners (Sucralose, Acesulfame K).
Now what the site should be doing is converting this to something like:
5010438013621 -> ingredients -> water
5010438013621 -> ingredients -> sugar
5010438013621 -> ingredients -> vitamin C
...
(Outpan person/people, I'm in SF, if you want to chat more around coffee, there's an email is in my profile)
Re: Show HN: A database of everything (over 55M keys)
#9So searches like "Linux" "California" "lincoln" "Hitler" "Disney" and "red" return 0 results
I was driving so couldn't stay on the phone. The number of keys for different concepts is just very large (if not infinite for the sake of avoiding philosophical debates). It will take a some time to have enough data to cover even popular keys. Fortunately I have a lot of time :D
Re: Show HN: A database of everything (over 55M keys)
#10Signup page is blanking out for me. Really curious to try it. It's a really neat idea. Has this been done before? I've never seen anything like it. The full value of this would likely come from interesting, productive, insightful visualizations of the underlying graph that is being built. Questions that come to my mind: - What if you write bots that scrape Wikipedia, Twitter, etc. and output entries from semantic ana…
I think graph overlap is what actually determines what data is "accurate". There are currently bots writing to the database by people who are not connected and I have yet to see the overlap (since the key space is too large for the number of bots right now). I'm excited to see how that plays out.
I will send you an email :)