Show HN: TidalWaves API – live, tokenized news metadata from around the world

Post by panarky » Wed, Oct 21, 2020, 4:58 PM UTC

prohobo wrote: post #35

Grimm1 wrote: post #30

Hey there, first off cool stuff I can see this being plenty useful for various projects. We're exploring GDELT data for our own needs and I was wondering if you wouldn't mind sharing what were some rough spots or gotchas using the project?

Thanks! There are a lot of issues with GDELT data, but the things that come to mind recently are: - Missing keywords. Huge topics like "brexit" won't be found, so you need to extract those yourself. - Dirty titles. You need to extract the article titles from the website metadata, and more often than not they'll pre/append their site name like "[title] - DailyStormer" and links like "[title]: Business, Stocks, News" e…

Post by keithwhor » Wed, Oct 21, 2020, 7:26 PM UTC

prohobo wrote: post #33

OJFord wrote: post #32

I've never heard of RapidAPI before, but it seems kind of scammy. As in, nice idea, bit just filled with low-quality spam APIs wrapping presumably free or cheaper services, not that RapidAPI itself is a scam. But then, Amazon's awash with tat too these days, and I wouldn't be surprised at anyone selling their product there.

Yeah, Rapid API is quite good in itself, but there are a lot of trash APIs on there. I see it as a good way to test the waters. If people are interested in it, I'll eventually move it off of Rapid API for direct access and use Stripe for subscriptions.

Post by op03 » Wed, Oct 21, 2020, 8:17 PM UTC

Post by abraxaz » Thu, Oct 22, 2020, 12:09 PM UTC

prohobo wrote: post #44

abraxaz wrote: post #17

Have you considered using JSON-LD/RDF for this? It would make it easier to integrate with WikiData and data from other sources.

No, but that actually sounds like a good idea. If there are a lot of people using this data along with WikiData, or if there's a kind of standard for "big" encyclopedic data, then I'd definitely like to support it.