Live data from Hacker News

Show HN: TidalWaves API – live, tokenized news metadata from around the world

tidalwaves.io

21–30 of 47 posts

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#21

As someone who has built event pipelines from GDELT data, I'd advise folks to not start (a GDELT project) with high expectations. Not only are there plenty of duplicates, but tagging quality is all over the place, especially for non-Western media.

Agreed, it's very hit or miss in some situations. I'm hoping to keep evolving this to clean it up and eek out more salient information.

I think GDELT is the best source, nothing else comes close to coverage in my opinion, it's just that all the articles it gathers revolve around world events, as opposed to all of the millions of topics of discussion and niche interests people post about.

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#24

Nice project. I actually did pretty much the same a long time ago: an API around GDELT data. Your map looks fantastic, btw. The problem with GDELT is that they don't have that much coverage. I discovered it when we began to build our own News API [1]. I failed to sell this data over an API. I think you should see much more interest for an application. API like that is a bit complex. Almost all potential clients who r…

Thanks! While I agree GDELT doesn't have the kind of coverage some people may be looking for, I think it's the best source for world news insights.

Oh nice, I actually saw Newscatcher on Rapid API but didn't see the marketing page. It looks like a great product, and I think suited for large-scale solutions. TidalWaves will likely find a niche for people making visualizations and personal apps.

What I want to do is make a mature pipeline that handles at least 90% of what amateur and solo users would want to use GDELT for, so hopefully people see the value in that. After all, I could make this visualization using the API - so I'm sure people can think of many other cool uses for it.

Also, thanks for reaching out, I'd definitely like to collaborate in the coming months.

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#26
For some uses (if accuracy is more important than speed), ICEWS is available as periodic dumps on Dataverse. They use CAMEO for event encoding which - if imperfect - at least is used elsewhere.

https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi...

https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi...

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#27

“All data is transformed to comply with ISO standards, reduce duplication, and maximize salience” Which ISO standards is this statement referring to?

Right now, country codes (ISO 3316 Alpha-2) converted from FIPS. In the future, anything that needs to be converted to ISO.

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#28

Is New Zealand left off TidalWaves' world map by design, or by accident? Asking on behalf of Rhys Darby. https://www.youtube.com/watch?v=HynsTvRVLiI

I don't know what to say except: apologies. It is on there, and you can drag the map around horizontally, though that feature isn't very apparent currently.

Re: Show HN: TidalWaves API – live, tokenized news metadata from around the world

#29
post #18

Earlier quoted context omitted.

Thanks for replying. I'm not sure what is useful yet - first of all I am trying to understand the scope of the data sources and the methodology for selections displayed in TidalWaves. I would like to see the list of publications if that is possible. I am wondering what "trending stories" means in TidalWaves? For example, if the NYT at any point contains 500 stories, and is refreshed say 3 times every 24 hours, with s…

I just did a quick query grouping all sources and number of articles per source in the database (currently about 4 days worth of data): https://pastebin.com/HyktFpML As you can see, there are already almost 8000 sources. I'm honestly not sure how GDELT scrapes its data though, but this is what they do for over 5 years now and it's backed by Google Cloud processing, so I'm sure it's ever expanding and very thorough. T…

Thanks for compiling that list. I have had a quick look and downloaded it for a longer look.

Some GDELT methodology is extolled here:

https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-...

Post reply on HN