Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Hacker News BigQuery dataset is updated daily again

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 12 posts

Tell HN: Hacker News BigQuery dataset is updated daily again

#1

Post by wietsevenema »

As you might know, there's a Hacker News BigQuery dataset [1].

I recently noticed that the export was broken (since Nov 2022) and I worked with the team to help fix it. It'll now again run daily.

SELECT timestamp FROM `bigquery-public-data.hacker_news.full` ORDER BY id DESC LIMIT 1 # 2024-06-09 09:12:55 UTC

[1]: https://console.cloud.google.com/marketplace/product/y-combinator/hacker-news

Re: Tell HN: Hacker News BigQuery dataset is updated daily again

#3

Post by izyda »

dang wrote: post #2

Thanks! I tried emailing my contacts on that team several times and never heard back, so I figured we no longer had contacts on the that team.

Any interest in getting this dataset on Snowflake too?

(I run a Snowflake funded data provider on Snowflake marketplace)

Re: Tell HN: Hacker News BigQuery dataset is updated daily again

#4

Post by dang »

izyda wrote: post #3
dang wrote: post #2

Thanks! I tried emailing my contacts on that team several times and never heard back, so I figured we no longer had contacts on the that team.

Any interest in getting this dataset on Snowflake too? (I run a Snowflake funded data provider on Snowflake marketplace)

The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API), so anyone is free to do this.

Re: Tell HN: Hacker News BigQuery dataset is updated daily again

#5

Post by Raed667 »

dang wrote: post #4
izyda wrote: post #3

Earlier quoted context omitted.

Any interest in getting this dataset on Snowflake too? (I run a Snowflake funded data provider on Snowflake marketplace)

The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API ), so anyone is free to do this.

a bit off topic: is there a way to display a HN page using the firebase API/sdk without doing N+1 requests ?

(sorry for noob question)

Re: Tell HN: Hacker News BigQuery dataset is updated daily again

#6

Post by dang »

Raed667 wrote: post #5
dang wrote: post #4

Earlier quoted context omitted.

The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API ), so anyone is free to do this.

a bit off topic: is there a way to display a HN page using the firebase API/sdk without doing N+1 requests ? (sorry for noob question)

No, unfortunately. We're going to make a new API that will be much easier, but there's at least one other major project that needs to get done first.

Re: Tell HN: Hacker News BigQuery dataset is updated daily again

#7

Post by wietsevenema »

dang wrote: post #4
izyda wrote: post #3

Earlier quoted context omitted.

Any interest in getting this dataset on Snowflake too? (I run a Snowflake funded data provider on Snowflake marketplace)

The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API ), so anyone is free to do this.

Yes, that’s how it works. Same data as in the API.

Re: Tell HN: Hacker News BigQuery dataset is updated daily again

#8

Post by Raed667 »

dang wrote: post #6
Raed667 wrote: post #5

Earlier quoted context omitted.

a bit off topic: is there a way to display a HN page using the firebase API/sdk without doing N+1 requests ? (sorry for noob question)

No, unfortunately. We're going to make a new API that will be much easier, but there's at least one other major project that needs to get done first.

Thanks for the response!
Post reply on HN