Great work, thank you so much.
Fwiw and getting formatted text from html did you try
lynx —-dump url >> file.plaintext
Show HN: how I built the largest open database of Australian law
51–60 of 63 posts
Re: Show HN: how I built the largest open database of Australian law
#52Good work reaching out to, and trying to get along with Australian government departments. As a fellow Australian, and one employed in government, I can very much say that many people in charge of operating these systems should not be.
Re: Show HN: how I built the largest open database of Australian law
#53That's really neat. Such a shame VIC couldn't be included though.
Re: Show HN: how I built the largest open database of Australian law
#54Nice. Thanks for sharing.
Re: Show HN: how I built the largest open database of Australian law
#55Earlier quoted context omitted.
Github might be difficult as they impose constraints on the size of repositories and the Corpus is around 5GB. The Internet Archive is a good idea, however, I’ll have a look into that. I’ve also been thinking about sticking it on Kaggle as well to increase its reach.
You could also consider one or more of the scientific data repositories like Zenodo, FigShare, DataDryad, etc. 5GB is small potatoes for those folks and they have serious data retention policies. As a bonus, they'll also allocate you a citable DOI.
Woot! Thanks for the DataDryad info. I am lodging a tiny test data set with them now...
Re: Show HN: how I built the largest open database of Australian law
#56What do you think of the Canadian legal case law website CanLii? What could it do better or do you think its done well? Is it overdue for innovation?
It actually looks quite clean. Certainly a lot better than some of our legal databases. I guess the only suggestion I'd have is that it seems like you can have an account with the website but there's login or register button on the front page (or maybe I'm just not seeing it?). One other point, and this is not specific to CanLII (I haven't checked whether this is the case) but I've seen that a lot of legislation data…
Like you're saying that there's no place to register on the front page?
Re: Show HN: how I built the largest open database of Australian law
#57Could you explain how the majority of your corpus is under CC BY 4.0? I realise that's the licence you have picked on HuggingFace, but if the source data was not already CC BY 4.0, how are you able to re-licence it as CC BY 4.0?
Re: Show HN: how I built the largest open database of Australian law
#58Could you explain how the majority of your corpus is under CC BY 4.0? I realise that's the licence you have picked on HuggingFace, but if the source data was not already CC BY 4.0, how are you able to re-licence it as CC BY 4.0?
The majority of the source data was already licensed under CC BY 4.0. Additionally, the Corpus, as a work constituting a curated and post-processed collation of other works, is also licensed under CC BY 4.0.
Re: Show HN: how I built the largest open database of Australian law
#59I'm floored by what you accomplished in a year. Here in Canada, case law is under an iron grip by Canlii, LexisNexis and Thomson Reuters (Westlaw). As far as I'm aware, there is no truly open digital dataset that the public can use. This situation suits all the players involved and it isn't changing - it's refreshing to hear that one person can make a difference in a very similar setting (Australia)
Re: Show HN: how I built the largest open database of Australian law
#60I'm floored by what you accomplished in a year. Here in Canada, case law is under an iron grip by Canlii, LexisNexis and Thomson Reuters (Westlaw). As far as I'm aware, there is no truly open digital dataset that the public can use. This situation suits all the players involved and it isn't changing - it's refreshing to hear that one person can make a difference in a very similar environment (Australia)