Show HN: how I built the largest open database of Australian law
1–10 of 63 posts
Re: Show HN: how I built the largest open database of Australian law
#2Hey HN,
Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on.
In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web technologies to hacking together a multitude of different solutions for extracting text from documents.
My hope is that the next time someone like me is interested in training an LLM to solve legal problems, they won't have to go down a year-long journey of trying to find the right data!
You can find my database on HuggingFace (https://huggingface.co/datasets/umarbutler/open-australian-l...) and the code used to create it on GitHub (https://github.com/umarbutler/open-australian-legal-corpus-c...).
Re: Show HN: how I built the largest open database of Australian law
#3Would it we worth getting your corpus replicated into other venues as well, such at the Internet Archive or on GitHub itself?
Re: Show HN: how I built the largest open database of Australian law
#4Would it we worth getting your corpus replicated into other venues as well, such at the Internet Archive or on GitHub itself?
Github might be difficult as they impose constraints on the size of repositories and the Corpus is around 5GB. The Internet Archive is a good idea, however, I’ll have a look into that. I’ve also been thinking about sticking it on Kaggle as well to increase its reach.
Re: Show HN: how I built the largest open database of Australian law
#5Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on. In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web techn…
[deleted]
Re: Show HN: how I built the largest open database of Australian law
#6Great work and congratulations on your tenacity dealing with bureaucrats. Open access and machine readable formats should be widely available.
Re: Show HN: how I built the largest open database of Australian law
#7Insanely great. Amazing work.
Re: Show HN: how I built the largest open database of Australian law
#8Would it we worth getting your corpus replicated into other venues as well, such at the Internet Archive or on GitHub itself?
Github might be difficult as they impose constraints on the size of repositories and the Corpus is around 5GB. The Internet Archive is a good idea, however, I’ll have a look into that. I’ve also been thinking about sticking it on Kaggle as well to increase its reach.
There are also national and university data repositories that might be interested and for which 5GB is not even noticable!
Re: Show HN: how I built the largest open database of Australian law
#9Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on. In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web techn…
Awesome work!
Re: Show HN: how I built the largest open database of Australian law
#10Is there a U.S. equivalent?