What do you think of the Canadian legal case law website CanLii? What could it do better or do you think its done well?
Is it overdue for innovation?
Show HN: how I built the largest open database of Australian law
21–30 of 63 posts
Re: Show HN: how I built the largest open database of Australian law
#22Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on. In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web techn…
Fantastic work, and really appreciate the write up. It's quite timely for me - I'm from a tech background and have just started studying Australian law, and was thinking about doing exactly this - so you are years ahead of me :).
Just one note - the link in your Github readme to https://umarbutler.com/open-australian-legal-corpus doesn't seem to go anywhere.
For someone interested in using the data (and help out with bugs/issues), where would you suggest starting?
Re: Show HN: how I built the largest open database of Australian law
#23Would it we worth getting your corpus replicated into other venues as well, such at the Internet Archive or on GitHub itself?
Github might be difficult as they impose constraints on the size of repositories and the Corpus is around 5GB. The Internet Archive is a good idea, however, I’ll have a look into that. I’ve also been thinking about sticking it on Kaggle as well to increase its reach.
You could also consider one or more of the scientific data repositories like Zenodo, FigShare, DataDryad, etc. 5GB is small potatoes for those folks and they have serious data retention policies. As a bonus, they'll also allocate you a citable DOI.
Re: Show HN: how I built the largest open database of Australian law
#24Australia has had free, searchable collections of Australian Law for 25+ years. Austlii is a prime example. There are Federal and State collections as well. The author is conscientious enough to read the scraping policy (or was blocked by anti-scraping tools) from feeding from one of these sites into his LLM.
As I point out in the introduction, there are a few free-to-access legal databases available in Australia, but none are truly open in the sense of being free from copyright restrictions. Neither AustLII nor Jade are licensed under an open source copyright licence such as CC BY 4.0 (which is what the majority of my Corpus is under).
Re: Show HN: how I built the largest open database of Australian law
#25This is cool and I'm a little surprised to see that Victoria is the one dragging the chain here. Is DataVic just talk, or does that not apply to law for some reason?
Yeah I was also disappointed to hear that. Afaik (I asked someone who previously worked in an adjacent field), it sounds like there’s no central system for publishing the judgements, it’s all published by each individual courthouse in a way that suits them, so lots of tedious individual scraping would be involved I’d imagine.
Re: Show HN: how I built the largest open database of Australian law
#26Australia has had free, searchable collections of Australian Law for 25+ years. Austlii is a prime example. There are Federal and State collections as well. The author is conscientious enough to read the scraping policy (or was blocked by anti-scraping tools) from feeding from one of these sites into his LLM.
As I point out in the introduction, there are a few free-to-access legal databases available in Australia, but none are truly open in the sense of being free from copyright restrictions. Neither AustLII nor Jade are licensed under an open source copyright licence such as CC BY 4.0 (which is what the majority of my Corpus is under).
Love what you're doing! Being able to more easily bring LLMs and other AI in will democratise the law quite a bit. Agreed that even though Austlii exists, it needs to be under a creative commons license, and it takes doing the legwork and dealing with bureaucrats to get it there
Re: Show HN: how I built the largest open database of Australian law
#27Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on. In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web techn…
Hey Umar,
Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions.
I've reach out via your LinkedIn profile - would be great if there was an opportunity to collaborate.
Re: Show HN: how I built the largest open database of Australian law
#28Earlier quoted context omitted.
Github might be difficult as they impose constraints on the size of repositories and the Corpus is around 5GB. The Internet Archive is a good idea, however, I’ll have a look into that. I’ve also been thinking about sticking it on Kaggle as well to increase its reach.
You could also consider one or more of the scientific data repositories like Zenodo, FigShare, DataDryad, etc. 5GB is small potatoes for those folks and they have serious data retention policies. As a bonus, they'll also allocate you a citable DOI.
Thanks for the suggestions! Distributing the Corpus widely will be my next focus.
Re: Show HN: how I built the largest open database of Australian law
#29Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on. In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web techn…
Hey Umar, Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions . I've reach out via your LinkedIn profile - would be great if the…
> Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions.
Absolutely, there's a lack of consistency even within the same jurisdiction and document type. It only gets worse once you want to add multiple jurisdictions and different types of documents. My best strategy so far has been to use recursive chunking where you begin chunking at the largest section of newlines. Ideally though you want some form of semantic chunking where you already know what parts of the document represent Parts, Divisions, Schedules, Sections, Sub-sections, etc...
> I've reach out via your LinkedIn profile - would be great if there was an opportunity to collaborate.
Great! Always happy to connect.
Re: Show HN: how I built the largest open database of Australian law
#30Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on. In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web techn…
Are you open to pushing one of your Work in Progress(WIP) models to https://ollama.ai/library to show off and let others try it out and provide feedback?