Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: how I built the largest open database of Australian law

umarbutler.com

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Show HN: how I built the largest open database of Australian law

#41

Post by ubutler »

triggercut wrote: post #40
ubutler wrote: post #2

Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on. In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web techn…

Just FYI my work's network has blocked your site as "Malicious" (Symantec Endpoint Protection chrome extension)

Weird. Thanks for the heads up. I'm using SiteGround to host it which is shared hosting so perhaps that's why? I'll have to investigate.

Re: Show HN: how I built the largest open database of Australian law

#42

Post by ubutler »

benn0 wrote: post #22
ubutler wrote: post #2

Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on. In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web techn…

Fantastic work, and really appreciate the write up. It's quite timely for me - I'm from a tech background and have just started studying Australian law, and was thinking about doing exactly this - so you are years ahead of me :). Just one note - the link in your Github readme to https://umarbutler.com/open-australian-legal-corpus doesn't seem to go anywhere. For someone interested in using the data (and help out with…

> Just one note - the link in your Github readme to https://umarbutler.com/open-australian-legal-corpus doesn't seem to go anywhere.

Thanks for the heads up! I've fixed that now.

> For someone interested in using the data (and help out with bugs/issues), where would you suggest starting?

I think the best place to start is by downloading the Corpus (visit https://huggingface.co/datasets/umarbutler/open-australian-l... , and then click "Files and versions" and then "corpus.jsonl"). You can then use my Python library orjsonl to parse the dataset (you'd run, `corpus = orjsonl.load('corpus.jsonl')`). At that point, there's any number of applications you could use the dataset for. You could pretrain a model like BERT, ELECTRA, etc... and share it on HuggingFace. You could connect the dataset to GPT and do RAG over it. Etc...

Re: Show HN: how I built the largest open database of Australian law

#43

Post by ubutler »

duttonw wrote: post #30
ubutler wrote: post #2

Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on. In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web techn…

Are you open to pushing one of your Work in Progress(WIP) models to https://ollama.ai/library to show off and let others try it out and provide feedback?

Unfortunately, I haven't been able to train a model on my database just yet (although I have seen great results using it as a RAG data source for GPT-4). I'll keep ollama in mind once I have something to share.

Re: Show HN: how I built the largest open database of Australian law

#44

Post by triggercut »

ubutler wrote: post #29
SgtBastard wrote: post #27

Earlier quoted context omitted.

Hey Umar, Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions . I've reach out via your LinkedIn profile - would be great if the…

> Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions. Absolutely, there's a lack of consistency even within the same jurisdicti…

Yes, this. I've been trying to find a general way to automatically semantically chunk various legislation for a while now. Partly so as to diff various versions/amendments, but also to graph connections to other referenced legislation.

Most of the time I end up having to just take half an hour to manually regex and format plain text.

A particular case I have is where there is a draft bill put out for industry/community consultation. Quickly diffing the releases is the goal but for now usually relies on one (preferably two) subject matter experts to read the whole thing top to bottom to build an understanding. I don't think these would be available via the means you've secured. They are usually hosted on a relevant government entities website as PDFs

One last question/comment, have you considered adding some additional reference info like the federal list of entities?[1]

[1] https://www.finance.gov.au/government/managing-commonwealth-...

Re: Show HN: how I built the largest open database of Australian law

#45

Post by ubutler »

Obscurity4340 wrote: post #21

What do you think of the Canadian legal case law website CanLii? What could it do better or do you think its done well? Is it overdue for innovation?

It actually looks quite clean. Certainly a lot better than some of our legal databases. I guess the only suggestion I'd have is that it seems like you can have an account with the website but there's login or register button on the front page (or maybe I'm just not seeing it?).

One other point, and this is not specific to CanLII (I haven't checked whether this is the case) but I've seen that a lot of legislation databases have poor SEO. In fact, AustLII is usually ranked higher than governments' own websites when searching for laws. I think it's something important to get right because a lot of people just use Google/Bing/Kagi to search websites nowadays rather than using internal search engines.

Re: Show HN: how I built the largest open database of Australian law

#46

Post by ubutler »

emmelaich wrote: post #37
Obscurity4340 wrote: post #21

What do you think of the Canadian legal case law website CanLii? What could it do better or do you think its done well? Is it overdue for innovation?

Interesting, https://www.canlii.org/en/ looks a bit like https://austlii.edu.au/ Probably copied / cooperated with each other, which is popular for Commonwealth countries.

Correct. They are a member of WorldLII which was founded by AustLII: http://www.worldlii.org/

Re: Show HN: how I built the largest open database of Australian law

#47

Post by ubutler »

triggercut wrote: post #44
ubutler wrote: post #29

Earlier quoted context omitted.

> Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout consistency within a piece of legislation, that's not true across pieces of legislation... so I can imagine the pain of trying to collate legislation across jurisdictions. Absolutely, there's a lack of consistency even within the same jurisdicti…

Yes, this. I've been trying to find a general way to automatically semantically chunk various legislation for a while now. Partly so as to diff various versions/amendments, but also to graph connections to other referenced legislation. Most of the time I end up having to just take half an hour to manually regex and format plain text. A particular case I have is where there is a draft bill put out for industry/communi…

> A particular case I have is where there is a draft bill put out for industry/community consultation. Quickly diffing the releases is the goal but for now usually relies on one (preferably two) subject matter experts to read the whole thing top to bottom to build an understanding. I don't think these would be available via the means you've secured. They are usually hosted on a relevant government entities website as PDFs

It's possible that they're in my database. I have included the as made version of all bills on the Federal Register of Legislation. However, if they haven't had a first reading yet, then probably not.

For processing PDFs, I recommend using `pdfplumber`, which is what I used to build the Corpus. Happy to discuss further if you'd like.

> One last question/comment, have you considered adding some additional reference info like the federal list of entities?

Do mean adding additional metadata? At the moment, I've kept the number of metadata attributes as low as possible. Every attribute added equates to more work to keep it standardised across all the jurisdictions and document types. My plan is to slowly add more attributes as I have time. I'd really like to associate a date with documents but even that is a hurdle. I have to decide what date should be the date of a document (is it the time it was issued, the time it was published, the time it came into force, the time the latest version was issued, etc... and what happens when a document doesn't have a date? should I extract it from its citation? how do I preserve time zone information? etc...).

Re: Show HN: how I built the largest open database of Australian law

#48

Post by emmelaich »

darcys22 wrote: post #19

So good! Its crazy how legal information is such a spread out mess. Whats worse is that git is such a perfect solution for legislation.

The French constitution has been on github for quite a while.

https://github.com/legifrance

Probably others, haven't looked.

Re: Show HN: how I built the largest open database of Australian law

#49

Post by defrost »

emmelaich wrote: post #48
darcys22 wrote: post #19

So good! Its crazy how legal information is such a spread out mess. Whats worse is that git is such a perfect solution for legislation.

The French constitution has been on github for quite a while. https://github.com/legifrance Probably others, haven't looked.

There's a bit of a difference between a few pages of founding documents and a complete history of precedent setting legal cases.

Re: Show HN: how I built the largest open database of Australian law

#50

Post by triggercut »

ubutler wrote: post #47
triggercut wrote: post #44

Earlier quoted context omitted.

Yes, this. I've been trying to find a general way to automatically semantically chunk various legislation for a while now. Partly so as to diff various versions/amendments, but also to graph connections to other referenced legislation. Most of the time I end up having to just take half an hour to manually regex and format plain text. A particular case I have is where there is a draft bill put out for industry/communi…

> A particular case I have is where there is a draft bill put out for industry/community consultation. Quickly diffing the releases is the goal but for now usually relies on one (preferably two) subject matter experts to read the whole thing top to bottom to build an understanding. I don't think these would be available via the means you've secured. They are usually hosted on a relevant government entities website as…

I've used a number of pdf libraries in python and C# over the years, none have worked reliably as needed (that's just pdf I guess), but haven't used pdfplumber, I'll be sure to give it a go, thanks for the suggestion.

Yes, additional metadata. Totally understand it adds in a lot of complexity but could help for fine-tuning an LLM.

With regards to dates, not a lawyer, but for Federal I would go with "Start Date", it's always the day following the End Date of the previous comp. The Date of Assent (well the year at least) is in the title, but also the first start date. The registration date can be either before or after the start date depending. [1][2]

The tricky part is when sections have different commencement dates that are detailed in the text. I don't know anywhere that is easily accessible. And, if you think about it, usually the most important information for say businesses being regulated.

I wouldn't worry with timezone per say, it's relative to each particular state.[3] i.e. why polling closes in a federal election at 6pm in each state rather than coordinated with ACT.

[1] Section 12 of the Legislation Act 2003 https://www.legislation.gov.au/Details/C2023C00213

[2] Sections 4 Acts Interpretation Act 1901 https://www.legislation.gov.au/Details/C2023C00213

[3] Sections 37 Acts Interpretation Act 1901 https://www.legislation.gov.au/Details/C2023C00213

Post reply on HN