Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: how I built the largest open database of Australian law

umarbutler.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Show HN: how I built the largest open database of Australian law

#31

Post by djbusby »

Shorn wrote: post #23
ubutler wrote: post #4

Earlier quoted context omitted.

Github might be difficult as they impose constraints on the size of repositories and the Corpus is around 5GB. The Internet Archive is a good idea, however, I’ll have a look into that. I’ve also been thinking about sticking it on Kaggle as well to increase its reach.

You could also consider one or more of the scientific data repositories like Zenodo, FigShare, DataDryad, etc. 5GB is small potatoes for those folks and they have serious data retention policies. As a bonus, they'll also allocate you a citable DOI.

This is DOI

https://www.doi.org/

Because, I just learned about it.

Re: Show HN: how I built the largest open database of Australian law

#32

Post by RagnarD »

In my experience, one of the devil details is continuously keeping such a database updated. Without a set of common standards among the various governments, they can capriciously change URLs, formatting, and other details that may make it somebody's fulltime job to keep it accurate and always up to date. Of course, not all use cases will require that, but many will.

Re: Show HN: how I built the largest open database of Australian law

#33

Post by ubutler »

nextworddev wrote: post #10

Is there a U.S. equivalent?

There is! It’s called the [Pile of Law](https://huggingface.co/datasets/pile-of-law/pile-of-law), and it’s actually what inspired this project. The only caveat is that it doesn’t appear to be regularly updated, and so is more of a snapshot of US law rather than a semi-live copy. Also I’m not entirely sure how comprehensive it is (ie, whether there’s anything missing). For those interested in building a true US equivalent of my corpus, I think it could be a great starting point, particularly since they published the code they used to build it.

Re: Show HN: how I built the largest open database of Australian law

#34

Post by ubutler »

RagnarD wrote: post #32

In my experience, one of the devil details is continuously keeping such a database updated. Without a set of common standards among the various governments, they can capriciously change URLs, formatting, and other details that may make it somebody's fulltime job to keep it accurate and always up to date. Of course, not all use cases will require that, but many will.

Absolutely. Luckily, the websites I scrape don’t change very often (ie, every 5-10 years based on Wayback Machine), and also a couple of them all use the same underlying legislation management system and APIs.

Ultimately, however, I’m hoping that in the long term, the Australian Government will see the use in this project and decide to maintain it.

Re: Show HN: how I built the largest open database of Australian law

#35

Post by hugh-avherald »

cookie_monsta wrote: post #17

This is cool and I'm a little surprised to see that Victoria is the one dragging the chain here. Is DataVic just talk, or does that not apply to law for some reason?

Victoria is notoriously difficult to work with. Lots of chauvinism/exceptionalism and copyright squatting.

Re: Show HN: how I built the largest open database of Australian law

#36

Post by emmelaich »

Nice, but I have to ask how does it compare with https://austlii.edu.au/, especially for completeness?

The Australasian Legal Information Institute is a great resource and yet seems strangely unknown (to the wider public at least.)

Trivia, the only reason I found out about it was when I did some work for an Aus govt agency and found out that they shared their web site with austlii! This was back in the early 2000s.

Re: Show HN: how I built the largest open database of Australian law

#37

Post by emmelaich »

Obscurity4340 wrote: post #21

What do you think of the Canadian legal case law website CanLii? What could it do better or do you think its done well? Is it overdue for innovation?

Interesting, https://www.canlii.org/en/ looks a bit like https://austlii.edu.au/

Probably copied / cooperated with each other, which is popular for Commonwealth countries.

Re: Show HN: how I built the largest open database of Australian law

#38

Post by Obscurity4340 »

emmelaich wrote: post #37
Obscurity4340 wrote: post #21

What do you think of the Canadian legal case law website CanLii? What could it do better or do you think its done well? Is it overdue for innovation?

Interesting, https://www.canlii.org/en/ looks a bit like https://austlii.edu.au/ Probably copied / cooperated with each other, which is popular for Commonwealth countries.

Any other closing arguments? ;)

Re: Show HN: how I built the largest open database of Australian law

#39

Post by ubutler »

emmelaich wrote: post #36

Nice, but I have to ask how does it compare with https://austlii.edu.au/ , especially for completeness? The Australasian Legal Information Institute is a great resource and yet seems strangely unknown (to the wider public at least.) Trivia, the only reason I found out about it was when I did some work for an Aus govt agency and found out that they shared their web site with austlii! This was back in the early 2000s.

So AustLII and Jade are what I was referring to when I say "While there were certainly a few free-to-access legal databases, none were truly open ...". They are free but not open in the sense of being licensed under an open source licence and being free to download in a raw format. Whereas everything in my corpus except decisions from the Federal Court of Australia is licensed under CC BY 4.0. And even decisions from the FCA are under a licence that permits both non-commerical and commerical usage.

In terms of completeness, however, AustLII and Jade win out. They seem to have almost everything if not everything. Their data is also much richer than mine. I must give props to AustLII for how they're able to hyperlink terms defined within legislation to their definition. I think they're an invaluable resource for members of the public.

The audience of my database was more so those who want to play around with raw legal data and want to feel secure that they are not breaching any laws in the process. The fact that it is stored in plain text is also beneficial for anyone trying to build ML models that only accept raw text.

Re: Show HN: how I built the largest open database of Australian law

#40

Post by triggercut »

ubutler wrote: post #2

Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on. In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web techn…

Just FYI my work's network has blocked your site as "Malicious"

(Symantec Endpoint Protection chrome extension)

Post reply on HN