A database being relational is (mostly) orthogonal to it being ACID. ACID is generally a property of the underlying storage engine (a key-value store, generally).
Relational databases aren’t dinosaurs, they’re sharks
101–110 of 135 posts
Re: Relational databases aren’t dinosaurs, they’re sharks
#102Earlier quoted context omitted.
In my case, I actually do know that; there's even a legal definition of what an address is and there's a national registry of all addresses. The problem is that I can't solve this for every country myself since I only have national knowledge (which is sufficient for my needs, fortunately).
Sounds like, for your purposes, you can define an address type that you can map to columns. But I think you're the outlier. We have fairly regular addresses in the UK. You can enforce a country and a postcode. You can enforce at least two lines of the local address. That's where it ends. I live right now I'm a house that's described as one village everywhere except by the local council, who address us by the other vi…
When validation and parsing fails, there should be room for local fixes. There's an interesting question as to what extent should corner cases be code-driven and to what extent they should be data-driven. Your case definitely sounds like one of those things that would need to be fixed by hand and then associatively recalled (so that they'd need to be fixed only once).
> What is the value of structuring this data?
Well, for example I definitely need it for analytical purposes, so I have to try.
Re: Relational databases aren’t dinosaurs, they’re sharks
#103Earlier quoted context omitted.
That might work if you have infinite resources and can create special handling for individual cities in the world or maybe even individual neighborhoods. Mailing works in a very decentralized way and has a lot of local variations when you go outside of places that have put a lot of effort into standardizing addresses. Most post offices won't be looking at your whole address. They just care about understanding enough…
IMO that just makes it a very interesting problem to work on. > Understanding the final local address might not even use written data. It might just be tacit knowledge that's shared between a few local postal workers. To me there seems to be a contradiction in those two statements. By definition, an address is written. You can't decide where to deliver an item in any other way. If one and the same written text of the…
> You can't decide where to deliver an item in any other way.
Only if you’re a robot, which postal workers aren't. They can use local context, such as no one lives at address X so they must have meant address Y. Or even, person at address X has a birthday this week, so this envelop that looks like a birthday card, and has their misspelled name on it, is obviously for address X not address Y.
I’ve had our friendly postal worker deliver post correctly to me, despite having a throughly munged and incorrect address, because she recognised my name, and knew someone with a similar name didn’t live at the more obvious interpretation of the incorrect address.
So address parsing and mail delivery is an extremely human and imprecise process. Full of nuance and edge cases that can’t even be observed, unless you actually follow the humans making deliveries and see what they’re doing.
Re: Relational databases aren’t dinosaurs, they’re sharks
#104Earlier quoted context omitted.
IMO that just makes it a very interesting problem to work on. > Understanding the final local address might not even use written data. It might just be tacit knowledge that's shared between a few local postal workers. To me there seems to be a contradiction in those two statements. By definition, an address is written. You can't decide where to deliver an item in any other way. If one and the same written text of the…
> You can't decide where to deliver an item in any other way. Only if you’re a robot, which postal workers aren't. They can use local context, such as no one lives at address X so they must have meant address Y. Or even, person at address X has a birthday this week, so this envelop that looks like a birthday card, and has their misspelled name on it, is obviously for address X not address Y. I’ve had our friendly pos…
> Or even, person at address X has a birthday this week, so this envelop that looks like a birthday card, and has their misspelled name on it, is obviously for address X not address Y.
That seems awfully contextual and ad-hoc. Surely this mechanism won't work in many instances unless you only receive mail on your birthday. It's a nice thing if it sometimes succeeds even when it shouldn't, but that's not something you can rely on. And should you get a different mail worker who doesn't know you, poof, your mail is gone.
> I’ve had our friendly postal worker deliver post correctly to me, despite having a throughly munged and incorrect address, because she recognised my name, and knew someone with a similar name didn’t live at the more obvious interpretation of the incorrect address.
Considering that this was presumably a problem with an address written on a physical item as a linear text, that's not quite in the purview of the problems that I'm trying to solve for my own application which needs to process physical addresses of objects (sometimes not even involving people in any way). So I can't comment on mail delivery specifically, sadly.
Re: Relational databases aren’t dinosaurs, they’re sharks
#105Earlier quoted context omitted.
I know what to write on my envelopes so they get to me. You do not. Your job is to make it possible for things you send to get to me, not the other 3 units in the apartment. Good luck with the Oakland addresses in Berkeley.
Yes, I said that the complete type would have to have components developed by others as well, since I can't provide input for other countries. Doesn't mean that a minimum complexity formalization is impossible (in fact, mathematically, by enumeration one such formalization must exist). An "as-given" component is obviously always going to work for you.
What's the point vs repeating what people give?
Re: Relational databases aren’t dinosaurs, they’re sharks
#106Earlier quoted context omitted.
> In my country's case, it's quite clearly defined by law. I theory the same is true in my country. But clearly someone forgot to inform the populous that not using the official standard is criminal, because I’ve seen plenty of “valid” addresses that don’t follow the standard. These addresses are clearly encoding local conventions, which makes decoding using the official standard not only impossible, but nonsensical,…
The type should always include provisions for manual fixes. IMO such an address type should even include a provision for manual geocoding for cases where automated geocoding fails ("This is how I write it, and this is where it's located").
In that case, whats the value of the system?
99% of the time an address is just a “unique” opaque identifier. You ingest the address, then you print it on the parcel. You avoid manipulating or trying to interpret it too much.
A system that attempts to codify address to a standard that can’t express all addresses (including their nuance) is useful. You can’t trust any analytics created from it, because by its nature, address that can’t be interpreted will appear in geographic clusters, and thus skew all your stats.
You can however collect address as opaque strings, and optionally request extra data of a know format (like zipcode or postal code) which is generally considered part of the address. You can then produce stats only on those well know identifiers, and ignore the rest.
But doing that doesn’t require a complicated address type, or supporting address manipulations or any other crap like that. It just requires a free text box, and a separate postal code box.
The most important thing to recognise is that any arbitrary address will fit many different address conventions, but each of those conventions will result in a different location. It practically impossible to definitively interpret an arbitrary address correctly, without significant amounts of additional local context. So its best not to bother, and let the postal workers figure it out using their local knowledge.
Re: Relational databases aren’t dinosaurs, they’re sharks
#107Earlier quoted context omitted.
> You can't decide where to deliver an item in any other way. Only if you’re a robot, which postal workers aren't. They can use local context, such as no one lives at address X so they must have meant address Y. Or even, person at address X has a birthday this week, so this envelop that looks like a birthday card, and has their misspelled name on it, is obviously for address X not address Y. I’ve had our friendly pos…
> Or even, person at address X has a birthday this week, so this envelop that looks like a birthday card, and has their misspelled name on it, is obviously for address X not address Y. That seems awfully contextual and ad-hoc. Surely this mechanism won't work in many instances unless you only receive mail on your birthday. It's a nice thing if it sometimes succeeds even when it shouldn't, but that's not something you…
> That seems awfully contextual and ad-hoc.
Yup, but that’s never stopped someone from relying on a method in the past. The vast majority of addresses aren't written by engineers. If you sent a letter using an address once, and it worked, then most people will just assume it'll always work. How would they know any better?
> It's a nice thing if it sometimes succeeds even when it shouldn't, but that's not something you can rely on.
Have you seen the internet? Or even just HTML? The entire world relies on things working when they shouldn’t. We can talk all day about the merits of that approach, but it wont change reality.
> Considering that this was presumably a problem with an address written on a physical item as a linear text
The text was printed perfectly if thats what your saying. It was just wrong. Some system somewhere had attempted to manipulate it, and ended up misinterpreting the original address, and produced something completely wrong as a result.
> I'm trying to solve for my own application which needs to process physical addresses of objects
That’s slightly different, and presumably you own far more of the process thats producing and interpreting these addresses. I’ve worked in systems that had to deal with addresses created by normal people, and let me tell you, normal people have a very diverse view on how to write addresses.
Re: Relational databases aren’t dinosaurs, they’re sharks
#108Earlier quoted context omitted.
I understand these issues. That's precisely why a custom type seems appropriate -- it should be able to cover all the alternatives while not burdening you with the problem of storing a discriminated union as a set of disparate relations. Or, at the very least, if this approach still has some issues remaining, it should still have fewer issues than any other approach I can think of, because there's nothing you can't d…
> I understand these issues. I seriously doubt you do. The closest you can get to “standard address format” is: Address Line 1 Address Line 2 Address Line n Postal code (which can be blank) Postal Area (which can be blank) Country There is no way to build “first-class” support for addresses. Because theres no such thing as a valid or invalid address, only whether or not someone can find the correct location by readin…
Address Line 1
Address Line 2
Street address
Apartment number
Re: Relational databases aren’t dinosaurs, they’re sharks
#109Great article, and I this particularly brought back memories for me: > [...] in some instances you might work with vast quantities of data, or deal with transactional systems that just don’t easily fit the operational limitations of relational databases. And in those cases, you should consider moving some, or all, of your data into a non-relational database. I've worked with a big application that utilised this appro…
> [...] in some instances you might work with vast quantities of data, or deal with transactional systems that just don’t easily fit the operational limitations of relational databases. And in those cases, you should consider moving some, or all, of your data into a non-relational database.
This is WRONG. With capital letters.
Confuse limitations of (the codebase/the enterprise/the specific RDBMS) with the MODEL:
> And in those cases, you should consider moving some, or all, of your data into a non-relational database.
Making data "non-relational" is not a cure for the above. Is more often than not, a regression (like some on this thread: Using JSON instead of regular tables!).
---
Now where we must focus instead is on make RDBMs more powerful (or more exactly: less constrained that cause less powers).
RDBMS are hurt more for the interfacing (SQL) and the design decisions of 40 years ago, but not because the "relational model" is the reason performance, scalability or anything like that suffer.
P.D: And also: Is incredible how many truly think the "relational model" is agains dynamic data. Even using SQL, that is a subset of it, is HIGHLY dynamic! What is a trouble, is that current RDBMS lack a easy way to embed relations in relation, forcing to detour to sub-optimal, ad-hoc, bug-ridden hacks.
Re: Relational databases aren’t dinosaurs, they’re sharks
#110Great article, and I this particularly brought back memories for me: > [...] in some instances you might work with vast quantities of data, or deal with transactional systems that just don’t easily fit the operational limitations of relational databases. And in those cases, you should consider moving some, or all, of your data into a non-relational database. I've worked with a big application that utilised this appro…
> [...] in some instances you might work with vast quantities of data, or deal with transactional systems that just don’t easily fit the operational limitations of relational databases. And in those cases, you should consider moving some, or all, of your data into a non-relational database. This is WRONG. With capital letters. Confuse limitations of (the codebase/the enterprise/the specific RDBMS) with the MODEL: > A…
> Making data "non-relational" is not a cure for the above.
ACID imposes a limitation on scalability, there is no way to escape that. If you want to claim that approximately nobody reaches that scale, that's true, and if you want to claim that you don't need to ditch your data schema, that's also true. But relations aren't worth much without global coherence, and global coherence requires ACID, and will only work up to some size (that fits almost everyone, but not literally everyone).