Live data from Hacker News

SQL nulls are weird

jirevwe.github.io

251–260 of 293 posts

Re: SQL nulls are weird

#251
post #233

Earlier quoted context omitted.

It’s not. Your link makes the same mistake I already addressed. It conflates nullable booleans with tri-state logic. Null is not a value. It is the absence of a value. > The SQL null value basically means “could be anything”. This is wrong. Null means it could be any valid value but that value is unknown. If the datatype is DATE then the value cannot be boolean TRUE or the string ‘purple’.

How is that different than “anything”? If I’m comparing a value of type date to a null I still think it works as it should if value is “unknown”. What greater insight or context do we have if it’s a small-int null?

> How is that different than “anything”?

Because the possible values are known.

> What greater insight or context do we have if it’s a small-int null?

The insight is that null is not a value. It’s not a smallint or a boolean or a date. It’s the absence of a possible value. The only way to see a null as tri-state is to conflate it with a nullable boolean. This is an incorrect mental model which leads to confusion.

Re: SQL nulls are weird

#252

SQL NULLs aren't weird , they're just based off of Kleene's TRUE-FALSE-UNKNOWN logic! If you want you can read NULL as UNKNOWN and suddenly a whole bunch of operations involving them become a lot more intuitive: 1. TRUE OR UNKNOWN = TRUE, because you know you have at least one TRUE already. 2. TRUE AND UNKNOWN = UNKNOWN, because you don't know whether you have two TRUEs or not. It's just out there. 3. UNKNOWN XOR UNK…

there is a pattern starting to emerge here on hackernews of highly voted posts by people who present themselves as experts and thought leaders who shamelessly put their lack of understanding at display. it's frightening.

> starting to emerge here on hackernews

It's not getting worse, you're getting better.

HN has for a long time been where I go for whatever you call the tech equivalent of watching stoners think they're having a deep conversation.

Re: SQL nulls are weird

#253
post #166

Earlier quoted context omitted.

No, it's not those other things, that's just using the tool incorrectly. A NULL is definitely "we dont know", not false, not N/A, especially not any known value.

Except in every other programming language with a null, null is the definite absence of something

Sure, and we're talking about SQL nulls in this context, which is why I am strict in my definition.

Re: SQL nulls are weird

#254
post #239

Earlier quoted context omitted.

> And in many contexts in SQL, NULL is treated as equivalent to false, such as within a WHERE clause. I don't think any databases treat `NULL` as `FALSE` in the WHERE clause. `SELECT * FROM foo WHERE bar = NULL` doesn't return rows with a NULL in the bar column. `SELECT * FROM foo WHERE bar != NULL` doesn't return rows without NULL in the bar column. `SELECT * FROM foo WHERE (bar = 'a') = NULL;` doesn't return rows w…

It treats the NULL/unknown value of the boolean as false 1 NULL => Boolean UNKNOWN, so SELECT * FROM foo WHERE 1 NULL returns nothing. 1 = NULL => Boolean UNKNOWN, so SELECT * FROM foo WHERE 1 = NULL returns nothing. That's the thing that's being treated as FALSE. That UNKNOWN. Not the value of NULL itself. > You're not doing `X = X -> false`, you're doing `X = UNKNOWN -> UNKNOWN` That's not how "=" works. If you wan…

> That's the thing that's being treated as FALSE. That UNKNOWN. Not the value of NULL itself.

No, it's being treated as UNKNOWN, and the semantics of SELECT...WHERE only returns rows where the value of the condition is TRUE.

I think you need to look into https://en.wikipedia.org/wiki/Negation_as_failure

Re: SQL nulls are weird

#255
post #121

Earlier quoted context omitted.

Stop thinking of ORMs as trying to hide the details of SQL and you'll stop hating them. Instead think of them as a way to compose relational queries dynamically, with the full power of your primary language, instead of inside of database stored procedures in a language totally devoid of any support for basic software engineering best practices. They shouldn't be hiding SQL from your primary language, they should be e…

>They shouldn't be hiding SQL from your primary language, they should be exposing the relational model to it! But this has never been their primary purpose and it's not what they are good at. ORMs are supposed to map the relational model into an object oriented model so that you can work with objects rather than sets of tuples. And that's exactly how people use them. ORMs incentivise people to replace simple and decl…

Exposing the relational model to be manipulated dynamically within the parent language is exactly what LINQ in C# is. That was its primary purpose. True, LINQ is not itself an ORM -- it was built to support other ORMs like LINQ to SQL and Entity Framework, which aren't as "pure" on this subject. I don't actually like the LINQ syntax that much since it's not as extensible, but its existence is proof that the C# team did in fact intend to expose the relational model to C#.

Entity Framework did try to cater to the "SQL is scary, let me use objects" crowd, and that is the majority of how it's used, and that is a mistake in my opinion. But it is also very good at supporting relational algebra within C# and composing queries dynamically; ironically, it's best at it if you disable or avoid many of its features (dynamic subclassing, linked objects). Pass IQueryables around and compose them together and life is good. Updating needs work, but updates have always been a weakness for SQL too.

Re: SQL nulls are weird

#256
post #224

Earlier quoted context omitted.

The SQL null is a database-specific keyword and not something that's part of the domain of your measurements. If you want some kind of sentinel value for your measurements that doesn't have the "unknown" meaning of SQL null, then you should use your own sentinel value and not reuse null for that purpose

Sentinel values suck especially when the language already has a perfectly good one built in. Is 0 for a temp measurement unknown sentinel or an actual measurement, how about 2,147,483,647 great probably not a measurement now its always included in greater than queries same with max negative with less than. Null separates the value into its own distinct group and prevents it from being including in range queries due t…

> Null separates the value

NULL is not a value.

NULL is a statement that a value is not available or unspecified reasons.

If you want a particular value where a query would return NULL, it's your job to replace the NULLs with the contextually-appropriate value, e.g., using COALESCE(), to provide it.

It's a convenience shortcut to allow more complicated data models to be rpersented in simpler table structures than a fully normalized NULL-free data model would require, and to provide information about missing data (which can be used with things like COALESCE, where appropriate) when a more complex data model is simplified into a resultset via a query with JOINS, etc.

Re: SQL nulls are weird

#257
post #120

Earlier quoted context omitted.

I don't know Django specifically but I'm always floored by how people talk about ORMs. They're only a leaky abstraction if you believe their point is to shield terrified junior devs of the inner workings of the scary relational database. That's an awful way to use ORMs, and the source of most of the flak they get. To be fair, some are designed that way, or at least strongly push you toward it. Stop thinking of ORMs a…

Are you thinking of something like LINQ or SQLAlchemy Core? You do not need to use an ORM to interface with a SQL database with basic software engineering principles. The reason for a data layer is because the underlying data representation might change. For example, you might change the schema to handle some new performance requirement. Now you have to hunt down and change everywhere you've queried that table in you…

> Are you thinking of something like LINQ or SQLAlchemy Core? You do not need to use an ORM to interface with a SQL database with basic software engineering principles.

Well you certainly can't do it all in SQL, because SQL doesn't support basic software engineering principles at all. That means you're picking a parent language that does support the principles you want, and attempting to build a library that allows you to build, pass around, and compose relational queries in a way that leverages the benefits of the parent language. To do this I'd argue you need these things:

1. a structured representation of the (possibly incomplete) relational query as it's being built, e.g. SQLAlchemy's internal representation or IQueryable in C#

2. a method of building those representations in a way that conforms to the norms of the parent language and leverages its benefits as much as possible, e.g. LINQ or direct use of the various extension methods on IQueryable. It seems like you could pick either SQLAlchemy or SQLAlchemy Core depending on whether you want to think in terms of objects or queries (I'd usually pick queries), but I'm not that familiar with SQLAlchemy.

3. a method of translating the query representation into a particular SQL dialect.

I don't know what exactly your definition of "ORM" is, but I'd argue that if any of those steps involve an object-level representation of the schema of your tables (e.g. "class Employee { int ID }" type of thing) then it's an ORM. Do you need that? Well, no, probably not, but why not? C# is strongly typed, so why wouldn't I tell its type system what the schema of my table is? That's part of leveraging the benefits of the parent language.

> The reason for a data layer is because the underlying data representation might change. For example, you might change the schema to handle some new performance requirement. Now you have to hunt down and change everywhere you've queried that table in your whole code base.

This is an argument for strong typing, not for putting all your code that has the word "select" in it in the same file. And building queries with a strongly typed representation of your schema in the parent language is the point of the ORM!

Without an ORM, you still have to hunt down tons of spots where things change when you change your schema. How do you ever not have to do this?

> Every time you directly call SQL you are coupling your code strongly to the database schema which is ultimately an implementation detail.

There exists some canonical relational model of your domain. This model is not an implementation detail, it is the foundation of most of your code. I choose to keep my SQL schema as close as possible to this canonical relational model as I can. The fact that a Monitoring Instrument has an Operating Status is a fundamental fact about my model; whether that column is called "operating_status" or "OperatingStatus" or "operating_status_ID" is an implementation detail. It's just hard to agree with you that "the database schema" is merely implementation detail -- clearly parts of it are not, unless it's completely untethered from your canonical domain model (which would be a nightmare, of course). Of course your code is strongly coupled to the fundamental domain model upon which it is operating. I'd certainly agree that it'd be nice to not have to worry about implementation details like the exact naming convention of the columns -- which, of course, ORMs let you do.

>> And convenience is a real thing that actually does reduce costs.

> I know convenience is a real thing. I also know that it very often increases costs in the long run.

Code that is easier to write is generally also easier to read and maintain. That's what I mean when I say "convenience". Simplicity and convenience go together.

Re: SQL nulls are weird

#258

Earlier quoted context omitted.

Then you're supposed to use another table with a foreign key to canonical measurement record. This is the concept of fully normalized schemas. What you're describing is closer to how people do it in practice.

I'm still a bit confused. Suppose you have another table, call it temperatures with columns id and temperature , where every row contains only a valid temperature (no NULL records), and you have a main logging table with date and temperature_id so that you can join on temperature_id = temperatures.id . This seems to be what you mean, with a canonical measurement record table related via the temperature_id foreign key…

Foreign key would probably go the other way:

LogEntry(LogEntryId, Date)

Temperature(TemperatureId, LogEntryId, DegreesF)

If there is no temperature measured, then you don't create a record in Temperature.

Re: SQL nulls are weird

#259

Earlier quoted context omitted.

Then you're supposed to use another table with a foreign key to canonical measurement record. This is the concept of fully normalized schemas. What you're describing is closer to how people do it in practice.

Ah but then how do you record that the measurement actually happened but did not produce a value? I want a record of that failure. I mean sure, you could do yet another table. But honestly that level of normalization is much more work than it's worth. Just because it's some academic definition doesn't make it right.

You put a record in the Measurement table, and none in the Value table.

> But honestly that level of normalization is much more work than it's worth

Yes. I question whether it's worth anything to begin with.

Re: SQL nulls are weird

#260

Earlier quoted context omitted.

This is confusing when you know that NULLs are not comparable, but it makes some sense if you consider the result of distinct/union as the output of a GROUP BY. You can consider everything that's NULL to be part of the same group, all the values are unknown. So NULLs are not comparable but they are part of the same set.

If nulls are distinct then group by should not group them together, this just ignores the problem. Why does group by treat them as equal?

This is where the foundation of a relational database semantics in set theory shows through. You can’t model the behaviour with pure boolean logic.

In the SQL spec by default unique indexes consider nulls distinct because you’re adding tuple to the relation, and this is done by equality.

When doing a select distinct or group by you’re not doing a boolean grouping, you’re doing a set projection. NULLs are considered part of the set of “unknown values”, so NULLs are grouped together but they’re still not equal to each other.

The behaviour is mathematically well defined, but it doesn’t match boolean logic.

I’ve been dealing with databases for well over 20 years now and I can only think of a couple of times when this behaviour wasn’t wanted, so I think it’s the right choice.

Post reply on HN