Earlier quoted context omitted.
How do you model "postal address"? Some postal addresses are PO Boxes, some are street addresses, etc. There are canonical representations of these different cases. Do we just shove it all in a string, and let the application perform domain validation?
Each type of postal address is a separate column. New postal address "types" would get new columns. This works particularly well when addresses can have both PO boxes as well as street addresses. This is actually more flexible than tagged unions/sum types, at least for this particular case.
Old, Good Database Design
81–90 of 167 posts
Re: Old, Good Database Design
#82Earlier quoted context omitted.
I'd like to know this as well. I think you'll just have to build things (potentially horribly) and fail. I took three semesters of database (granted, baby database classes) and I still have no idea how you can do something pretty straightforward like creating a room reservation system. If there is a reservation beginning at 10:15 AM and ending at 12:30 PM and someone tries to book a reservation from 10:00 AM to 10:30…
The documentation for Postgresql range types describes how to do exactly this. https://www.postgresql.org/docs/11/rangetypes.html#RANGETYPE... Edit: and if you didn't want to use postgres, you could have "starttime" and "endtime" columns and reject any bad bookings with a before insert / before update trigger.
I had to solve this recently, where the actual start/end times were stored on a related table. I'm no SQL wizard, but I'd love to share my solution in case it helps others (it might be terrible).
Note: I changed the actual tables/domain to be generic, this is a poor example and it made more sense for my usecase, but this shows general approach.
-- Let's pretend we have these tables (awful design, but for sake of example):
-- room reservation reservation_info
-- Where "reservation_info" has "start_time" and "end_time"
CREATE FUNCTION check_for_overlapping_reservations()
RETURNS trigger
LANGUAGE plpgsql AS
$$ BEGIN
IF (
-- Find the newly created reservation and join it with the info record to grab "start_time" and "end_time" for check below
with this_reservation as (
select * from reservation
inner join reservation_info on reservation_info.id = reservation.reservation_info_id
where reservation.id = NEW.id
), bookings_for_timerange as (
-- Select every other reservation, where the reservation is happening in the same room
select * from reservation as other_reservation, this_reservation
inner join reservation_info as other_reservation_info
on other_reservation_info.id = other_reservation.reservation_info_id
where other_reservation.room_id = this_reservation.room_id AND
-- And the timerange from start to end overlaps the newly created record
tstzrange(this_reservation.start_time, this_reservation.end_time) &&
tstzrange(other_reservation_info.start_time, other_reservation_info.end_time)
-- Get a count of all the records, it should only be 1. If it's greater than one, there's overlap.
select count(*) from bookings_for_timerange
) > 1
THEN
RAISE EXCEPTION 'Room is already reserved during this time period';
END IF;
RETURN NEW;
END;$$;Re: Old, Good Database Design
#83My least favorite part of database design is the bit where you have to pick lengths for your char columns. Twenty years in and I'm still picking these pretty much by guessing. And when I guess wrong it causes really annoying problems further down the line. I love how SQLite doesn't make me do this - it just has a TEXT type which is always unlimited in length.
Why don’t you use varchar(max) as the range always. The varchar data type specified that the length of this attribute is variable in each record and the memory allocated depends only on the number of actual characters stored in the column.
I’ve worked on more projects than I care for which had nvarchar(max) columns for storing the contents of a small 3-4 line HTML textarea: most users were expected to type in less than 100 words or copy-paste the output of another program.
One day, that other program had a bug that made it generate about a gigabyte of textual output. That program had a “Copy output” button so the user didn’t realise how much data they were copying. I don’t know how it didn’t timeout when it was inserted, but that user brought the system down for everyone because that gigabyte-sized text value was used in lots of places.
Re: Old, Good Database Design
#84Some people choose nosql alternatives because they've spent time analyzing the performance of a proper relational model and have determined that an RDBMS will generate too much overhead for their data load and consciously accept the tradeoffs involved in giving up automated referential integrity. Most people, though, choose nosql alternatives because they're too lazy to learn how to model data.
Start off with relational, if perf is a problem then look at denormalising, after that then consider other alternatives for special cases. But to see run-of-the-mill apps with no near future scalability issues jumping right into mongo et al from day one makes me want to run away.
Re: Old, Good Database Design
#85I'm wondering what you guys think about columnar databases and wide tables. We use Vertica and from senior colleagues and even Vertica developers I got the impression that big wide tables are good because it eliminates the needs of join. Thus we don't use star schema and just wide tables. However I think data modelling is also about embedding proper business logic and it would be a lot more confusing if two unrelated…
- Wide tables in columnar DBs can make some analytics queries easier to write and sometimes more performant.
- Wide tables can come at high storage cost and make other queries less performant (like SELECT *)
- How much of these things happen is extremely DB dependent. How does the DB's underlying storage mechanism work? How is the data partitioned and distributed? How sophisticated and storage-aware is the query planner? How large is your data? How fast is your data growing? How fast do you need a new data point to be reflected in your dashboard?
There's no free lunch when it comes to this stuff. A perfect solution doesn't exist, but the technology is getting better all the time. I've personally never used sql server as a data warehouse but plenty of folks do.
The stuff I use that I recommend very highly - Snowflake, TimescaleDB, vanilla Postgres. Also, use dbt.
Re: Old, Good Database Design
#86If one of the purposes of relational databases is data modeling, I've always wondered why there aren't good semantics for sum types. The real world is full of them, but databases can't express them. When I bring this up, some people respond that this is the purpose of ORMs; however, this implies that we have an arbitrary bifurcation in which some of the processing happens efficiently in SQL and anything that depends…
What's difficult about them? I typically use nullable columns and then a check constraint to specify a custom condition for nullability. Columns belonging to the same alternative in the sum type must be all null or all not null. And then there's check only one active alternative.
Re: Old, Good Database Design
#87Earlier quoted context omitted.
I am forever grateful that I took a full semester of database design in my undergrad. This single skill has stood with me for my entire career so far and has enabled me to figure out the root cause of many production issues. Plus people really like it when you can answer ad-hoc questions like "what inspections are still open and when were they first opened". If y'all can understand Angular / React / Vue there's no re…
Could you suggest resources (books, articles, videos, moocs or others) to learn good database design. I am picking up skills about sql but want to better understand and learn about databases. As someone who doesnt have that background, a lot of the times I am just googling for stuff and just trying out bits and pieces.
Re: Old, Good Database Design
#88> A well-thought design can save us many hours of coding, testing, and troubleshooting. That is the very definition of a waterfall design model. I've turned into a fluid-design advocate over the years, where every design principle follows a next question - "okay, this is good but how would I change it?". So you start with a unique constraint and four months later, you find out that it is not actually unique (like "tw…
I think you're conflating two different things. I've spent the last two days thinking about the impact of adding five new tables to our database for a feature, two of them just lookups. I've thought about the short term benefits and the long term possible problems (for something that isn't even spec'd yet).
This isn't "waterfall" this is just getting a design about right for the current circumstances.
If during the design phase you're already asking "okay, this is good but how would I change it?" then you're already waterfalling your design.
Being thoughtful about your design up front (waterfall as your claim) hopefully solves many a problem down the line when you've suddenly got 5900M+ rows of data in the wrong shape in your production database; because a production database of that size doesn't take kindly to being "agiled" around.
Re: Old, Good Database Design
#89I'm wondering what you guys think about columnar databases and wide tables. We use Vertica and from senior colleagues and even Vertica developers I got the impression that big wide tables are good because it eliminates the needs of join. Thus we don't use star schema and just wide tables. However I think data modelling is also about embedding proper business logic and it would be a lot more confusing if two unrelated…
A few comments based on lots and lots of experience: - Wide tables in columnar DBs can make some analytics queries easier to write and sometimes more performant. - Wide tables can come at high storage cost and make other queries less performant (like SELECT *) - How much of these things happen is extremely DB dependent. How does the DB's underlying storage mechanism work? How is the data partitioned and distributed?…
>How much of these things happen is extremely DB dependent. How does the DB's underlying storage mechanism work? How is the data partitioned and distributed? How sophisticated and storage-aware is the query planner? How large is your data? How fast is your data growing? How fast do you need a new data point to be reflected in your dashboard?
I think most of my frustration comes from not knowing these stuffs. I work as a BA-BI hybrid as I found my data analysis skills are good complements to data modelling/airflow type of work, so I persuaded my manager to give me some BI work. But that also means I don't have the technical knowledge such as DB internals (and TBH I can't even find a book for Vertica on that matter).
Our DB and DBAs and all ETL processes are located in HQ and we actually don't own our databases. This, I guess, adds an extra layer of discomfirt as we are effectively cut off from database-level optimization. Our data engineer is about to leave because he has no DE work to do (every ETL has to go through HQ's process and we only need to submit some configuration files).
We don't have access to the databases you recommended (again HQ controls that), but I do believe I could try DBT, may I ask how do you use it? From my understanding it is mostly a transformation tool, but what makes it stand out?
Re: Old, Good Database Design
#90Earlier quoted context omitted.
Can you give an example of real world data modeling where you want more expressive sum types over just using enums? Enums are technically a subclass of sum types, but even those are non-trivial to use at a data format level (Try evolving them in an on-the-wire message format like Avro or Protobuf).
How do you model "postal address"? Some postal addresses are PO Boxes, some are street addresses, etc. There are canonical representations of these different cases. Do we just shove it all in a string, and let the application perform domain validation?
the most common schema I've seen is usually something like line1, line2, line3, city, state, country, zip, etc. if it's a reporting database then city/state/country/zip is often mashed into some sort of location id.