Earlier quoted context omitted.
So you have more complicated SQL then, but it doesn't make a difference between DBs selecting from a 100M table with indexes. If it ends up being a full table scan then PG only recently got query compilation and parallel execution so at best it would be even with, but not better than, MSSQL/Oracle that can do vectorized executions on columnstores.
You need different index structures. Yes, it matters. Hard to believe that spatial queries over geometries more complex than points will meaningfully benefit from "vectorized executions on columnstores".
PostgreSQL is the worlds’ best database
311–320 of 365 posts
Re: PostgreSQL is the worlds’ best database
#312I get the Postgres love but as someone who’s been doing mysql for a long time I found postgres really hard to use. Like even setting it up on Mac was a pain in the ass. How do I create a new db? Why is pgcli not as friendly as mysqlcli? After 2 weeks of head bashing and lots of stack overflow reading I gave up and went back to good ol MySQL. For ease of use for a new comer, mysql seems to have nicer tooling and out o…
What was your problem? All you need to call is:
docker run --rm --name pg-docker -e POSTGRES_PASSWORD=docker -d -p 5432:5432 -v $HOME/docker/volumes/postgres:/var/lib/postgresql/data postgres:latest
...and you are good to go. In depth guide: https://hackernoon.com/dont-install-postgres-docker-pull-pos...
Re: PostgreSQL is the worlds’ best database
#313Having migrated a postgres database to dynamodb, I would say that I'm never going to use postgres again. The problem with all SQL databases is that they are too easy to query and use. You add all kinds of select queries, joins and foreign keys and when traffic hits scramble to make it scale. NoSQL is hard to design but you can atleast be sure that once traffic hits, you don't have to redesign the schema to make it sc…
You don't have joins in NoSQL so you have to keep data denormalized. That means data isn't consistent and you can have an will have (at a high enough data volume) errors if you aren't extremely cautious on how you update and add new data. That might matter or not in a particular business case.
Re: PostgreSQL is the worlds’ best database
#314It is hard to explain, at this remove, what horrible perversions were performed in order to be able to say "yes" to such questions, or to avoid need to answer them.
One example was a router that, at opposite ends of a long-haul network, managed data flow that was utterly insensitive to the magnitudes of latency or drop rate. They sold hardly any. It turned out the algorithm could be run in user space over UDP, and that became a roaring business, for a different company, because you could deploy without getting network IT involved. Mostly IT didn't even want bribes. They just didn't want anything around that was unfamiliar.
Database products had to pretend to be an add-on to Oracle, and invisibly back up to, and restore from, an Oracle instance, because Oracle DBAs only ever wanted to back up or restore an Oracle instance. The Oracle part typically did absolutely nothing else, but Oracle collected their $500k (yes, really) regardless.
Re: PostgreSQL is the worlds’ best database
#315Earlier quoted context omitted.
You could design a SQL database to be denormalized from the start, but then you are losing many of the advantages of a SQL database. I never said that schemas aren't necessary, just that using a strict NoSQL database forces you to think about scaling constraints early. This avoids(atleast partially) a schema redesign later.
Ah yes! Optimize early, that's what my teachers always told me to do. /s
Re: PostgreSQL is the worlds’ best database
#316Isn't this a fairly big issue? I would think the convenience of automated scaling is a primary reason to use these other tools.
Re: PostgreSQL is the worlds’ best database
#317Earlier quoted context omitted.
Oracle has not managed the shift to cloud so if you look at where the puck is going, it's moving away from Oracle.
Oracle has their own cloud platform and a great database to go along with it, if anything history has shown us that Oracle is able to adapt their database offerings to cater to current enterprise needs and I think this time will be no different.
Re: PostgreSQL is the worlds’ best database
#318Re: PostgreSQL is the worlds’ best database
#319Earlier quoted context omitted.
Good list. Also true indexed organized table aka real clustered indexes. Oh and real cross connection query plan caching, prepared statements are only for the connection and must be explicitly used. No need to use prepared statements in MSSQL since the 90's
Another thing I'd add is a quasi-sorted uuid generator built-in. Real clustered indexes need approximately sorted UUIDs - which could be version 1 UUIDs with an artificial, per-database MAC address, or they could be something more esoteric such as Twitter's snowflake ID generator. Using UUIDs for PKs is fine and dandy but clustered indexes and type 4 UUIDs do not play well. Many MS SQL users discover this the hard wa…
Re: PostgreSQL is the worlds’ best database
#320For most of the projects where the DB really mattered, throughout my 10+ freelancer carrier, it came down to one thing that client really cared about. Performance. Nothing else mattered, not license price, not whistles and bells, not hype. My clients wanted to have data in front of their eyes the same second when they clicked the button. And when you have a table with 100 million rows in it, and an application is not…
Honestly, if it's taking your database more than a second to pull 1000 of 100M rows on a simple query, that means you need to figure out what's wrong with your indexes, not that your choice of database vendor is bad. It is true that postgres is a very nice database, but for basic tasks, honestly any database will be fine if you know how to use it effectively.
Demo in mysql:
-- create a helper table for inserting our 100M points
create table nums (
id bigint(20) unsigned not null
);
insert into nums (id)
values (0), (1), (2), (3), (4), (5), (6), (7), (8), (9);
-- the table in which we will store our 100M points
create table points (
id bigint(20) unsigned NOT NULL AUTO_INCREMENT,
latitude float NOT NULL,
longitude float NOT NULL,
latlng POINT NOT NULL,
primary key (id),
spatial key points_latlng_index (latlng)
);
-- insert the 100M points with latitude between 32 and 49
-- and longitude between -120 and -75, which is roughly a
-- rectangle covering the continental US, placing them
-- randomly. Be patient, as it takes a while to create
-- 100M rows.
insert into points (latitude, longitude, latlng)
select
ll.latitude,
ll.longitude,
ST_GeomFromText(concat('POINT(',
ll.latitude, ' ', ll.longitude,
')')) as latlng
from (
select
32 + 17 * conv(left(sha1(concat('latitude', ns.n)), 8), 16, 10) / pow(2, 32) as latitude,
-120 + 45 * conv(left(sha1(concat('longitude', ns.n)), 8), 16, 10) / pow(2, 32) as longitude
from (
select
a.id * pow(10, 0) + b.id * pow(10, 1)
+ c.id * pow(10, 3) + d.id * pow(10, 4)
+ e.id * pow(10, 5) + f.id * pow(10, 6)
+ g.id * pow(10, 7) + h.id * pow(10, 8)
as n
from nums a join nums b join nums c join nums d
join nums e join nums f
join nums g join nums h
) as ns
) as ll;
-- Generate a roughly circular boundary polygon 7km around
-- the center of Las Vegas
set @earth_radius_meters := 6371000,
@lv_lat := 33.17,
@lv_lng := -115.14,
@search_radius_meters := 7000;
set @boundary_polygon := (select
ST_GeomFromText(concat('POLYGON((', group_concat(
concat(boundary_lat, ' ', boundary_lng)
order by id
separator ', '), '))')
) as boundary_geom
from (
select
nums.id,
@lv_lat + @search_radius_meters * cos(nums.id * 2 * pi() / 9)
/ (@earth_radius_meters * 2 * pi() / 360) as boundary_lat,
@lv_lng + @search_radius_meters * cos(@lv_lat * 2 * pi() / 360) * sin(nums.id * 2 * pi() / 9)
/ (@earth_radius_meters * 2 * pi() / 360) as boundary_lng
from nums
) t);
explain select count(*) from points where ST_Contains(@boundary_polygon, latlng)\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: points
partitions: NULL
type: range
possible_keys: points_latlng_index
key: points_latlng_index
key_len: 34
ref: NULL
rows: 987
filtered: 100.00
Extra: Using where
1 row in set, 1 warning (0.01 sec)
mysql> select count(*) from points where ST_Contains(@boundary_polygon, latlng)\G
*************************** 1. row ***************************
count(*): 1228
1 row in set (0.01 sec)