Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible schema from a clueless programmer

rachelbythebay.com

Post reply on HN
Print view View original

481–490 of 493 posts

Re: A terrible schema from a clueless programmer

#481

Post by tyingq »

rkuykendall-com wrote: post #456
tyingq wrote: post #15

Feels like you could just concatenate and hash the 4 values with MD5 and store the hash and time. Edit: I guess concatenate with a delimiter if you're worried about false positives with the concat. But it does read like a cache of "I've seen this before" . Doing it this way would be compact and indexed well. MD5 was fast in 2002, and you could just use a CRC instead if it weren't. I suppose you lose some operational…

I'm not super familiar with this stuff, but I believe you could then use a key-value store with automatic expiry like Redis for automatic pruning and faster lookups.

Redis and memcached didn't exist in that timeframe.

Re: A terrible schema from a clueless programmer

#482

Post by onionjake »

GuB-42 wrote: post #172

The problem, I think is that most people, me included, don't really know what databases really do. There is a whole lot about optimizing procedural code, with a variety of tools, the dangers of premature optimization and the tradeoff with readability. Anyone with a computer science degree has heard about algorithmic complexity, caches, etc... But databases are just magic. You try things out, usually involving a CREAT…

Database indexes are definitely one of the most approachable concepts of databases for computer science majors. If you have studied data structures and Big O notation you have all the concepts needed.

While I agree that how a query planner works is one of the most ‘magic’ aspects, I think the output from the query planner in most databases is very approachable as well to regular common programmers and will get you quite far in solving performance issues. We know what full scans are (searching through every element of an array), etc.

The challenge is usually discovering that the jargon used in your database really maps to something you do already have a concept about and then reading the database documentation…

Re: A terrible schema from a clueless programmer

#483

Post by ncmncm »

thaumasiotes wrote: post #403
ncmncm wrote: post #388

Earlier quoted context omitted.

You need to distinguish authentication from authorization. "auth9n" and "auth8n" might be a slight bit too obsure for the purpose.

What a completely nonresponsive "response". To the best of my knowledge, no one has ever even tried to use auth9n or auth8n. But that's not where the 'n' in authn comes from. authn and authk are both equally distinct from authz.

You appear to be unfamiliar with the extremely common, and problematic, abbreviation "i18n" for "internationalization".

For my part, I have not encountered "authn", "authz", or "authk" before. But prior discussion did not even mention the word "authorization", and that seemed worth bringing out in the open.

So, not nonresponsive, just responsive to things you weren't personally interested in.

Re: A terrible schema from a clueless programmer

#484

Post by goto11 »

ufmace wrote: post #470
goto11 wrote: post #433

Earlier quoted context omitted.

I was wondering about it in the context of "yet, other languages would do the right thing by default". Repeatedly concatenating to the same string (as opposed to concatenating an array of strings in one go) would be slow in any language I know of, unless you allocate a larger buffer up front, which is what StringBuilder does. Some languages have mutable strings, but you would still need to allocate a sufficiently lar…

I don't think StringBuilder is faster specifically because it allocates a large buffer. Pretty much every language with growable data structures can already grow any such structure pretty fast, including arrays, maps, and mutable strings. They already have a bunch of pre-set constants about stuff like how big to allocate for the initial empty one and how much more to allocate every time you overflow to balance speed…

The cost is not the allocation per se, the cost is copying the bytes. When concatenating two strings, the bytes from both strings are copied into the new string. If you repeatedly concatenate, the bytes end up getting copied many times. E.g if you concatenate 100 strings by appending one by one, the bytes in the first string is copied 99 times, the bytes in the second string is copied 98 times and so on.

Using a StringBuilder, the strings are only copied once when copied into the buffer. If the buffer need to grow, the whole buffer is copied into a larger buffer, but if the buffer grows by say doubling the size, then this cost is amortized, so each string on average is still only copied twice at worst.

Faster yet is concatenating an array of strings in a single concat operation. This avoids the buffer resize issue, since the necessary buffer size is known up front. But this leads to the subtle issue where a single concat is faster than using a StringBuilder, while the cost for multiple appends grows exponentially for concat but only lineary for StringBuilder.

Re: A terrible schema from a clueless programmer

#485

Post by goto11 »

FreezerburnV wrote: post #458
goto11 wrote: post #433

Earlier quoted context omitted.

I was wondering about it in the context of "yet, other languages would do the right thing by default". Repeatedly concatenating to the same string (as opposed to concatenating an array of strings in one go) would be slow in any language I know of, unless you allocate a larger buffer up front, which is what StringBuilder does. Some languages have mutable strings, but you would still need to allocate a sufficiently lar…

Repeatedly concatting a string is the fastest way I am aware of to build a longer one in Javascript. (This has been deliberately optimized for) I believe PHP this might also be the case for, or at least very fast. Perl might be pretty fast at this but I could be wrong.

I believe joining an array of strings is still faster in JavaScript, since the combined size can be known up front.

Re: A terrible schema from a clueless programmer

#486

Post by zeroimpl »

ComputerGuru wrote: post #417
zeroimpl wrote: post #283

Earlier quoted context omitted.

Yeah I’ve run into this a few times. In Postgres, you can’t add a foreign key referencing columns that aren’t indexed. I assume this is for performance reasons so inserts/updates are efficient, but there is no such check that the source columns are also indexed, so updates/deletes on the referenced table can be terribly slow.

Isn’t the source almost always an indexed primary key?

No. The target is normally a primary key but the source is not. The source is rarely even unique.

Re: A terrible schema from a clueless programmer

#487

Post by path411 »

Aeolun wrote: post #360
path411 wrote: post #276

Earlier quoted context omitted.

Can't imagine the pain to even discover this problem. It normally takes me a long time to be like, "maybe I didn't make a mistake, and I have a third party bug?". Third party bugs are always the most annoying to me, usually the longest to diagnose and then ultimately I have to just tell my boss I can't do anything to fix it, but hopefully I can find a way to avoid it.

Fork and fix? I’ve personally been pleasantly surprised by how fast maintainers will merge a PR I submit. If it causes an issue with our own systems it also gives me a very good justification for doing it.

Sometimes I've had a pr merged. Sometimes I've had stubborn maintainers that think that their project should not follow common domain standards for /shrug reasons.

And then many other times I honestly don't have the knowledge to be able to contribute into large third party systems and it seems very rare to get an issue resolved without submitting a pr.

And lastly sometimes it's hard to judge if unexpected behavior is really a 'bug'. Similar to the last point, it can honestly be hard to tell the difference sometimes.

Re: A terrible schema from a clueless programmer

#488

Post by hoppla »

mewpmewp2 wrote: post #336
hoppla wrote: post #117

Earlier quoted context omitted.

I usually store IPv4 as unsigned int, and IPv6 as two 64 bit unsigned ints. But supporting both variants require that I have an a table for each type and a glue table to reference the record in the correct table. Storing simple stuff quickly turns complicated…

Is there much point in optimising this as opposed to keeping it in strings, at what QPS do you think it would matter and how much?

The main reason to store it as numbers is because I select records based on what subnet an IP address is in. I can only do this when I use integers.

Re: A terrible schema from a clueless programmer

#489

Post by lern_too_spel »

nlitened wrote: post #306
lern_too_spel wrote: post #292

Earlier quoted context omitted.

If the amount charged isn't proportional to the work done, the cloud provider would quickly go out of business.

My VPS provider gives me unlimited traffic, unlimited disk reads, and unlimited DB rows touched for fixed 30 dollars a month, regardless of how much CPU and RAM I keep loaded. This makes me think that these things are at least 100x cheaper than AWS might want to make me believe.

On a single instance. The more instances you use, the more you will be charged. This service handles automatic scalability and resilience with multiple instances for you, allowing it to charge less than $30.

Re: A terrible schema from a clueless programmer

#490

Post by lbill »

It's quite hard to create something that works optimally on a first try. I think I never managed to do so! The first schema was slow, but hey, it worked, good job! That is a huge first step. Modifying it so it runs much faster can definitely come next.
Post reply on HN