Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

They don't even know the fundamentals

blog.royalsloth.eu

Post reply on HN
Print view View original

271–280 of 323 posts

Re: They don't even know the fundamentals

#271

Post by kragen »

ACID is the kind of fundamental which can make the difference between easily constructing a reliable system and constructing an unreliable system with great difficulty. It's a mistake to class it with the MVC pattern (now diluted past all recognition) and SOLID just because they're all acronyms and your source of information about ACID was a shitty Wild Hog book by someone who wasn't clear about what it was either.

You can build bridges that don't fall down without knowing the difference between stress and strain; you can build an audio amplifier without knowing the difference between voltage and current; and you can build a database-backed application without knowing what ACID means. But you shouldn't, except maybe as a learning exercise, because it's an enormous amount of effort to produce mediocre results. Understanding ACID allows you to treat a database as "a black box with an API" with enormously better results.

I have no patience for the kind of glorification of ignorance in this post. Perhaps it's true that "the majority of programmers" think "something being RESTful means that you can feed it chunks of JSON over HTTP." The majority of audiophiles might think that oxygen-free copper will improve their sound quality, the majority of computer users might think that Google built the Internet, and the majority of people don't speak English. That has no bearing on the knowledge needed to actually build a working hi-fi system, debug an internet connection problem, or successfully converse in English, and neither does popular ignorance about REST (or ACID) have any bearing on the knowledge needed to successfully design a distributed system that works reliably in the face of constant partial failures.

(Ivanhoe's post at https://news.ycombinator.com/item?id=29065438 is an excellent explanation of what can happen when you lack the understanding of these fundamentals but mistakenly think you can use the database as a black box anyway: "I lost a lot of data when traffic suddenly jumped, and since those were affiliate clicks lots of people got very pissed about not getting paid, and it almost ruined both my client's business as well as my own, as it was our core client.")

If over 40 years you spend 83000 hours working, and you spend 10% of them on programming, that's 8300 hours. (If your job title is "software engineer" you probably spend more like 50% of them on programming, but maybe you're a web designer, a database administrator, a sysadmin, a mechanical engineer, or a biologist, so programming is a relatively minor part of your work.) The optimal fraction of that time spent on getting to know the fundamentals --- real fundamentals like ACID, not fake fundamentals like SOLID, though this presumes that you have someone to guide you who knows the difference --- is surely not 100%, because then you'll never get any work done. But it's not 0% either; you won't get much done that way either. It's probably in the range of 20% to 60%, which would be 1700-5000 hours. Learning in your bones what ACID is might take you 5 hours spread over a month or two, so it only takes up 0.3%-0.1% of that time.

Re: They don't even know the fundamentals

#272

Post by benlivengood »

lmilcin wrote: post #241
ivanhoe wrote: post #43

> It turns out my day to day work doesn’t require deep knowledge about database internals and I can mostly treat them as a black box with an API 2. Because of such attitude of the previous dev team my client ended up with DB integrity ruined. Previous guys somehow didn't know they should use transactions when updating/deleting stuff in the DB, because hey, it's just API you call, who cares of mambo-jumbo happening be…

One of projects I was called to help some time ago set up a service with over a thousand servers. They reached out to help improve the performance because in their opinion the network and network filesystem were too slow. They invested incredible amount of time in learning technology and scaling their application but forgot about need to learn fundamentals -- data structures and efficiency. Fast forward 1,5 years, th…

Something people seem to forget when designing systems based on what they've learned is that modern individual machines have approximately the power of the top supercomputer ~20 years ago, and a few racks of modern machines can in many ways match the top supercomputer of ~10 years ago. Approaches to solve large problems are continually changing and it's almost always worth a big-picture design session when looking at a familiar-seeming problem. By the time students graduate from a 4-year program computing is about 10X better than when they started.

I still remember some interviews within the last decade where single-core machines with a few GB of RAM were a default assumption for whiteboard designs, or that spinning disks were the default.

Re: They don't even know the fundamentals

#273

Post by kragen »

marcosdumay wrote: post #252
Koshkin wrote: post #142

Earlier quoted context omitted.

This doesn’t sound correct to me. The original statement is, Let ϕ(x) be a property provable about objects x of type T. Then ϕ(y) should be true for objects y of type S where S is a subtype of T. Here, objects of a base class are replaced with objects of derived classes.

> This doesn’t sound correct to me. Yet the GP says exactly the same thing as you said. He just ignored that the property belongs to the base class.

The GP said "usable interchangeably", which is vague enough to admit a wide variety of interpretations, including the more rigorous one Koshkin posted.

Re: They don't even know the fundamentals

#274

Post by Kranar »

roenxi wrote: post #146
ZephyrBlu wrote: post #90

Earlier quoted context omitted.

The author says, "I wouldn’t be able to explain from the top of my head what the ACID term means" . I think there is a big difference between not knowing something off the top of your head and being completely unaware of it. If you have awareness of something, you can make decisions based on that awareness without necessarily having a deep understanding. When you have no awareness is when the big issues arise. For ex…

To some degree the more competent the person in using ACID databases, the less likely they are to remember what 4 random words are hidden behind ACID. If someone uses a database that makes ACID guarantees there is no benefit to remembering what they are. It is just how the database works. People only need to know about ACID if they are arguing about whether to use a database that doesn't implement it, or switching be…

I disagree entirely with this, ACID only makes guarantees at the database level itself not the system as a whole; it's possible that using ACID in the database could render the system as a whole more vulnerable than not using it. Take for example MySQL/InnoDB which offers durability (the D in ACID). If you have a write heavy database you could end up in a situation where your server ends up stalling waiting for MySQL to commit the write to disk. In effect, your server ends up building a large buffer of queries to commit to MySQL and sits there waiting for MySQL to become available. If your server or system crashes, all of that data gets lost.

You may have been much better off, both performance and correctness, to relax MySQL's durability from flushing on every write, to flushing/committing every 1 second (using innodb_flush_log_at_trx_commit). Doing so improves performance by multiple orders of magnitude (it can be the order of 10-100x) to the point that your server is no longer sitting there queuing up queries for writes to complete. The database loses some guarantees about durability but the system as a whole gains better durability.

I could make similar arguments about the other properties as well, the point is that a competent person should actually know this stuff, understand it, not just memorize an acronym and boil the entire issue down to whether a database has ACID or not. It's not a binary matter, it's more of a continuum where each property can be tuned to various degrees. MySQL and Postgres both offer many options to tune the various ACID guarantees from having complete conformance to relaxing them with various upper bounds. Relaxing guarantees about a local property of the system can improve the guarantees of system as a whole.

Re: They don't even know the fundamentals

#275

Post by kragen »

addicted wrote: post #139
ivanhoe wrote: post #43

> It turns out my day to day work doesn’t require deep knowledge about database internals and I can mostly treat them as a black box with an API 2. Because of such attitude of the previous dev team my client ended up with DB integrity ruined. Previous guys somehow didn't know they should use transactions when updating/deleting stuff in the DB, because hey, it's just API you call, who cares of mambo-jumbo happening be…

The author’s point still stands. If you’re implementing interactions with a fairly new database (let’s face it. 10 years ago MongoDB was basically experimental) then it is indeed incumbent on you to understand in greater details what you’re doing. But if you’re building a CRUD web app, that’s maybe only used internally by your company and backed by SQL Server you probably have to never give it a thought. Either way,…

> But if you’re building a CRUD web app, that’s maybe only used internally by your company and backed by SQL Server you probably have to never give it a thought.

Until you get complaints that the hour-long reconciliation transaction you run in the morning is locking out all the users, or there's an API a customer hits that's erroring out 1% of the time because Sybcoughuh, Wonderful Microsoft SQL Server has detected a deadlock, and this is causing us to lose customer orders, or...

No, you really need to know what ACID means, and what tradeoffs are made to balance ACID with other considerations in your chosen database implementation. Lumping ACID in with vague heuristics like SOLID on the basis that they're both acronyms is just plain ignorance.

Re: They don't even know the fundamentals

#276

Post by SteadBytes »

dan-robertson wrote: post #145
smcl wrote: post #54

Earlier quoted context omitted.

I think the main thing we want to avoid here is situations like the following: - interviewer asks candidate to determine the complexity of some algorithms - candidate is nervous, has a minor brain-fart and flubs a couple of them - interviewer evaluates candidate as lacking basic CS knowledge you learn in 1st year of any CS course Ok in this case we're talking about big-O, but it could really be anything.

Just say everything is O(2^N) and you’ll be technically correct most of the time. If the interviewer says that it’s wrong, they don’t know their fundamentals.

I had a good chuckle at this, +1 !

Re: They don't even know the fundamentals

#277

Post by CheezeIt »

marcosdumay wrote: post #270
nostrebored wrote: post #73

Earlier quoted context omitted.

Because an implementation of a binary tree is dead simple.

The fact that you didn't even think about asking if it's balanced or not, and just went into declaring stupid everyone that considers the problem hard is annoying. I wouldn't want to have an interview with you, or you selecting people for a team I'm on.

Why would it be balanced. Nobody said it was a search tree.

Re: They don't even know the fundamentals

#278

Post by lmilcin »

benlivengood wrote: post #272
lmilcin wrote: post #241

Earlier quoted context omitted.

One of projects I was called to help some time ago set up a service with over a thousand servers. They reached out to help improve the performance because in their opinion the network and network filesystem were too slow. They invested incredible amount of time in learning technology and scaling their application but forgot about need to learn fundamentals -- data structures and efficiency. Fast forward 1,5 years, th…

Something people seem to forget when designing systems based on what they've learned is that modern individual machines have approximately the power of the top supercomputer ~20 years ago, and a few racks of modern machines can in many ways match the top supercomputer of ~10 years ago. Approaches to solve large problems are continually changing and it's almost always worth a big-picture design session when looking at…

Well, I catch myself that I am not "updating" the state of my understanding of hardware. Just recently I caught myself putting a lot of effort into solving a problem that only exists if persistent storage is too slow to be used for calculations. Then I facepalmed myself hard when I realized that I can just move that entire 500GB data structure to an NVMe and treat it almost as if it was in memory.

But in general I think that the problem isn't that people are not "updating" their understanding. Even 10 years ago it wasn't a huge problem getting hundreds of thousands of transactions per second on a single modest machine.

The problem rather is people relying on more and more layers of abstractions for vary small gains.

Example: I get that Python is a nice language (for somebody that does not know Lisp). But is it worth it to choose Python for a little bit improvement in productivity for a problem that requires a lot of throughput, to then suffer performance issues, to then spend many times more effort on trying to improve performance? I don't think so.

Or more in my space: is Spring Data (Java) worth the very incremental productivity improvements if it completely destroys your application performance? The application I described in my parent post used Spring Data MongoDB which kinda means it was fetching entities one by one which is extremely costly.

By replacing it with bare MongoDB reactive driver and ensuring data is being streamed in large batches (why read one user data if you can read 10k at a time) and getting rid of costly aggregations in favour of application side processing we have improved throughput by many orders of magnitude WHILE reducing load on MongoDB.

Granted, there is a little bit of additional complexity (on the order of 10% more of application code) but just the performance improvements mean that the team can breathe and focus on other problems like modelling the domain correctly.

Re: They don't even know the fundamentals

#279

Post by kragen »

tchalla wrote: post #154
smallerfish wrote: post #135

Earlier quoted context omitted.

Well hopefully there's some layering in the question if it matters to the job. If you have an engineering position in which a high degree of CS competence is required and the candidate can't even make it past the initial layer of "regurgitating binary tree operations" then you have the wrong candidate.

> If you have an engineering position in which a high degree of CS competence is required and the candidate can't even make it past the initial layer of "regurgitating binary tree operations" then you have the wrong candidate. Or you have a candidate who understands binary trees, can implement it but can't regurgitate binary tree operations from the top of their memory.

You mean, "regurgitate" operations like this?

    find = lambda k, t: t if t is None or k == t.k else find(k, t.left) if k
I mean, figuring out how to rebalance a red-black tree after insertion is more demanding, and you could legitimately quibble with my golfing here, but I feel like the reasoning required to write that function (given the tree definition) is just not that advanced. It's like the FizzBuzz of pointers.

Re: They don't even know the fundamentals

#280

Post by jhgb »

ascar wrote: post #235
jhgb wrote: post #195

Earlier quoted context omitted.

Correct me if I'm wrong, but isn't this exactly the case where you don't want an unbalanced tree? A degenerate tree is bad enough; a degenerate tree implemented as an array in this way is catastrophic. So it seems to me that if you're refining the topic further in some sort of talk, the topic of balancing should precede the topic of physically representing the tree as an array.

It all depends on the specific topic. If I just want to store my data in binary tree structure without the possibility of deletes and just with an insertion-order requirement, then the tree is automatically balanced and using an array like that is totally fine. The use-cases are probably rare (I can't think of a situation from the top of my head), but binary tree doesn't have to mean search tree. The self-balancing p…

> The use-cases are probably rare (I can't think of a situation from the top of my head)

I imagine there's quite a lot of them. Things like VP-trees and such get often built statically and could be implemented in a similar way. Not sure if the simple (2i, 2i+1) schema is optimal for this but perhaps it could be adapted.

Post reply on HN