Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The compiler is your best friend

blog.daniel-beskin.com

Post reply on HN
Print view View original

111–120 of 149 posts

Re: The compiler is your best friend

#111

Post by mkleczek »

bambax wrote: post #108
supermdguy wrote: post #46

> A common pattern would be to separate pure business logic from data fetching/writing. So instead of intertwining database calls with computation, you split into three separate phases: fetch, compute, store (a tiny ETL). First fetch all the data you need from a database, then you pass it to a (pure) function that produces some output, then pass the output of the pure function to a store procedure. Does anyone have a…

What I'm currently doing could be called compute-fetch-store: the compute part is done entirely in the database with SQL views stacked one on top of the other. Then the program just fetches the result of the last view and stores it where it needs to be stored. Stacked views are sometimes considered an anti-pattern, but I really like them because they're purely functional, have no side-effects whatsoever and cannot br…

This.

In-RDBMS computation specified in declarative language with generic, protocol/technology specific adapters handling communication with external systems.

Treating RDBMS as a computing platform (and not merely as dumb data storage) makes systems simple and robust. Model your input as base relations (normalized to 5NF) and output as views.

Incremental computing engines such as https://github.com/feldera/feldera go even further with base relations not being persistent/stored.

Re: The compiler is your best friend

#112

Post by einpoklum »

doug_durham wrote: post #29

Typing is great, presuming that the developer did a thorough job of defining their type system. If they get the model wrong, or it is incomplete then you aren't really gaining much out of a strictly typed language. Every change is a fight. You are likely to hack the model to make the code compile. There is a reason that Rust is most successful at low level code. This is where the models are concrete and simple to cre…

Well, you could argue whether or not coding is math or not, but coding is _certainly_ religion. Complete with wars among sects, inquisition, excommunication, priesthood and crusades.

Re: The compiler is your best friend

#113

Post by shakna »

swiftcoder wrote: post #110
cozzyd wrote: post #86

Earlier quoted context omitted.

Until you have a bit flip or a silicon error. Or someone changed the floating point rounding mode.

> Until you have a bit flip These are vanishingly unlikely if you mostly target consumer/server hardware. People who code for environments like satellites, or nuclear facilities, have to worry about it, sure, but it's not a realistic issue for the rest of us

Bitflips are waaay more common than you think they are. [0]

> A 2011 Black Hat paper detailed an analysis where eight legitimate domains were targeted with thirty one bitsquat domains. Over the course of about seven months, 52,317 requests were made to the bitsquat domains.

[0] https://en.wikipedia.org/wiki/Bitsquatting

Re: The compiler is your best friend

#114

Post by rmunn »

josephg wrote: post #48
thatoneengineer wrote: post #38

Earlier quoted context omitted.

Ideally, if you can convince yourself something cannot happen, you can also convince the compiler, and get rid of the branch entirely by expressing the predicate as part of the type (or a function on the type, etc.) Language support for that varies. Rust is great, but not perfect. Typescript is surprisingly good in many cases. Enums and algebraic type systems are your friend. It'll never be 100% but it sure helps fil…

Yes, this has been my experience too! Another tool in the toolbox is property / fuzz testing. Especially for data structures, and anything that looks like a state machine. My typical setup is this: 1. Make a list of invariants. (Eg if Foo is set, bar + zot must be less than 10) 2. Make a check() function which validates all the invariants you can think of. It’s ok if this function is slow. 3. Make a function which ta…

This is indeed a great technique. The only way it could be improved is to expand on step 3 by keeping a list of the random mutation functions called and the order in which they were called, then if the test passes you throw that list away and generate a new list with the next seed. But if the test fails then you go through the following procedure to "shrink" the list of mutations down to a minimal (or nearly minimal) repro:

1. Drop the first item in the list of mutations and re-run the test. 2. If the test still fails and the list of mutations is not empty, goto step 1. 3. If the test passes when you dropped the first item in the mutation list, then that was a key part of the minimal repro. Add it to a list of "required for repro" items, then repeat this whole process with the second (and subsequent) items on the list.

In other words, go through that list of random mutations and, one at a time, check whether that particular mutation is part of the scenario that makes the test fail. This is not guaranteed to reach the smallest possible minimal repro, but it's very likely to reach a smallish repro. Then in addition to printing the failing seed number (which can be used to reproduce the failure by going through that shrinking process again), you can print the final, shrunk list of mutations needed to cause the failure.

Printing the list of mutations is useful because then it's pretty simple (most of the time) to turn that into a non-RNG test case. Which is useful to keep around as a regression test, to make sure that the bug you're about to fix stays fixed in the future.

Re: The compiler is your best friend

#115

Post by bambax »

mkleczek wrote: post #111
bambax wrote: post #108

Earlier quoted context omitted.

What I'm currently doing could be called compute-fetch-store: the compute part is done entirely in the database with SQL views stacked one on top of the other. Then the program just fetches the result of the last view and stores it where it needs to be stored. Stacked views are sometimes considered an anti-pattern, but I really like them because they're purely functional, have no side-effects whatsoever and cannot br…

This. In-RDBMS computation specified in declarative language with generic, protocol/technology specific adapters handling communication with external systems. Treating RDBMS as a computing platform (and not merely as dumb data storage) makes systems simple and robust. Model your input as base relations (normalized to 5NF) and output as views. Incremental computing engines such as https://github.com/feldera/feldera go…

Ha! I don't yet know much about 'incremental computing engines' but Feldera seem to be something I need. Because at some point I inevitably have to create materialized views to speed up some parts of the pipeline. Materialized views are of course a side effect and can become mildly dangerous if you're not careful to destroy/recreate them in time.

I was trying to think of a way to "only update new or changed rows" but it's not trivial. But Feldera seems to do exactly that. So thanks!

Re: The compiler is your best friend

#116

Post by Lvl999Noob »

addaon wrote: post #97
josephg wrote: post #93

Earlier quoted context omitted.

There is no recovery. When an invariant is violated, the system is in a corrupted state. Usually the only sensible thing to do is crash. If there's a known bug in a program, you can try and write recovery code to work around it. But its almost always better to just fix the bug. Small, simple, correct programs are better than large, complex, buggy programs.

> Usually the only sensible thing to do is crash. Correct. But how are you testing that you successfully crash in this case, instead of corrupting on-disk data stores or propagating bad data? That needs a test.

You don't. Assertions are assumptions. You don't explicitly write recovery paths for individual assumptions being wrong. Even if you wanted to, you probably wouldn't have a sensible recovery in the general case (what will you do when the enum that had 3 options suddenly comes in with a value 1000?).

I don't think any C programmer (where assert() is just debug_assert!() and there is no assert!()) is writing code like:

    assert(arr_len > 5);
    if (arr_len
They just assume that the assertion holds and hope that some thing would crash later and provide info for debugging if it didn't.

Re: The compiler is your best friend

#117

Post by MobiusHorizons »

lmm wrote: post #94
LegionMammal978 wrote: post #15

Earlier quoted context omitted.

Do you not make such a tacit assumption every time you index into an array (which in almost all languages throws an exception on bounds failure)? You always have to make assumptions that things stay consistent from one statement to the next, at least locally. Unless you use formal verification, but hardly anyone has the time and resources for that.

> Do you not make such a tacit assumption every time you index into an array (which in almost all languages throws an exception on bounds failure)? Yes, which is one reason why decent code generally avoids doing that.

Are you saying decent code avoids indexing into arrays? Or are you saying it avoids doing so without certainty the bounds checks will succeed?

Re: The compiler is your best friend

#118

Post by swiftcoder »

shakna wrote: post #113
swiftcoder wrote: post #110

Earlier quoted context omitted.

> Until you have a bit flip These are vanishingly unlikely if you mostly target consumer/server hardware. People who code for environments like satellites, or nuclear facilities, have to worry about it, sure, but it's not a realistic issue for the rest of us

Bitflips are waaay more common than you think they are. [0] > A 2011 Black Hat paper detailed an analysis where eight legitimate domains were targeted with thirty one bitsquat domains. Over the course of about seven months, 52,317 requests were made to the bitsquat domains. [0] https://en.wikipedia.org/wiki/Bitsquatting

> Bitflips are waaay more common than you think they are... Over the course of about seven months, 52,317 requests...

Your data does not show them to be common - less than 1 in 100,000 computing devices seeing an issue during a 7 month test qualifies as "rare" in my book (and in fact the vast majority of those events seem to come from a small number of server failures).

And we know from Google's datacenter research[0] that bit flips are highly correlated hard failures (i.e. they tend to result from a faulty DRAM module, and so affect a small number of machines repeatedly).

It's hard to pin down numbers for soft failures, but it seems to be somewhere in the realm of 100 events/gigabyte/year - and that's before any of the many ECC mechanisms do their thing. In practical sense, no consumer software worries about bit flips in RAM (whereas bit flips in storage are much more likely, hence checksumming DB rows, etc).

[0]: https://static.googleusercontent.com/media/research.google.c...

Re: The compiler is your best friend

#119

Post by voidUpdate »

joeriddles wrote: post #49

Great read. C# has the concept of nullable reference types[1] which requires you to be explicit if a variable can be null and the compiler is aware of this. I would love to see a similar feature in languages like TypeScript and Go. [1]: https://learn.microsoft.com/en-us/dotnet/csharp/nullable-ref...

Oh wow, I didn't know this was a thing. I knew about nullable value types, and have started to use them a bit, but this looks like it could be very useful

Re: The compiler is your best friend

#120

Post by saagarjha »

svantana wrote: post #56
LegionMammal978 wrote: post #2

> How many times did you leave a comment on some branch of code stating "this CANNOT happen" and thrown an exception? Did you ever find yourself surprised when eventually it did happen? I know I did, since then I at least add some logs even if I think I'm sure that it really cannot happen. I'm not sure what the author expects the program to do when there's an internal logic error that has no known cause and no defini…

> At some level, the simplest thing to do is to give up and crash if things are no longer sane. The problem with this attitude (that many of my co-workers espouse) is that it can have serious consequences for both the user and your business. - The user may have unsaved data - Your software may gain a reputation of being crash-prone If a valid alternative is to halt normal operations and present an alert box to the us…

So you don't get a crash log? No, thanks.
Post reply on HN