A significant amount of programming is done by superstition • Hacker News Board

Re: A significant amount of programming is done by superstition

#101

Post by ObviousScience » Thu, Mar 26, 2015, 12:20 AM UTC

GauntletWizard wrote: post #46

anon4 wrote: post #14

I'll have to admit something about myself -- I've never been able to learn how to do anything from just reading its first principles. I need to have a working example that I can then adapt and change and observe changes in the result. This has been especially true in programming. I've been writing programs in some capacity for 18 years now and not once in that time have I been able to simply read the reference manual…

Almost nobody can. Some people think they can, and charge forward with it, but rarely do they produce something that actually works. Usually, they bang on it and test it and cargo-cult it until it works, then convince themselves that it's from 'first principles'. Even if it were a common ability to produce engineering work from written design principles, I'm not certain that's what we want. Experimentation is the key…

Any time I have to develop something from first principles, I do it 2-3 times in different ways, and only then make a serious attempt at actually writing it. It's still terrible, of course, but at least has a chance of working since I have a bit of experience in what can go wrong.

Re: A significant amount of programming is done by superstition

#102

Post by forgueam » Thu, Mar 26, 2015, 12:21 AM UTC

santosha wrote: post #41

I've always enjoyed what I call uncoding, which is taking a working piece of code and taking stuff away until things break, and examining how they break. It's really helped me understand what each piece does, and be more efficient when I'm writing it myself.

I love that term. You described exactly how I work: take an existing block of code, break it apart until I fully understand every line, then reassemble to meet my needs.

Re: A significant amount of programming is done by superstition

#103

Post by danbruc » Thu, Mar 26, 2015, 12:22 AM UTC

svisser wrote: post #99

danbruc wrote: post #95

This does not resonate with me at all. I can not stand having things in my code I don't fully understand. Without reading the source code of libraries, without reading relevant standards, without understanding the mathematics behind algorithms, there is always this feeling I may make really dumb mistakes. Really understanding all the bits and pieces obviously does not make your code bug free, but I am usually pretty…

But what does it mean to fully understand your own code? Everyone relies on millions of lines of code that they didn't write themselves.

For me that really depends on what I am dealing with. Most of the time the library source code is good enough. Sometimes - I am a C# developer - I will have to have a look at the .NET runtime specification. Sometimes I will have to learn details of Windows. In some cases I will even look into Intel's reference manuals to understand what is happening. I would say I stop at the level where I don't expect the layer below to be relevant to what I am actually dealing with.

Re: A significant amount of programming is done by superstition

#104

Post by perlgeek » Thu, Mar 26, 2015, 12:24 AM UTC

sysv init scripts in particular are so pointless to write.

In the end, the whole script usually revolves around four lines of actual code (starting a process, stopping it, querying the status, optionally sending it a SIGHUP to make it reload its configuration), but the result is a script with 50 lines of boilerplate.

Which, of course, could nearly all be automated away by a sane, declarative format.

Leaving all the political mess aside, that's the part that systemd got right. You write sometthing like

    [Unit]
    Description=some human-readable text here

    [Service]
    Type=simple
    User=...
    ExecStart=/path/to/deamon --with=options
    KillSignal=SIGKILL

    [Install]
    WantedBy=multi-user.target

and that's it.

Quick comparison of sysv init script lengths on Debian Wheezy vs. systemd service files on Debian Jessie: ssh: 162 vs. 15 lines, cron: 92 vs 11 lines, dbus: 122 vs 9 lines

My point being: it's a bit too much to demand that people carefully engineer their sysv init files, when about 90% of those very same init files is simply boring, repetitive boiler plate code. Also, what happened to the "don't repeat yourself" principle?

I tried writing a sysv init script from scratch. I stopped because of sheer boredom, and resorted to copy&paste to save both time and nerves.

Re: A significant amount of programming is done by superstition

#105

Post by ackalker » Thu, Mar 26, 2015, 12:26 AM UTC

danbruc wrote: post #95

This does not resonate with me at all. I can not stand having things in my code I don't fully understand. Without reading the source code of libraries, without reading relevant standards, without understanding the mathematics behind algorithms, there is always this feeling I may make really dumb mistakes. Really understanding all the bits and pieces obviously does not make your code bug free, but I am usually pretty…

This piques my interest. When was the last time anyone read the source code of (g)libc "cover to cover" before using it? Does anyone know the precise implications of (not) defining any of the myriad sensible permutations of its "feature test macros"? Is anyone even sure that those implications are actually implemented correctly before relying on them?

Re: A significant amount of programming is done by superstition

#106

Post by krick » Thu, Mar 26, 2015, 12:42 AM UTC

That's right, but I guess we're missing the real problem here.

The real gold are these words:

> You've really read the LSB specification for init scripts and your distribution's distro-specific documentation? If so, you're almost certainly a Debian Developer or the equivalent specialist for other distributions.

Many of us would agree, that it's true. But why is it true? Shouldn't that make us suspicious in the first place? It's almost paradoxical, even funny: making the work done is easier, than finding out, how to do that work. You see, init scripts or makefiles aren't exactly rocket science. I mean, things we are willing to achieve with them are pretty simple: just start some process, just build some project (which means "execute bunch of shell commands in the right order", and not implementing out own superoptimizing compiler).

So when we arrive to situation like one we've got today, the right message wouldn't be "come on guys, you see what we're doing because of being lazy? we should stop being so lazy!". That is, yeah, we all are lazy and maybe we should stop it, but that's not the point. What this situation actually means is that our tools for doing simple stuff are overcomplicated shit, and our documentation for these tools is shit. If anything, it's pretty obvious that a couple of lines of code, showing how to do the work is better documentation than complete list of possible flags and parameters with no examples whatsoever. So no wonder people are reading stackoverflow instead of man-pages. And so on.

Re: A significant amount of programming is done by superstition

#107

Post by dasil003 » Thu, Mar 26, 2015, 12:47 AM UTC

GauntletWizard wrote: post #89

avmich wrote: post #81

Earlier quoted context omitted.

> Experimentation is the key to science. Computer science is a branch of mathematics. Here you build models, reason about them, find properties, then prove them, then rely on them. Tests - a.k.a. experiments - are usually not exhaustive and don't provide guarantees of correctness. Having said that, I agree that experiment is of crucial importance - how else you would validate your models in the first place? However,…

Programming is a branch of engineering. Very little programming is done by proof; It's done by experimentation, debugger, and tests are human-reasoned and hopefully cover all codepaths and critical junctures (for every x > 0, you test x=1, x=0, x=-1). Proof is of critical importance - It's how you find those junctures, for example, how you optimize, how you design and how you reason about runtime, etc. But programmin…

Reminds me of the famous Knuth quote:

> Beware of bugs in the above code; I have only proved it correct, not tried it.

Deep thinking and mathematical proofs are tremendously useful when thinking about a well-defined and scoped problem at a more or less fixed level of abstraction.

The problem with software engineering is that in practice all abstractions are leaky, and so you inevitably find yourself dealing with issues that are too far-flung and random to be mathematically tractable. The best you can do is be rigorous about the core problem you are solving, but there is always some amount of hammering and duct tape to build it into a non-trivial real-world system. It's possible to attack this problem asymptotically with a rigorous engineering process such as NASA employs, but we don't because it's simply not cost effective for the majority of software.

Re: A significant amount of programming is done by superstition

#108

Post by ackalker » Thu, Mar 26, 2015, 12:48 AM UTC

mreiland wrote: post #38

rbanffy wrote: post #13

Earlier quoted context omitted.

If you do not understand the code, you cannot say you tested it. It may have corner cases you did not anticipate. The test may be incomplete and only cover what you think the program should do.

why write unit tests when we don't understand the microcode in the CPU? We can't have "tested" it because we don't understand said microcode. Or wait... no, maybe your characterization is wrong. Yeah... that's probably it.

That's why they're called unit tests. They are supposed to test precisely one unit of functionality, while assuming everything else works as specified (using mocks if necessary). Taking into account the effects of microcode execution on the test result is in fact a kind of integration testing.

Re: A significant amount of programming is done by superstition

#109

Post by danbruc » Thu, Mar 26, 2015, 12:55 AM UTC

ackalker wrote: post #105

danbruc wrote: post #95

This does not resonate with me at all. I can not stand having things in my code I don't fully understand. Without reading the source code of libraries, without reading relevant standards, without understanding the mathematics behind algorithms, there is always this feeling I may make really dumb mistakes. Really understanding all the bits and pieces obviously does not make your code bug free, but I am usually pretty…

This piques my interest. When was the last time anyone read the source code of (g)libc "cover to cover" before using it? Does anyone know the precise implications of (not) defining any of the myriad sensible permutations of its "feature test macros"? Is anyone even sure that those implications are actually implemented correctly before relying on them?

I obviously do not and can not trace the entire code path from a print statement through the class library, the operating system, the display driver and the hardware, some things I will still take for granted. But if there is a really critical aspect to what I am doing, say some thread synchronization, I will definitely not just follow some sample code, but read the relevant parts of the memory model of the language specification detailing why this works as expected. If the underlying hardware matters, I will grab the reference manual for it, too.

A lot of this is surly curiosity and you can get away without it most of the time, but at least as soon as I run into the first problem it really helps me a lot to have at least a basic understanding of what is going on in the layer of abstraction I am interacting with and maybe a layer or two below. And it gives me a really comforting feeling, that I know what is going on better than describing it as some black magic.

Re: A significant amount of programming is done by superstition

#110

Post by GauntletWizard » Thu, Mar 26, 2015, 12:59 AM UTC

mpdehaan2 wrote: post #85

A very sigifigant part of Ansible service module code was done to deal with misbehaving init scripts. They are very much cargo-culted just like RPM or debian package formats - borrow the ones from your previous project. They are hard to get right and many apps don't daemonize correctly or return OK before they are ready to be running - and then are actually running much later. Upstart/systemd didn't neccessarily make…

I dislike tools that claim to automatically generate your configuration for you. At Google, one of the worst abuses that anyone had to deal with was auto-generated configuration; a tool purported to deal with configuration for you, but really just blutrted a boilerplate mess into your directory, many of the properties and setting set by cargo-cult or overriding sane defaults with baked-in versions of the same. Facebook is proceeding down this path; There's a new tool that will do all of your service stuff for you! I look forward to what a boondoggle those services that use it will be.

There was a discussion I recently stumbled upon about init systems in containers. Many processes spawned lots of zombies! You need a full init! Well, no; If your process is forking new processes, it should probably wait for them. If it's forking other processes that fork other processes, and they're not cleaning up zombies when they're done, that is a bug in those processes. If all else fails, your root-level process should probably implement a simple init on it's own (a wait() on loop in the background, dumping wait objects into a LRU map that other processes can do an equivalent of waitpid() on). Or just run under bash.