A significant amount of programming is done by superstition • Hacker News Board

Re: A significant amount of programming is done by superstition

#181

Post by coldnebo » Thu, Mar 26, 2015, 12:31 PM UTC

nostrademons wrote: post #150

coldnebo wrote: post #135

Earlier quoted context omitted.

This sounds like Engineering, but be aware that in mechanical engineering the requirements are well-defined for materials and load. Consider the famous bridge resonance problem and you will see why wind resonance is now a required subject for structural engineering. ME has an advantage over CSE in this regard because in CS the requirements are usually very poorly stated. About the best and most rigorous requirements…

It's interesting that you mention TCP/IP, because that's the most famous example of a technology that "works in practice, but not in theory". Take a look at Van Jacobsen's presentation on the history of ARPANET: http://www.youtube.com/watch?v=oCZMoY3q2uM Packet switching was "utter heresy" (20m in) when it was invented. It "wasn't a network, it was an inefficient way to use an existing network". And it almost collaps…

Good points. We might be focusing on different evidence for good design. Certainly a system has to be tested -- the theory is not enough. But areas of TCP that were designed up front were things like being able to carry arbitrary payloads, including itself! This is what makes ssh and vpns possible. By contrast, many web protocols break when tunneling: soap within soap.

Also, for those unfamiliar with the actual process of protocol development back in those days, it was a wire protocol, which meant formal modeling and testing. Sure, it doesnt catch everything, but the web is far less formal. For example, the w3c originally said it wasn't going to provide an XML parser reference implementation because any graduate student should be able to code it up in two weeks. WTH?! While I don't doubt that is true, in practice it has meant that dozens of slightly different parsers were written, leading to hundreds of slightly different incompatibilities. Anyone who has had to integrate two different XML stacks will know.

I use Ruby, which was inspired by smalltalk; modern Java is also becoming much more functional.

In some ways, it has taken the larger community 20 years to understand Kay's vision. Also, he always said that the systems he worked on were prototypes -- he's commented before that he fully expected real-world systems to have surpased his long ago.

But now we have Ruby, Node and Rust. Even Java and Spring.io have dramatically reshaped things towards a "smalltalkish" future. So I still put a lot of weight behind some of Kay's observations of the industry.

Re: A significant amount of programming is done by superstition

#182

Post by icebraining » Thu, Mar 26, 2015, 12:42 PM UTC

sklogic wrote: post #173

anon4 wrote: post #14

I'll have to admit something about myself -- I've never been able to learn how to do anything from just reading its first principles. I need to have a working example that I can then adapt and change and observe changes in the result. This has been especially true in programming. I've been writing programs in some capacity for 18 years now and not once in that time have I been able to simply read the reference manual…

It's the ability which is best trained by studying mathematics (even if totally unrelated to programming). Do not listen to those saying it's not possible. We do implement programming languages based on their specifications, we implement protocols by reading RFCs, we implement numeric algorithms by reading pseudocode in the papers, etc. It's all trivial and mechanical . The latter notion is very important, one have t…

We do implement programming languages based on their specifications, we implement protocols by reading RFCs, we implement numeric algorithms by reading pseudocode in the papers, etc.

We implement spherical-cow versions of languages, protocols, etc by reading those papers and specs. Then we spend months or years fixing it to implement the actual languages and protocols.

Re: A significant amount of programming is done by superstition

#183

Post by coldnebo » Thu, Mar 26, 2015, 12:45 PM UTC

coldnebo wrote: post #178

olavk wrote: post #168

Earlier quoted context omitted.

The web might have been created by amateurs, but it works incredibly well. Sure, we have browser incompatibilities and whatnot, but this is an effect of having multiple independent implementations of the standards, which is part of what makes the web work in the first place.

no, it doesn't. Having multiple independent implementations is a PITA, but that's why you have testing & validation labs along with standards. Take windows graphics driver labs... They test for pixel perfect compliance of output across hundreds of vendor implementations. Contrast that to the web where it took a separate group outside the w3c to embarrass browsers with the ACID2&3 tests. Now separate browsers look a l…

Regarding Postscript vs web: I'm only talking about device independence. Specifically in the case of pixel perfect layouts. I am not talking about the layout constraint problem, which is extremely challenging no matter the technology. But layout constraints depend on a solid notion of coordinate system, which the web lacks. Device independence gives you that in postscript and SVG.

Besides, windows faces a similar problem of multiple resolutions and devices. How do they v&v? They set the resolutions the same for certain tests! Even if you do this for browsers, they cant pass the test. Yes, it would be nice if we could have device independent layout constraints as well, but even the simplest most constrained test not involving layouts fails. At least now, its close. Before ACID it wasn't even close.

If webdevs can't even rely on their browser coordinate system in the most heavily constrained case, how can they hope to trust it when they try to solve challenging problems of dynamic layouts across multiple resolutions?

Re: A significant amount of programming is done by superstition

#184

Post by the_why_of_y » Thu, Mar 26, 2015, 1:05 PM UTC

lmm wrote: post #170

This is specific to shell scripting and to the history of nix standardization. No-one follows or cares about the LSB[1]; certification is expensive and based on buggy test programs that practically force you to "code to the test" and not to the specification. And when so many implementations ignore the spec, it would be more superstitious to code to the magical spec and assume it's going to work, rather than writing…

IMHO "alien" isn't much of an issue since LSB packages are allowed to use only a small subset of RPM features; the fancier features aren't portable between different RPM-based distros either.

http://refspecs.linuxfoundation.org/LSB_4.1.0/LSB-Core-gener...

A bigger problem is Debian gratuitously using different SONAME than required by LSB for some libraries - your LSB binary can't use those system libraries if it wants to run on Debian. But at least it looks like they fixed that problem in Wheezy.

Re: A significant amount of programming is done by superstition

#185

Post by sklogic » Thu, Mar 26, 2015, 1:15 PM UTC

icebraining wrote: post #182

sklogic wrote: post #173

Earlier quoted context omitted.

It's the ability which is best trained by studying mathematics (even if totally unrelated to programming). Do not listen to those saying it's not possible. We do implement programming languages based on their specifications, we implement protocols by reading RFCs, we implement numeric algorithms by reading pseudocode in the papers, etc. It's all trivial and mechanical . The latter notion is very important, one have t…

We do implement programming languages based on their specifications, we implement protocols by reading RFCs, we implement numeric algorithms by reading pseudocode in the papers, etc. We implement spherical-cow versions of languages, protocols, etc by reading those papers and specs. Then we spend months or years fixing it to implement the actual languages and protocols.

No. We spend months and years fixing the issues with the specs (especially if they're committee-driven, they tend to be crappy). Yet it's easy to keep your implementation conformant to the specification without any hand weaving and looking at "examples".

Re: A significant amount of programming is done by superstition

#186

Post by chii » Thu, Mar 26, 2015, 1:21 PM UTC

Swizec wrote: post #93

avmich wrote: post #84

Earlier quoted context omitted.

And that applies not only to engineers, but to managers - i.e., task definers. Managers aren't interested to ask engineers to build something which will withstand the test of time unless they are sure that time will be actually needed - and with limited information (and high speed of changes) you have today managers tend to think short term. So the whole civilization works along the principle of dog chasing the rabbi…

It sounds like a good task definer (or engineer) should be able to know when thinking a bit further ahead matters and when it doesn't. For instance, you probably need to take a bit more care with defining your core database structure, than you do with positioning a button 20px to the left or right.

> take a bit more care with defining your core database structure, than you do with positioning a button 20px to the left or right.

not when you are trying to prototype the UX - and don't care about the backend. There's no rule of thumb - it's all very subjective and intuition based. That's where experience comes in, and no amount of book studying will help you.

Re: A significant amount of programming is done by superstition

#187

Post by TheOtherHobbes » Thu, Mar 26, 2015, 1:30 PM UTC

sklogic wrote: post #174

GauntletWizard wrote: post #46

Earlier quoted context omitted.

Almost nobody can. Some people think they can, and charge forward with it, but rarely do they produce something that actually works. Usually, they bang on it and test it and cargo-cult it until it works, then convince themselves that it's from 'first principles'. Even if it were a common ability to produce engineering work from written design principles, I'm not certain that's what we want. Experimentation is the key…

Engineering is not a science, there is very little space for discovery there. In engineering you're applying science, not creating a new scientific knowledge.

Engineering lives on a continuum from almost pure maths to pure cookbook canned solutions, with a much smaller arrow going in the opposite direction from unexplained observations to theorising and model building.

Things like Shannon's description of information-as-entropy were certainly a mathematical discovery about engineered systems, and led directly to a lot of coding and data compression theory.

There wasn't a whole lot of interest in quantum channels at the time, so the fact that the theory fed something back to mainstream physics was a bonus.

Re: A significant amount of programming is done by superstition

#188

Post by engi_nerd » Thu, Mar 26, 2015, 1:31 PM UTC

dandelany wrote: post #20

rjbwork wrote: post #11

I think a better title might be "A vast majority of programming is done at a level of abstraction that this author is slightly uncomfortable with". If you work in Java or Ruby or Python or C#, or even C++ all day with say, an IoC container for argument's sake, you're certainly not constantly thinking "okay this interface will be resolved to this type, which has these dependencies which will be resovled to these types…

I don't think this is about abstraction. Abstraction implies having a deep understanding of the problem domain and encapsulating it in such a way that the implementation details can be ignored and the domain can be reasoned about at a higher level. I think a better title would be "A significant amount of (shipped, so-called production-quality) code is not fully understood by its authors".

I don't think it's about the act of creating an abstraction, but rather, about using these abstractions with a deficient understanding of the limitations and implications of these abstractions.

I agree with you that a significant amount of code is not fully understood by its authors. I see similar phenomena at work in many engineering projects. Relatively few engineers work at the component or board level, rather they work at the subsystem and system levels. They piece parts of systems together to make larger systems-of-systems (sound familiar, programmers?) often with a poor understanding of when the black-box abstractions they deal with can break down and cause havoc.

Currently fighting against this in my own workplace, where it's become clear that many engineers do not understand the system we support in sufficient detail to properly troubleshoot faults with that system.

Re: A significant amount of programming is done by superstition

#189

Post by icebraining » Thu, Mar 26, 2015, 1:42 PM UTC

sklogic wrote: post #185

icebraining wrote: post #182

Earlier quoted context omitted.

We do implement programming languages based on their specifications, we implement protocols by reading RFCs, we implement numeric algorithms by reading pseudocode in the papers, etc. We implement spherical-cow versions of languages, protocols, etc by reading those papers and specs. Then we spend months or years fixing it to implement the actual languages and protocols.

No. We spend months and years fixing the issues with the specs (especially if they're committee-driven, they tend to be crappy). Yet it's easy to keep your implementation conformant to the specification without any hand weaving and looking at "examples".

You can keep it conformant, but that's not enough, because the real world isn't. Try writing a conformant HTML parser that rejects invalid input, and then run a crawler that uses it. You'll reject half the web!

Re: A significant amount of programming is done by superstition

#190

Post by rbanffy » Thu, Mar 26, 2015, 1:47 PM UTC

mreiland wrote: post #38

rbanffy wrote: post #13

Earlier quoted context omitted.

If you do not understand the code, you cannot say you tested it. It may have corner cases you did not anticipate. The test may be incomplete and only cover what you think the program should do.

why write unit tests when we don't understand the microcode in the CPU? We can't have "tested" it because we don't understand said microcode. Or wait... no, maybe your characterization is wrong. Yeah... that's probably it.

Where exactly did I say you should understand the code down to how electrons flow through the chip's gates?

I said if you do not understand what you are testing (not the compiler, not the runtime libraries, not the OS, not the instruction decode, not the microcode, not the gates), you can't say you actually tested it. You merely observed what could be a side-effect.