Applying programming language research ideas to transform spreadsheets

We’re starting to use Airtable in a pretty sophisticated way. It allows you to organize yourselves and be pretty dynamic as you learn and adapt your process. We can then have our engineers come in once the data model is robust but frankly I see our team using the Airtable UI for a long time.

Telco, HVAC, message switches, various embedded systems, alarm systems and so on.

Crazy stuff indeed. I saw a type of "click-once", auto-update deployment for vba code at an IBank in 2000 that worked wonderfully. Excel models built by trading desks and investment banking groups themselves are often constructed under a lot of time pressure and one thing about Excel is you can do it wrong and it may still work reasonably well. I have seen some specialty Excel software teams in these environments that refactored those models and made them remarkably robust and coherent. Much of it is just having the time to study alternatives: what part of the model should be on the sheet in formulas, what part in vba, what part in another language via an xll? What questions are the users going to ask and what calculations do they need to be able to see and understand? Where does the flexibility need to be? If a user moves a sheet to another workbook, will the button on it still work? How do you deploy vba code so that if you fix a bug all the models that use that vba code, and all their 'save as' descendants, run the fixed code.

There are clever ways to answer these questions but if you condescend to Excel as a development platform, and believe you can't build anything great with it, you may never brainstorm enough to uncover them.

CS indoctrination is most definitely not a failure of education. Mediocre kids thinking they are writing good code because they've written unit tests is.

EDIT: Let me explain my thinking a bit more - we can try to categorize software development into two categories:

1. People write messy, unorganized code that is often in the head of just one or a few folks, who are smart but unorganized and often without formal CS training. The code often has almost no tests and doesn't follow any of the standard best practices of software engineering. The code is often impossible to pass down to new people, often ending up forcing the new people to redo a large fraction of the work.

2. People write clean, modular, testable code with good unit and integration tests, a robust build framework, etc. The code is written in a manner thats super easily transferable, most devs don't even have to understand the entire codebase to start meaningfully contributing.

Obviously, the second category is the preferred category. A good SDE with a CS bg should follow (and often do follow) the second method. However, category 2 could, at least from my experience, be split into two sub-categories:

2A. The framework for both the code and the dev ecosystem was laid down by (often just one) really good engineers who think through what the problem really is and make sure that the fundamental structure and architecture of the codebase works towards solving that real world problem. This kind of code is absolute pleasure to work with and extend.

2B. The framework and the majority of the code are written by average engineers; often the first few eng hires in the company put the groundwork and make poor design choices and the engineering team that follows never wants to change anything fundamental because that's "tech debt" which the company can never afford to take a step back and look at. The average engineers have a good heart, but often their test cases never test real-world edge cases, they often don't even remember the architecture of the code they themselves wrote a few months back, and the code breaks all the time. Furthermore, the engineers would generally balk at adding any new feature because the codebase is fundamentally evolved into something that just cannot be extended without significant rework, and often they cant even see how they can rework it to add the required feature. In the end, good SDE practices and testing doesn't do shit if the person who wrote it didn't think hard enough.

Here again, 2A is the preferred method of doing dev, but unfortunately, the thoughtful smart SDEs aren't that many, and would often be found in a well-paid job in a big company. Most regular devs can't step up to that level and the end result is 2B happens.

Now the question is, which is the better of the two evils, if they are the only choices? 1 or 2B? I'd choose 1. My experience has been that while the scrappy code is unmaintenable by anyone new, at the least the guys who wrote it (assuming you can keep them long enough) will at least own up to it and make sure it keeps running, and they can at least try (and practically, generally succeed) to ship a new feature as opposed to the 2B case where often the categorical answer would be no.

http://steve-yegge.blogspot.com/2008/02/portrait-of-n00b.htm... This point reminds me of "Portait of a Noob". For people who understand what the codebase is doing, it's more effective to have as much code on screen as possible. For "noobs", since you don't know what the code does, less code on screen is better. Additionally, these 'noobs' like more meta-program stuff like type declarations (or unit tests) which don't actually contribute to the program itself, but provide a safety barrier to prevent things from going wrong. (Whereas, if you know what the code is doing and it goes wrong, you can just fix it).

It's fair to say "a program not covered by unit tests can still be good enough to get the job done" (& to suggest that this is under-appreciated), but not having sufficient test coverage significantly raises the barrier to entry for those not familiar with the code ("noobs") from contributing.

(EDIT: I wrote my comment before parent's edit)