I was wrong about spreadsheets (2017) • Hacker News Board

Re: I was wrong about spreadsheets (2017)

#361

Post by kungfugz » Sat, Jul 13, 2019, 12:41 AM UTC

zelphirkalt wrote: post #351

behringer wrote: post #295

Earlier quoted context omitted.

What's your solution? That the manager who wanted some report comes to you and begs you to do it and in 6 months it's finally delivered and all wrong?

The manager goes to the people, who really know how to make a report (maybe that is "accounting"?) and asks them to make the report taking into account some single report specific things that they talk about. Then those people, who know how to do it deliver. They might have someone in their team, who knows how to work with dataframes, which would come in handy. Those people store the program written to generate the r…

I guess no one has used Power BI for reporting... Link the data source, save the view that "Manager A" wants every week or month, setup on premesis gateway, and your new reports are there before your are asked for them again. Always up to date. Then share your dashboard. Also, you guys should be ashamed of yourselves. "not sure who does the reports, maybe finance?" Eceryone should know how to make their own reports.

Re: I was wrong about spreadsheets (2017)

#362

Post by DonHopkins » Sat, Jul 13, 2019, 12:52 AM UTC

anextomp wrote: post #206

jacquesm wrote: post #200

The spreadsheet is the original 'serverless' programming environment. It allows relatively untrained people to augment their brains to levels that they would otherwise definitely not achieve. This has obvious limitations but I still believe that the spreadsheet is the 'killer app' for all of computing, back when VisiCalc was first released the sales of small computers went up in an incredible way.

Serverless is really the wrong term to use here. That refers to a specific way of handling cloud infrastructure (the point being that there are still servers, you just don't care) Excel is used for client-side only programs. There are no servers at all. All programs originally were client-side only originally because networks didn't exist.

Google sheets (and other google docs) can be programmed in "serverless" JavaScript that runs in the cloud somewhere. It's hellishly slow making sheets API calls, though. Feels like some kind of remote procedure call. (Slower than driving Excel via OLE Automation even, and that's saying something!) Then it times out on a wall clock (not cpu time) limit, and breaks if you take too long.

A CS grad student friend of mine was in a programming language class, and the instructor was lecturing about visual programming languages, and claimed that there weren't any widely used visual programming languages. (This was in the late 80's, but some people are still under the same impression.)

He raised his hand and pointed out that spreadsheets qualified as visual programming languages, and were pretty darn common.

They're quite visual and popular because of their 2D spatial nature, relative and absolute 2D addressing modes, declarative functions and constraints, visual presentation of live directly manipulatable data, fonts, text attributes, background and foreground colors, lines, patterns, etc. Some even support procedural scripting languages whose statements are written in columns of cells.

Re: I was wrong about spreadsheets (2017)

#363

Post by DonHopkins » Sat, Jul 13, 2019, 1:35 AM UTC

The thing that's missing from "Google Docs" is a decent collaborative outliner called "Google Trees", that does to "NLS" and "Frontier" what "Google Sheets" did to "VisiCalc" and "Excel".

And I don't mean "Google Wave", I mean a truly collaborative extensible visually programmable spreadsheet-like outliner with expressions, constraints, absolute and relative xpath-like addressing, and scripting like Google Sheets, but with a tree instead of a grid. That eats drinks scripts and shits JSON and XML or any other structured data.

Of course you should be able to link and embed outlines in spreadsheets, and spreadsheets in outlines, but "Google Maps" should also be invited to the party (along with its plus-one, "Google Mind Maps").

It should be like the collaborative outliner Douglass Englebart envisioned and implemented in his epic demo of NLS:

https://www.youtube.com/watch?v=yJDv-zdhzMY&t=8m49s

Engelbart also showed how to embed lists and outlines in maps:

https://www.youtube.com/watch?v=yJDv-zdhzMY&t=15m39s

Dave Winer, the inventor of RSS and founder of UserLand Software, originally developed a wonderful outliner on the Mac originally called "ThinkTank" and then "MORE", which later evolved into the "Frontier" programming language, and ultimately the "Radio Free Userland" desktop blogging and RSS syndication tool.

https://en.wikipedia.org/wiki/Dave_Winer

https://en.wikipedia.org/wiki/UserLand_Software

More was great because it had a well designed user interface and feature set with fluid "fahrvergnügen" that made it really easy to use with the keyboard as well as the mouse. It could also render your outlines as all kinds of nicely formatted and stylized charts and presentations. And it had a lot of powerful features you usually don't see in today's generic outliners.

https://en.wikipedia.org/wiki/MORE_(application)

>MORE is an outline processor application that was created for the Macintosh in 1986 by software developer Dave Winer and that was not ported to any other platforms. An earlier outliner, ThinkTank, was developed by Winer, his brother Peter, and Doug Baron. The outlines could be formatted with different layouts, colors, and shapes. Outline "nodes" could include pictures and graphics.

>Functions in these outliners included:

>Appending notes, comments, rough drafts of sentences and paragraphs under some topics

>Assembling various low-level topics and creating a new topic to group them under

>Deleting duplicate topics

>Demoting a topic to become a subtopic under some other topic

>Disassembling a grouping that does not work, parceling its subtopics out among various other topics

>Dividing one topic into its component subtopics

>Dragging to rearrange the order of topics

>Making a hierarchical list of topics

>Merging related topics

>Promoting a subtopic to the level of a topic

After the success of MORE, he went on to develop a scripting language whose syntax (for both code and data) was an outline. Kind of like Lisp with open/close triangles instead of parens! It had one of the most comprehensive implementation of Apple Events client and server support of any Mac application, and was really useful for automating other Mac apps, earlier and in many ways better than AppleScript.

https://en.wikipedia.org/wiki/UserLand_Software#Frontier

http://frontier.userland.com/

Then XML came along, and he integrated support for XML into the outliner and programming language, and used Frontier to build "Aretha", "Manila", and "Radio Userland".

http://manila.userland.com/

http://radio.userland.com/

He used Frontier to build a fully programmable blogging and podcasting platform, with a dynamic HTTP server, a static HTML generator, structured XML editing, RSS publication and syndication, XML-RPC client and server, OPML import and export, and much more.

He basically invented and pioneered outliners, RSS, OPML, XML-RPC, blogging and podcasting along the way.

>UserLand's first product release of April 1989 was UserLand IPC, a developer tool for interprocess communication that was intended to evolve into a cross-platform RPC tool. In January 1992 UserLand released version 1.0 of Frontier, a scripting environment for the Macintosh which included an object database and a scripting language named UserTalk. At the time of its original release, Frontier was the only system-level scripting environment for the Macintosh, but Apple was working on its own scripting language, AppleScript, and started bundling it with the MacOS 7 system software. As a consequence, most Macintosh scripting work came to be done in the less powerful, but free, scripting language provided by Apple.

>UserLand responded to Applescript by re-positioning Frontier as a Web development environment, distributing the software free of charge with the "Aretha" release of May 1995. In late 1996, Frontier 4.1 had become "an integrated development environment that lends itself to the creation and maintenance of Web sites and management of Web pages sans much busywork," and by the time Frontier 4.2 was released in January 1997, the software was firmly established in the realms of website management and CGI scripting, allowing users to "taste the power of large-scale database publishing with free software."

https://en.wikipedia.org/wiki/RSS

https://en.wikipedia.org/wiki/OPML

https://en.wikipedia.org/wiki/XML-RPC

Re: I was wrong about spreadsheets (2017)

#364

Post by DonHopkins » Sat, Jul 13, 2019, 2:20 AM UTC

ryanjshaw wrote: post #101

discordance wrote: post #82

Earlier quoted context omitted.

If it's O365 Excel, then you can share and work on the same spreadsheet in a similar way to Google Sheets (seeing other peoples real time cursor in your spreadsheet etc).

Be very, very careful with this. There are some update/race condition bugs in Excel Online that result in multiple users causing bulk updates to incorrectly update the filtered/sorted view of other users.

Douglass Engelbart rolls over in his grave.

Re: I was wrong about spreadsheets (2017)

#365

Post by DonHopkins » Sat, Jul 13, 2019, 2:33 AM UTC

magicnubs wrote: post #317

tonygrue wrote: post #43

Earlier quoted context omitted.

I'd be over the moon to see one change to Excel. Native support for a language other than VBA. Perhaps the CLR. Perhaps Java/Typescript. Just something (optional) for programmers who want to use Excel and not want to deal with the garbage that VBA is. Some workarounds exist, but afaik they require collaborators to also have the tool install, which is dead in the water. I know MS has considered it, I'm still pretty su…

> I know MS has considered it, I'm still pretty surprised they've haven't followed through. Didn't they add Javascript support in 2018? I agree though. They brought up that they were considering Python3 integration like 2 years ago and haven't said a word about it since.

You've been able to control all the Microsoft Office and other apps like Internet Explorer through "out of process" OLE automation for ages, and Python has a great win32com module (part of pywin32) for controlling "in process" and "out of process" OLE controls / ActiveX components.

https://pypi.org/project/pywin32/

http://timgolden.me.uk/pywin32-docs/html/com/win32com/HTML/d...

Intgrating COM into Python is one approach, but another approach is integrating Python into Active Scripting. (The age old extending/embedding debate.)

https://docs.python.org/3/extending/index.html

And Active Scripting (1996) let you plug different "in process" interpreters into the web browser and other multi-lingually scriptable applications, and call back and forth between (many but not all) ActiveX components and OLE automation interfaces more directly, without using slow "out of process" remote procedure calls. (Some components still require running in separate process, like Word, Excel, etc, which work, but are just slower to call).

https://en.wikipedia.org/wiki/Active_Scripting

https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1m...

>The Microsoft Windows Script Host (WSH) (formerly named Windows Scripting Host) is an automation technology for Microsoft Windows operating systems that provides scripting abilities comparable to batch files, but with a wider range of supported features.

>It is language-independent in that it can make use of different Active Scripting language engines. By default, it interprets and runs plain-text JScript (.JS and .JSE files) and VBScript (.VBS and .VBE files).

>Users can install different scripting engines to enable them to script in other languages, for instance PerlScript. The language independent filename extension WSF can also be used. The advantage of the Windows Script File (.WSF) is that it allows the user to use a combination of scripting languages within a single file.

>WSH engines include various implementations for the Rexx, BASIC, Perl, Ruby, Tcl, PHP, JavaScript, Delphi, Python, XSLT, and other languages.

>Windows Script Host is distributed and installed by default on Windows 98 and later versions of Windows. It is also installed if Internet Explorer 5 (or a later version) is installed. Beginning with Windows 2000, the Windows Script Host became available for use with user login scripts.

Re: I was wrong about spreadsheets (2017)

#366

Post by beatgammit » Sat, Jul 13, 2019, 2:33 AM UTC

halfeatenpie wrote: post #26

I work in critical infrastructure planning. My organization builds software in R, Python, and other programming languages customized for these major organizations. So many critical infrastructures, billions of dollars in planning, and just systems are built out of Excel. It's amazing. You'd assume something that services millions of people a day would have some more sophisticated and customized solution, but you're w…

A friend of mine works converting spreadsheets to Python/Java programs. Basically people at the company solve the problem with Excel, and when that gets to slow, he codes it up to read from a CSV and they're off to the races.

It's amazing how far these business types can get before needing a programmer to optimize things. As a programmer, I tend to think in terms of logic, and then I tried to build a complicated budget tool in Python and the turnaround was just atrocious. So, I did it in a spreadsheet instead and was far more productive.

Spreadsheets are awesome, and whoever came up with the concept should get some kind of award. I can't think of a single software tool that has enabled more productivity than a spreadsheet.

Re: I was wrong about spreadsheets (2017)

#367

Post by DonHopkins » Sat, Jul 13, 2019, 2:46 AM UTC

viraptor wrote: post #21

My issue with spreadsheets is that they could be improved a lot with minimal changes, not nobody seems to do it in popular office packages. Making them more database-like and making table data first-class (at least you can make named tables in excel on windows) could be used to push people a bit more towards organised data, without changing how anything works. Half of the mess that makes excel hell comes from the fac…

Clippy was a "Microsoft Agent" ActiveX component, that (or rather, who) you could script!

https://en.wikipedia.org/wiki/Microsoft_Agent

https://docs.microsoft.com/en-us/windows/win32/lwef/programm...

Clippy, Genie, Plany, Peedy the Parrot, Merlin the Wizard, Milton the Bear, Oscar the Cat, Max the Search Doggie, and all of their other happy friends were cheerfully retired in 2009, and are now merrily frolicking on a nice farm in the country side with a very loving family. Or so we are told.

http://bellcraft.com/mash/chars.aspx

Re: I was wrong about spreadsheets (2017)

#368

Post by kgwgk » Sat, Jul 13, 2019, 12:29 PM UTC

punnerud wrote: post #225

Spreadsheet is great and dangerous! Try to find duplicates or do vlookup in a table/sheet with more than 10.000 rows. It will only look at the first part of your data, and skip a lot if you don’t remember to sort your columns first. Yes spreadsheet are a great start. I often ask people to “prototype” in Excel, after that it is SQL that rules. And remember: Pivot in Excel = ‘Group by’ in SQL

I don't know what do you mean by "find duplicates", but VLOOKUP has a option to specify an exact match so you can use it even if the data is not sorted (MATCH does as well).

Re: I was wrong about spreadsheets (2017)

#369

Post by smsm42 » Sun, Jul 14, 2019, 1:48 AM UTC

PaulHoule wrote: post #282

eej71 wrote: post #218

Earlier quoted context omitted.

I think it will be easier to solve those problems in the Excel development space than it will be to export the Excel solutions to another solution domain. Also the kind of mistakes you've described can (and do) happen in other programming domains that are deemed more respectable.

Unfortunately no. Excel is hamstrung by having to maintain backwards compatibility for an endless number of hacks. For instance, it is notorious that 0.1 + 0.2 != 0.3 in binary exponent floating point math. Excel does funny stuff with number formatting that hides this, but it is like having a bubble under a plastic sheet that moves someplace else when you push on it -- numeric strangeness appears in different places.…

That sounds like you want to rebuild the whole paradigm of how 99% of software world treats numbers (which may be, at the end, a right thing to do but unlikely to happen at least in the near term) and you are laying the fault of it not happening at Excel's door. That would be very misplaced blame.

Re: I was wrong about spreadsheets (2017)

#370

Post by thirdsun » Sun, Jul 14, 2019, 7:45 AM UTC

ozim wrote: post #207

halfeatenpie wrote: post #26

I work in critical infrastructure planning. My organization builds software in R, Python, and other programming languages customized for these major organizations. So many critical infrastructures, billions of dollars in planning, and just systems are built out of Excel. It's amazing. You'd assume something that services millions of people a day would have some more sophisticated and customized solution, but you're w…

Yeah it works and you don't need support contracts and then: - Fidelity's "Minus Sign Mistake": loss of $1.3 billion - TransAlta "Clerical Error": loss of $24 million - Fannie Mae "Honest mistake": loss of $1.3 billion Then you get employee turn over where new employees don't get "arcane" knowledge passed down by people who left and took their spreadsheet foo with them. Excel does not have "access control", "auditing…

> Excel does not have "access control", "auditing", "change tracking". Just getting work done is not enough.

Doesn’t the web version of Excel already solve that? I’m not familiar with it but Google Sheets does these things, at least to some extent - I’d expect Office 365 to do so as well.