Live data from Hacker News

Excel 2.0 – Is there a better visual data model than a grid of cells?

subset.so

211–220 of 223 posts

Re: Excel 2.0 – Is there a better visual data model than a grid of cells?

#211
post #85

Trillions of decisions have been made off a tool that has poor error handling and data consistency issues. Blame is not entirely on either the user or the software, but the tool is too trusted without validation. [Study that was at the core for Europe's austerity and European debt crisis contained excel errors when fixed showed the inverse of original hypothesis.]( https://www.nytimes.com/2013/04/19/opinion/krugman-t…

Give https://exploratory.io/ a look. It's free/cheap.

It's a nice easy GUI wrapper for R and just works.

I stumbled across it a year ago and now use it daily.

Re: Excel 2.0 – Is there a better visual data model than a grid of cells?

#212

Earlier quoted context omitted.

Airtable is not really that -- it does not do much/anything in the way of calculations or aggregations. I love it and it's great for creating very structured ways of managing data, but it can't do even very basic aggregations.

Not entirely true. Airtable has formula fields [1] and it can indeed summarize columns, and even slice and dice using crosstabs [2]. [1] https://support.airtable.com/hc/en-us/articles/203255215-For... [2] https://support.airtable.com/hc/en-us/articles/115013249307-...

This in principle accurate, but the real story is less polished. If you look through the AirTable forums, you’ll see lots of cases where users are asking how to do a thing, and are told to abandon the highly limited scripting and to use JavaScript. This is what I characterize as a learning cliff.

Re: Excel 2.0 – Is there a better visual data model than a grid of cells?

#213
post #165

For a while now I’ve been experimenting with the idea of a strongly typed spreadsheet. There is no grid model; instead computation occurs in discrete cells, where each cell can store one value, with ‘value’ defined as in programming languages to include data structures like lists and tables. Each cell gets its own type, and the formula language is type-checked before running, to allow flexible yet error-checked repre…

See my app for an implementation of this.

I have in fact seen Inflex already! And I must admit to being quite surprised at seeing pretty much exactly the same UI I had imagined, already implemented. But I suspect there are still some major underlying differences between Inflex and what I’ve been doing. As a Haskeller, my emphasis has been very much on strong typing and associated language features: mine uses a Hindley–Milner variant with subtyping, allowing some information to be stored at the type level (e.g. units) and functions to be automatically broadcast to higher dimensions. The user can explicitly specify the type of each cell, allowing for more control over data modelling. Basically, I want to make a tool such that the user can create arbitrarily large spreadsheets, without running into the scalability problems of Excel etc. By contrast, Inflex seems to have a somewhat different and rather more minimal focus: simply making an easy-to-understand analogue of traditional spreadsheets which solves their biggest problems, without worrying about extraneous stuff.

That being said, I will confess to not having tried Inflex all that much, so I may be misunderstanding it. I would be very interested in hearing your thoughts on this topic.

Re: Excel 2.0 – Is there a better visual data model than a grid of cells?

#214
post #87
post #83

Earlier quoted context omitted.

Many things that Numbers seems to me to be better than Excel from a formatting perspective: * Freeze header rows & columns. * Naming header rows & columns. * Graphs that don't overlap the sheet. Things that I find Excel does better than Numbers from a data perspective: * Data validation * Large tables * Formula Error checking

You can freeze header rows in Excel. This happens automatically if you use Insert -> Table. That also gets you the ability to name columns.

Excel tables are so amazing. They solve so many problems.

Re: Excel 2.0 – Is there a better visual data model than a grid of cells?

#215
post #93

Earlier quoted context omitted.

I think holy grail is ai-assisted system where you can ask by plain English queries. Something like Wolfram alpha + gtp-3 + erp.

I am currently building exactly this. A CLI that connects to an SQL database, or allows users to upload .tsv, .csv, .xls files and then run plain English queries against the data source.

I have had same kind idea but then moved on becouse getting user trust the system is very hard and this needs more qui than ai work. But good luck it is definetly those unicorn ideas if it succees.

Re: Excel 2.0 – Is there a better visual data model than a grid of cells?

#216
post #213

Earlier quoted context omitted.

See my app for an implementation of this.

I have in fact seen Inflex already! And I must admit to being quite surprised at seeing pretty much exactly the same UI I had imagined, already implemented. But I suspect there are still some major underlying differences between Inflex and what I’ve been doing. As a Haskeller, my emphasis has been very much on strong typing and associated language features: mine uses a Hindley–Milner variant with subtyping, allowing…

Inflex is statically typed, it’s pretty much PureScript with more familiar-to-Excel syntax, with row types used for records and also polymorphic variants, which aren’t in PureScript but are in OCaml. Numbers and ordering and comparisons are dealt with via type classes.

You can annotate a cell’s expression with a type signature, though this is not explicitly mentioned or supported intentionally as I’m not decided fully on the syntax. Example: when you make a “table” via the Table button, it just makes a cell whose source code is: [] :: [{"column1":_, "column2":_}] as tables are just lists of records. If you go to https://inflex.io/try (work not saved, this page doesn’t hit the DB whatsoever) you can hit Formula to write code. But all cells are code underneath (hit the triangle).

My next addition to Inflex will be push-pull based FRP, for dealing with time, buttons and external events and outputs. I also plan on having it scale so that cells with large tables are refined into real database rows for more efficient operations. So the goal of Inflex is to embrace the small scale (lists and easy spreadsheet stuff) with a smooth migration path towards more advanced programming, in a coherent whole. But you have to start with the simple and work your way to the fancy stuff, or so I think.

I’ve considered units of measure, but they are complicated from an end-user perspective and complicate the type system, I’m not sure whether they’re worth it over providing very good automated property based testing. That’s an ongoing consideration.

Development is slow in my spare time which is limited, but I’m comfortable going at my own pace. I’d like to roll out a discourse forum to document things, but hosting is quite expensive. It’s worth resolving this, though, because I’m doing a poor job at explaining the product. One is always choosing between adding more polish/features and documenting!

Re: Excel 2.0 – Is there a better visual data model than a grid of cells?

#217
post #213

Earlier quoted context omitted.

I have in fact seen Inflex already! And I must admit to being quite surprised at seeing pretty much exactly the same UI I had imagined, already implemented. But I suspect there are still some major underlying differences between Inflex and what I’ve been doing. As a Haskeller, my emphasis has been very much on strong typing and associated language features: mine uses a Hindley–Milner variant with subtyping, allowing…

Inflex is statically typed, it’s pretty much PureScript with more familiar-to-Excel syntax, with row types used for records and also polymorphic variants, which aren’t in PureScript but are in OCaml. Numbers and ordering and comparisons are dealt with via type classes. You can annotate a cell’s expression with a type signature, though this is not explicitly mentioned or supported intentionally as I’m not decided full…

> Inflex is statically typed, it’s pretty much PureScript with more familiar-to-Excel syntax, with row types used for records and also polymorphic variants, which aren’t in PureScript but are in OCaml. Numbers and ordering and comparisons are dealt with via type classes.

Huh, I never realised. I think it’s worth at least mentioning this in the documentation, if only to note its existence.

> You can annotate a cell’s expression with a type signature, though this is not explicitly mentioned or supported intentionally as I’m not decided fully on the syntax. Example: when you make a “table” via the Table button, it just makes a cell whose source code is: [] :: [{"column1":_, "column2":_}] as tables are just lists of records.

I ended up with pretty much the same design, except tables were a primitive type, stored as a record-of-lists, and type annotations were a single colon.

> I also plan on having it scale so that cells with large tables are refined into real database rows for more efficient operations.

Ooh, clever! I like this idea.

> So the goal of Inflex is to embrace the small scale (lists and easy spreadsheet stuff) with a smooth migration path towards more advanced programming, in a coherent whole.

Another thought along these lines: quite a lot of my spreadsheet use involves doing calculations on whole columns. This is really painful in Excel, where I have to manually select just the right areas of my sheet, then write the formula—without disturbing my selections!—making sure of course that I put the result in a place where it fits. And this just gets worse with more data. In my own proof-of-concept, I implement broadcasting of all functions over arrays (à la APL or MATLAB), which makes this a lot easier, and is quite possibly the best UX improvement I’ve found. Do you have any plans to do something like this in Inflex? (I know it already has higher-order functions, but e.g. `voltage / current` is a lot easier to read and write than `zipwith(x y:x/y, voltage, current)`…)

> I’ve considered units of measure, but they are complicated from an end-user perspective and complicate the type system, I’m not sure whether they’re worth it over providing very good automated property based testing. That’s an ongoing consideration.

From my perspective, they’re definitely worth it: most of my biggest Excel mistakes have involved botched unit conversions. (If it matters, my background is physics and chemistry; most of my spreadsheet usage has been for assignments throughout my degree.)

From the implementation point of view, I found units pretty easy to integrate. The most difficult part was probably inserting the automatic conversions: in order to make this possible, I had to structure my inference algorithm slightly differently to the usual pattern, by applying substitutions at the same time as unification, rather than afterwards. (Which ended up having other benefits as well, e.g. the aforementioned broadcasting.)

> Development is slow in my spare time which is limited, but I’m comfortable going at my own pace.

If it helps, I’d be more than happy to help out with development! This is an area I find incredibly interesting to work on, but my own attempt never really got past the proof-of-concept stage, and I consider duplicate development a bit of a waste.

Re: Excel 2.0 – Is there a better visual data model than a grid of cells?

#218
post #217

Earlier quoted context omitted.

Inflex is statically typed, it’s pretty much PureScript with more familiar-to-Excel syntax, with row types used for records and also polymorphic variants, which aren’t in PureScript but are in OCaml. Numbers and ordering and comparisons are dealt with via type classes. You can annotate a cell’s expression with a type signature, though this is not explicitly mentioned or supported intentionally as I’m not decided full…

> Inflex is statically typed, it’s pretty much PureScript with more familiar-to-Excel syntax, with row types used for records and also polymorphic variants, which aren’t in PureScript but are in OCaml. Numbers and ordering and comparisons are dealt with via type classes. Huh, I never realised. I think it’s worth at least mentioning this in the documentation, if only to note its existence. > You can annotate a cell’s…

> Another thought along these lines: quite a lot of my spreadsheet use involves doing calculations on whole columns. This is really painful in Excel, where I have to manually select just the right areas of my sheet, then write the formula—without disturbing my selections!—making sure of course that I put the result in a place where it fits. And this just gets worse with more data. In my own proof-of-concept, I implement broadcasting of all functions over arrays (à la APL or MATLAB), which makes this a lot easier, and is quite possibly the best UX improvement I’ve found. Do you have any plans to do something like this in Inflex? (I know it already has higher-order functions, but e.g. `voltage / current` is a lot easier to read and write than `zipwith(x y:x/y, voltage, current)`…)

I haven’t seriously put thought into that use-case. There’s the facility via type classes, but I’m not sure what the trade offs are with that kind of overloading. The same question applies to FRP behaviors and events. Do you explicitly require a map or list comprehensions or do you “broadcast” automatically, and if you do, what are the pitfalls?

I have put some thought into something related which is aggregations. On a given table I want to be able to add a column which implements any mapAccum over the table, such as balance over time. The code generated would be table.accum(state: row: state + row.x) for example. The UI would make it easy to pick common accumulators and folds, but also trivial to write an expression. Your voltage current example would work like that.

This arrays-of-records overlaps a bit with your records-of-arrays naturally. The two are often considered sides of the same coin and I’ve thought about using RoA internally for efficiency, but “broadcasting” to make that nice didn’t occur to me. The written simplicity is compelling, for sure.

> From my perspective, they’re definitely worth it: most of my biggest Excel mistakes have involved botched unit conversions. (If it matters, my background is physics and chemistry; most of my spreadsheet usage has been for assignments throughout my degree.)

Yeah, it’s definitely a valuable thing, I’m just not sure about the trade offs. At some point you have to stop adding features to the language, and that one didn’t convince me enough at the time.

> applying substitutions at the same time as unification

Right, I’ve seen that also on a language we develop at work. Incidentally this is needed if you want to combine two row types and assign a type to the expression.

> If it helps, I’d be more than happy to help out with development! This is an area I find incredibly interesting to work on, but my own attempt never really got past the proof-of-concept stage, and I consider duplicate development a bit of a waste.

I might only have time to discuss ideas and no time to collaborate on dev right now, but happy to swap contact details. If you drop a mail anything@inflex.io I’ll reply via my personal email.

Re: Excel 2.0 – Is there a better visual data model than a grid of cells?

#219
post #126

I think Eirik Bakke deserves a shoutout on this thread. He's the creator of UltOrg ( https://www.ultorg.com/ ) and has literally spent his career pursuing this question with a level of focus unlike anything I've ever seen. UltOrg is roughly "spreadsheets re-built atop the RDBMS datamodel". The UI supports nested joins, aggregations, filtering, for both display and data update. The result is essentially a general purp…

> UltOrg is roughly "spreadsheets re-built atop the RDBMS datamodel". That's how AirTable is described as well. And Google Tables. And Amazon Honeycode.

Looking at UltOrg's website, it is not like Airtable. It's not a spreadsheet or spreadsheet interface (though I suppose it could be used that way); it is a much more automated, sophisticated UI for complex relational data. It looks very impressive.

Re: Excel 2.0 – Is there a better visual data model than a grid of cells?

#220
post #217

Earlier quoted context omitted.

> Inflex is statically typed, it’s pretty much PureScript with more familiar-to-Excel syntax, with row types used for records and also polymorphic variants, which aren’t in PureScript but are in OCaml. Numbers and ordering and comparisons are dealt with via type classes. Huh, I never realised. I think it’s worth at least mentioning this in the documentation, if only to note its existence. > You can annotate a cell’s…

> Another thought along these lines: quite a lot of my spreadsheet use involves doing calculations on whole columns. This is really painful in Excel, where I have to manually select just the right areas of my sheet, then write the formula—without disturbing my selections!—making sure of course that I put the result in a place where it fits. And this just gets worse with more data. In my own proof-of-concept, I implem…

> I haven’t seriously put thought into that use-case. … Do you explicitly require a map or list comprehensions or do you “broadcast” automatically, and if you do, what are the pitfalls?

It’s all automatic. Basically, any function a→b can be used as [a]→[b], [[a]]→[[b]], and so on; functions with multiple arguments work similarly. In the intermediate language, each argument is then annotated with the amount by which it needs to be lifted. (So e.g. if `xs : [Int]`, then `10/xs` gets translated to something like `(/)({10,1}, {xs,0})`.

As for pitfalls… well, I haven’t encountered all that many yet. The implementation is a little unprincipled at this point, but there’s probably some way to make it more rigorous. It’s basically as close as I can get to J or MATLAB in a typed language, and people don’t often have problems with broadcasting in those languages.

> I have put some thought into something related which is aggregations. On a given table I want to be able to add a column which implements any mapAccum over the table, such as balance over time. The code generated would be table.accum(state: row: state + row.x) for example. The UI would make it easy to pick common accumulators and folds, but also trivial to write an expression. Your voltage current example would work like that.

This does sound quite similar to what I’m doing, though perhaps more limited.

> Incidentally [applying substitutions at the same time as unification] is needed if you want to combine two row types and assign a type to the expression.

Hmm, interesting. You don’t happen to have a link, do you? (Though I seem to recall hearing TAPL has something about this… I really do need to read it one of these days.)

> I might only have time to discuss ideas and no time to collaborate on dev right now, but happy to swap contact details. If you drop a mail anything@inflex.io I’ll reply via my personal email.

Sure, that sounds great! I’ll send you an email now.

Post reply on HN