Table Oriented Programming (2002) • Hacker News Board

Re: Table Oriented Programming (2002)

#11

Post by chrisaycock » Thu, Dec 30, 2021, 1:31 PM UTC

I built my own table-oriented language out of frustrations I had with with time-series analysis:

Empirical has statically typed Dataframes. It can infer the type of a file's contents at compile time using a ton of metaprogramming techniques.

  >>> let trades = load("trades.csv")
  
  >>> trades
   symbol                  timestamp    price size
     AAPL 2019-05-01 09:30:00.578802 210.5200  780
     AAPL 2019-05-01 09:30:00.580485 210.8100  390
      BAC 2019-05-01 09:30:00.629205  30.2500  510
      CVX 2019-05-01 09:30:00.944122 117.8000 5860
     AAPL 2019-05-01 09:30:01.002405 211.1300  320
     AAPL 2019-05-01 09:30:01.066917 211.1186  310
     AAPL 2019-05-01 09:30:01.118968 211.0000  730
      BAC 2019-05-01 09:30:01.186416  30.2450  380
      CVX 2019-05-01 09:30:01.639577 118.2550 2880
      ...                        ...      ...  ...

Functions have generic typing by default; the caller determines the type instantiation. Here is a weighted average:

  >>> func wavg(ws, vs) = sum(ws * vs) / sum(ws)

Queries are built into the language. Here is a five-minute volume-weighted average price:

  >>> from trades select vwap = wavg(size, price) by symbol, bar(timestamp, 5m)
   symbol           timestamp       vwap
     AAPL 2019-05-01 09:30:00 210.305724
      BAC 2019-05-01 09:30:00  30.483875
      CVX 2019-05-01 09:30:00 119.427733
     AAPL 2019-05-01 09:35:00 202.972440
      BAC 2019-05-01 09:35:00  30.848397
      CVX 2019-05-01 09:35:00 119.431601
     AAPL 2019-05-01 09:40:00 204.671388
      BAC 2019-05-01 09:40:00  30.217362
      CVX 2019-05-01 09:40:00 117.224763
      ...                 ...        ...

Everything is statically typed. Misspelled column names, for example, result in an error before the script is even run!

Re: Table Oriented Programming (2002)

#12

Post by naasking » Thu, Dec 30, 2021, 1:37 PM UTC

jasode wrote: post #5

I remember reading this essay when it first came out. To try and reword it using modern terms: The author wishes that programming languages had database persistence capabilities as 1st-class built-in syntax instead of cumbersome bolted-on API functions. Examples where database syntax (i.e. SQL syntax) is 1st-class without noisy syntax of function calls, without command strings in quotes, etc : - business languages li…

> Examples where database syntax (i.e. SQL syntax) is 1st-class without noisy syntax of function calls, without command strings in quotes

There more examples which I think qualify but don't quite fit into your categories:

* The E language runs all code in "Vats", each of which is a single threaded compartment with transparent persistence.

* Taking inspiration from E, the Waterken server did this for Java, but required annotating mutable fields in a certain way so the persistence layer could track them.

* FoxPro doesn't neatly fit into your categories.

Re: Table Oriented Programming (2002)

#13

Post by rscho » Thu, Dec 30, 2021, 1:38 PM UTC

chrisaycock wrote: post #11

I built my own table-oriented language out of frustrations I had with with time-series analysis: https://www.empirical-soft.com Empirical has statically typed Dataframes. It can infer the type of a file's contents at compile time using a ton of metaprogramming techniques. >>> let trades = load("trades.csv") >>> trades symbol timestamp price size AAPL 2019-05-01 09:30:00.578802 210.5200 780 AAPL 2019-05-01 09:30:00.58…

How does it handle dirty data? Does it assign an "any" type?

Also, why do you think embedding data frames is not possible?

Re: Table Oriented Programming (2002)

#14

Post by mwexler » Thu, Dec 30, 2021, 1:42 PM UTC

jasode wrote: post #5

I remember reading this essay when it first came out. To try and reword it using modern terms: The author wishes that programming languages had database persistence capabilities as 1st-class built-in syntax instead of cumbersome bolted-on API functions. Examples where database syntax (i.e. SQL syntax) is 1st-class without noisy syntax of function calls, without command strings in quotes, etc : - business languages li…

I think you are missing the big twist. It's not just tables as 1st class citizens, but allowing logic to be driven by the tables.

Instead of config files, you update the table. Changes to the processing flow? Update the table, including dates for when the new rules apply. The tables held code which drove the processing, along with tables holding data.

It's not just orm or persistence, and not just programming in the database as stored procedures. It was an odd melange of all of this.

I ran into this in the 90s, and it was great for RAD. But it felt odd to have to code into tables, and each tool was proprietary such that moving off of the table system was a full rebuild. They usually allowed migration to new databases systems to scale, but that was all they had.

I don't expect to see a language like this come around again anytime soon, but the ideas were really interesting in a world before git-ops and yaml configs.

Re: Table Oriented Programming (2002)

#15

Post by ModernMech » Thu, Dec 30, 2021, 1:42 PM UTC

jasode wrote: post #5

I remember reading this essay when it first came out. To try and reword it using modern terms: The author wishes that programming languages had database persistence capabilities as 1st-class built-in syntax instead of cumbersome bolted-on API functions. Examples where database syntax (i.e. SQL syntax) is 1st-class without noisy syntax of function calls, without command strings in quotes, etc : - business languages li…

> In any case, I don't see any trend where a general-purpose programming language will include DB SQL as 1st-class. Even the recent languages like Rust and Zig don't have basic SQLite3 db persistence as convenient built-in syntax.

There is a trend, but you’ll have to look further off the beaten path than even Zig to find it. Languages like Eve [0] tried to do this circa 2015 in the tradition of Datalog. Code was written in “blocks” that resembled Prolog horn clauses, but which featured set semantics on selected records. Natural joins happened automatically on records using identifiers. The whole language was actually a database!

Eve died [1], but you’ll see many such projects that have the same ethos in communities in the web, such as this one [2].

There aren’t a lot of users of these languages, but this is where a lot of big ideas are percolating right now.

And we can verify it’s a trend because the hallmark of all CS trends, the formation of a conference, has made itself known in this area [3].

0: http://witheve.com/

1: https://groups.google.com/g/eve-talk/c/YFguOGkNrBo?pli=1

2: https://futureofcoding.org/catalog/

3: https://www.hytradboi.com/

Re: Table Oriented Programming (2002)

#16

Post by chrisaycock » Thu, Dec 30, 2021, 1:45 PM UTC

rscho wrote: post #13

chrisaycock wrote: post #11

I built my own table-oriented language out of frustrations I had with with time-series analysis: https://www.empirical-soft.com Empirical has statically typed Dataframes. It can infer the type of a file's contents at compile time using a ton of metaprogramming techniques. >>> let trades = load("trades.csv") >>> trades symbol timestamp price size AAPL 2019-05-01 09:30:00.578802 210.5200 780 AAPL 2019-05-01 09:30:00.58…

How does it handle dirty data? Does it assign an "any" type? Also, why do you think embedding data frames is not possible?

Missing and poorly formatted input is given a type-specific value. Eg., Float64 is nan and Int64 is nil.

  >>> Int64("5")
  5

  >>> Int64("5b")
  nil

If inferencing cannot determine a consistent type from a CSV file, then the column will just be a String.

I don't know what you mean by "embedding" a Dataframe.

Re: Table Oriented Programming (2002)

#17

Post by naasking » Thu, Dec 30, 2021, 1:48 PM UTC

gpderetta wrote: post #9

My wish-list for my ideal (non-system) programming language: - first class tables and named tuples as the primary datastructure. Includes the full set of relational operations, and transaction support. Optional persistence. Everything is not a table though. Tables are great but pragmatism trumps dogmatism. - structural typing (ties neatly with the above) and support for row polymorphism - shared nothing, distributed,…

I'm no longer convinced of the need for row or record polymorphism. It encourages passing around types that have no clear domain or purpose, so I think it inhibits understanding in general. Do you have any examples where it's indispensable?

Re: Table Oriented Programming (2002)

#18

Post by tyingq » Thu, Dec 30, 2021, 1:53 PM UTC

tkindy wrote: post #10

zokier wrote: post #8

I'm in the opinion that tables would make a lot of sense as first-class citizens for shell environments. Lots of data typically handled in shells is inherently tabular in nature (for example the outputs of ls and ps etc) and some of the common tools also are intended for tables (awk in the forefront, but also cut and sort as examples). But in practice lot of it is currently very ad-hoc, and handles any sort of edge c…

I think Microsoft Powershell [0] sort of approaches what you’re describing. It’s not exactly table-oriented, but object-oriented such that there’s a lot more structure to data than in traditional command line environments. For example, their equivalent of ls returns an array of objects (i.e. rows) which you can filter, sort, etc. based on the properties of those objects. [0]: https://docs.microsoft.com/en-us/powershe…

Seconded. It also comes with Import-CSV and Export-CSV. And cmdlets like Select-Object and Where-Object.

  Get-Service | Where-Object {$_.Status -eq "Stopped"}

Looks pretty close to what's being described.

Re: Table Oriented Programming (2002)

#19

Post by jasode » Thu, Dec 30, 2021, 2:02 PM UTC

mwexler wrote: post #14

jasode wrote: post #5

I remember reading this essay when it first came out. To try and reword it using modern terms: The author wishes that programming languages had database persistence capabilities as 1st-class built-in syntax instead of cumbersome bolted-on API functions. Examples where database syntax (i.e. SQL syntax) is 1st-class without noisy syntax of function calls, without command strings in quotes, etc : - business languages li…

I think you are missing the big twist. It's not just tables as 1st class citizens, but allowing logic to be driven by the tables. Instead of config files, you update the table. Changes to the processing flow? Update the table, including dates for when the new rules apply. The tables held code which drove the processing, along with tables holding data. It's not just orm or persistence, and not just programming in the…

>, but allowing logic to be driven by the tables. Instead of config files, you update the table. Changes to the processing flow? Update the table, [...]

I didn't miss that angle and I think it's actually a minor part of his thesis. If you look at the entire essay, the vast majority of his bullet points and supporting examples are mostly about ergonomics of builtin syntax to manipulate tables. If his ideal _language_ (aka the syntax) did that, it would naturally support table-oriented-programming (aka the philosophy). He starts the essay with critique of OOP-the-syntax.

But to your point about config and code itself being persisted in the database, the SAP ABAP environment already works like that. SAP has over 10,000 db tables for configuration -- instead of YAML or JSON files. Change the values in the config tables to alter behavior instead of modifying IF/THEN/ENDIF statements in code. And when ABAP programmers hit "save", the code gets saved to a database table instead of a text file. So if one squints a certain way, the SAP system is a giant million-line stored procedure in the database.

Re: Table Oriented Programming (2002)

#20

Post by rscho » Thu, Dec 30, 2021, 2:02 PM UTC

chrisaycock wrote: post #16

rscho wrote: post #13

Earlier quoted context omitted.

How does it handle dirty data? Does it assign an "any" type? Also, why do you think embedding data frames is not possible?

Missing and poorly formatted input is given a type-specific value. Eg., Float64 is nan and Int64 is nil. >>> Int64("5") 5 >>> Int64("5b") nil If inferencing cannot determine a consistent type from a CSV file, then the column will just be a String. I don't know what you mean by "embedding" a Dataframe.

On the website you linked:

"Embedding Dataframes into an existing language would not be possible."

I don't think it would be an issue for languages with good metaprogramming facilities.