Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Table Oriented Programming (2002)

web.archive.org

Post reply on HN
Print view View original

51–60 of 91 posts

Re: Table Oriented Programming (2002)

#51

Post by mdcfrancis »

iamwil wrote: post #45
chrisaycock wrote: post #43

Earlier quoted context omitted.

Empirical is statically typed. Python and q/kdb+ are dynamically typed. I spent years using those products in finance. I would set-up a simulation that would crash after four hours because of a misspelled column name. Empirical prevents that by refusing to run a script that has a type error or unresolved identifier. No more crashed overnight sims!

You should say this under the question of how it's different than Julia. It's not enough to say it's statically typed, since not everyone is convinced of the benefits based on the context they're coming from. I just saw a talk by Rich Hickey about Clojure, and he eschews static typing, since he thinks of it as a coupling in a language. And based on the types of programs he writes and runs, he hasn't seen a benefit. S…

I can answer for the type stable julia case, if you have a struct in julia that is composed only of primitive types this is stored as a C struct with zero overhead and fixed byte length. An array of these is then crazy efficient when it comes to streaming into the CPU etc. If you dig around the GPU support in Julia you can see this used to good effect.

Re: Table Oriented Programming (2002)

#52

Post by CRConrad »

jasode wrote: post #5

I remember reading this essay when it first came out. To try and reword it using modern terms: The author wishes that programming languages had database persistence capabilities as 1st-class built-in syntax instead of cumbersome bolted-on API functions. Examples where database syntax (i.e. SQL syntax) is 1st-class without noisy syntax of function calls, without command strings in quotes, etc : - business languages li…

> In any case, I don't see any trend where a general-purpose programming language will include DB SQL as 1st-class. Even the recent languages like Rust and Zig don't have basic SQLite3 db persistence as convenient built-in syntax. If anyone proposed to add such db syntax, they would most likely reject it.

What poor old BottomFeeder missed was that with a good object library / framework, you can get so close that it almost doesn't matter. I tried to convince him to even try Delphi, with its marvellous TDataSet descendants in the VCL... But AFAIK he never even downloaded the free version I pointed him to, whatever it may have been called back then.

Re: Table Oriented Programming (2002)

#53

Post by [deleted] »

nine_k wrote: post #22
jasode wrote: post #5

I remember reading this essay when it first came out. To try and reword it using modern terms: The author wishes that programming languages had database persistence capabilities as 1st-class built-in syntax instead of cumbersome bolted-on API functions. Examples where database syntax (i.e. SQL syntax) is 1st-class without noisy syntax of function calls, without command strings in quotes, etc : - business languages li…

Such tools existed and were popular in 1990s: DBase, Clipper, FoxPro. They worked pretty well in their domain: data entry and report generation, with lightweight transaction processing and general computation. Then happened the internet and client-server architectures, and these do not map as neatly onto local, single-user, single-transaction tables.

[deleted]

Re: Table Oriented Programming (2002)

#54

Post by maest »

beaumayns wrote: post #34
chrisaycock wrote: post #11

I built my own table-oriented language out of frustrations I had with with time-series analysis: https://www.empirical-soft.com Empirical has statically typed Dataframes. It can infer the type of a file's contents at compile time using a ton of metaprogramming techniques. >>> let trades = load("trades.csv") >>> trades symbol timestamp price size AAPL 2019-05-01 09:30:00.578802 210.5200 780 AAPL 2019-05-01 09:30:00.58…

This is pretty cool. I've had thoughts (or dreams, more accurately :) of a language like this every time I get a runtime 'type error in q. I gotta say, I prefer q's syntax, though :)

q with static typing and a sensible pricing model would be amazing.

I do think that q's main strength is not its speed, but the fact that qSQL statements are a first class citizen in the language - no network hops, no awkward marshalling and unmarshalling of data, no awkward mismatch around how to use nulls, nans, tz-aware timestamps etc.

Re: Table Oriented Programming (2002)

#55

Post by maest »

xwolfi wrote: post #32
chrisaycock wrote: post #26

Earlier quoted context omitted.

Ah, I see what you're referring to. The hardest thing is the load() function, particularly in the REPL. It looks dynamic, but is actually static. Pulling off this slight-of-hand requires both type providers and automatic compile-time function evaluation on arbitrary expressions. F# is the only other language I know of that has type providers. They invented it. As for CTFE, languages like Zig and D require the user to…

I don't think I get it. I do a lot of pandas in a bank so I recognize your dataframes for what they are, but what advantage do you have over python+pandas ? I hate Python (I'm a Java dev helping Quants), but it's that or KDB, and I think I could murder the creator of KDB :D And I have to admit Pandas is instinctive, Python is easy enough to extend, what are you doing that's so important you made a language for it ?

> python+pandas

Another advantage is supporting sql-like syntax natively (and not having to use pandas' awkward, bolted-on API)

Re: Table Oriented Programming (2002)

#56

Post by CRConrad »

nine_k wrote: post #22
jasode wrote: post #5

I remember reading this essay when it first came out. To try and reword it using modern terms: The author wishes that programming languages had database persistence capabilities as 1st-class built-in syntax instead of cumbersome bolted-on API functions. Examples where database syntax (i.e. SQL syntax) is 1st-class without noisy syntax of function calls, without command strings in quotes, etc : - business languages li…

Such tools existed and were popular in 1990s: DBase, Clipper, FoxPro. They worked pretty well in their domain: data entry and report generation, with lightweight transaction processing and general computation. Then happened the internet and client-server architectures, and these do not map as neatly onto local, single-user, single-transaction tables.

> Such tools existed and were popular in 1990s: DBase, Clipper, FoxPro.

Also Crystal Reports, Paradox...

> They worked pretty well in their domain: data entry and report generation, with lightweight transaction processing and general computation.

Having bought Ashton Tate, Borland got DBase and Interbase in addition to Paradox, and built data access components into the VCL class library (in effect "almost-first-class citizens" of the language), which IMO made Delphi the natural and superior successor to those languages: Not just "lightweight", but fully advanced (i.e, ~C++-level) general computation. (And with transaction processing built into the RDBMS connection components.)

> Then happened the internet and client-server architectures, and these do not map as neatly onto local, single-user, single-transaction tables.

Weeelll... Seen the spate of recent posts on here about how SQLite is good enough for pretty much anything? :-) And arguably, that's where Delphi was at too, over twenty years ago: AFAICR, there was a "Fishbase" (facts about tropical fish) demo included with Delphi, which in one variant could be built as a standalone Web service / server.

Also, AFAICS, that's where Free Pascal / Lazarus is at now, only using SQLite / Firebird / MySQL / PostgreSQL (and lots of other DBMSes) in stead of DBase / Clipper / FoxPro / Crystal Reports / Paradox. (I've been planning to look into that a bit closer myself, but haven't got around to it. Procrastinating away too much of my time on Hacker News, I suppose. :-( )

Re: Table Oriented Programming (2002)

#57

Post by chrisaycock »

maest wrote: post #54
beaumayns wrote: post #34

Earlier quoted context omitted.

This is pretty cool. I've had thoughts (or dreams, more accurately :) of a language like this every time I get a runtime 'type error in q. I gotta say, I prefer q's syntax, though :)

q with static typing and a sensible pricing model would be amazing. I do think that q's main strength is not its speed, but the fact that qSQL statements are a first class citizen in the language - no network hops, no awkward marshalling and unmarshalling of data, no awkward mismatch around how to use nulls, nans, tz-aware timestamps etc.

I started Empirical with the goal of "q like Haskell". The end result went in a radically different direction, but the guiding light has always been to have a statically typed language where tables and queries are a first-class operation.

The source code is publicly available under AGPL with the Commons Clause:

https://github.com/empirical-soft/empirical-lang

Re: Table Oriented Programming (2002)

#58

Post by CRConrad »

civilized wrote: post #30
jasode wrote: post #5

I remember reading this essay when it first came out. To try and reword it using modern terms: The author wishes that programming languages had database persistence capabilities as 1st-class built-in syntax instead of cumbersome bolted-on API functions. Examples where database syntax (i.e. SQL syntax) is 1st-class without noisy syntax of function calls, without command strings in quotes, etc : - business languages li…

> I remember reading this essay when it first came out. To try and reword it using modern terms: The author wishes that programming languages had database persistence capabilities as 1st-class built-in syntax instead of cumbersome bolted-on API functions. This reminds me of M/MUMPS, used by Epic to power the biggest EHR system by market share in the US. Perhaps the big difference is that the M "database" is key-value…

> Is this what ORMs are about? Translating between the programmer model of data and the relational DB model?

Pretty much, AFAICS.

With varying degrees of success.

Re: Table Oriented Programming (2002)

#59

Post by rak1507 »

xwolfi wrote: post #36
gpderetta wrote: post #24

Earlier quoted context omitted.

The K language of kdb also count, but being proprietary and having a fairly impenetrable and alien syntax haven't helped it branching out of the niche where it is very successfully.

Isn't Q the language, and Kx the company ? I have to use it at work, helping quants try to actually maintain production code rather than just vomit horrible one-time scripts they patch together in an endless stream of layers on top of layers. This thing should be banned. Everything is one-letter, it's impossible to Google, it takes the opposite decision of every imaginable convention, I have yet to see someone who ca…

Why is it still used then? Are there just no good alternatives? Or legacy reasons?

Re: Table Oriented Programming (2002)

#60

Post by beagle3 »

rak1507 wrote: post #59
xwolfi wrote: post #36

Earlier quoted context omitted.

Isn't Q the language, and Kx the company ? I have to use it at work, helping quants try to actually maintain production code rather than just vomit horrible one-time scripts they patch together in an endless stream of layers on top of layers. This thing should be banned. Everything is one-letter, it's impossible to Google, it takes the opposite decision of every imaginable convention, I have yet to see someone who ca…

Why is it still used then? Are there just no good alternatives? Or legacy reasons?

On the things it does well - being a memory mapped column store, it hardly had any competition a few years ago (clickhouse might be getting there these days, perhaps)

And if your fluent in K, one-of scripts and queries are significantly shorter and easier to get right.

But it is not good as a general purpose language in a commercial setting - not because of the language itself which is fine if a bit spartan - but because it is hard to find people who are willing and able to work with it.

Much like its predecessor APL, it’s a tool for thought more than a tool for implementation.

Post reply on HN