21–30 of 44 posts
As a lone data wrangler, I am dreaming of an "APL like R", i.e. geared specifically towards data manipulation and stats with an integrated columnar store (I am spoilt by J). People always say that such a thing will never fly in teams due to syntactic issues, but APL really is a productivity secret weapon for loners and small teams!
As a lone data wrangler, I am dreaming of an "APL like R", i.e. geared specifically towards data manipulation and stats with an integrated columnar store (I am spoilt by J). People always say that such a thing will never fly in teams due to syntactic issues, but APL really is a productivity secret weapon for loners and small teams!
Is it a matter of writing a comprehensive stats library?
Following the same principle, R dplyr allows you to seamlessly interact with the DB by using a translation layer. However, every time I open R I find myself having to write tens if not hundreds of loc to shape the data where I'd do that in a few lines of J. For single researchers, it's actually much easier to read your one-page of J code 6 months later than it is for your 500-loc R script (again IMO).
Although I imagine it could be possible to really make a specialized APL geared towards data analysis as a strict DSL (APL is not a DSL). Meaning for example, making it more static and therefore statically compileable at the expense of losing things such as first-class environments (namespaces) or the "execute" primitive. One could also specialize the notation further towards statistics. There really is a whole realm of possibilities, here!
In a word, there is a market for lone scientists. It would be nice to have tools for that market ;-)
Earlier quoted context omitted.
If you think APL is less expressive than Matlab, you probably haven't really grasped it, IMO. Having said that, Matlab is optimized for manipulating matrices and replicating the notation of normal linear algebra and has excellent implementations of basically any numerical algorithm that frequently comes up in this . So writing something like chol(X'*X+diag(eig(X))) in an APL will look uglier and quite possibly slower…
> It is completely impossible to implement something like "under" in matlab. I’m a little curious about this. Does J have a notion of the relationship between certain functions and their inverse? What is it that enables “under” in J which makes it impossible in Matlab?
Yes. Many built-in words have inverses assigned, and you can assign inverse functions to your own words with :. https://code.jsoftware.com/wiki/Vocabulary/codot
EDIT: and here's a table with predefined inverses: https://code.jsoftware.com/wiki/Vocabulary/Inverses
Earlier quoted context omitted.
Is it a matter of writing a comprehensive stats library?
A comprehensive stat library for J would be close (there is an RServe lib), but the most important thing is the integrated data store. For big companies, it makes perfect sense to want a separate data store. But for small shops or loners, tight language integration is light-years better! With J, I am allowed a real database (no effing .csv) where I can use the same (terse) language as for the analysis. This is the ki…
Commercial k variants come with a columnar data store (see Kx's q/kdb+, Shakti's k9).
Even the abstractions we build into programming languages invites a certain way of thinking which is why there’s so many different paradigms like functional, imperative, logic, declarative, etc.
This is Kenneth Iverson's 1979 Turing Award lecture.
Yes. There are some deep insights in this exposition. The irony is (in my opinion) APL is the worst Array/Matrix based programming language. In fairness it was also the first, but compared to Matlab or Julia it is not as expressive and feels much harder to use.
Earlier quoted context omitted.
A comprehensive stat library for J would be close (there is an RServe lib), but the most important thing is the integrated data store. For big companies, it makes perfect sense to want a separate data store. But for small shops or loners, tight language integration is light-years better! With J, I am allowed a real database (no effing .csv) where I can use the same (terse) language as for the analysis. This is the ki…
You didn't mention it, so for avoidance of doubt: have you heard of k? Commercial k variants come with a columnar data store (see Kx's q/kdb+, Shakti's k9).
I was thinking of more radical changes that would certainly disappoint APL purists by exchanging some well-known and highly-regarded APL capabilities for a more specialized language. Maybe I'll put my thoughts in code someday, if life doesn't get in the way.
The strength of APLs relies on having a fixed set of versatile primitives. From there, the terseness of the language allows one to write small but expressive and useful programs. This strongly limits the need for external libs and this is where the real value lies. Whereas in Python, you'd reach for multiple libs made in another language and often by other people, in APL you learn the prmitives and you're off to the races! Therefore, the fixed fraction of the programs you write (primitives vs functions) is much smaller (no libs with poor documentation) and you don't risk the carpet being swept from under you by changing lib APIs.
BTW J is stellar for data wrangling, and I encourage everyone to endure the multiple weeks of effort required to learn the basics of the language. Spend the time, it will be really rewarding!
Earlier quoted context omitted.
> It is completely impossible to implement something like "under" in matlab. I’m a little curious about this. Does J have a notion of the relationship between certain functions and their inverse? What is it that enables “under” in J which makes it impossible in Matlab?
> Does J have a notion of the relationship between certain functions and their inverse? Yes. Many built-in words have inverses assigned, and you can assign inverse functions to your own words with :. https://code.jsoftware.com/wiki/Vocabulary/codot EDIT: and here's a table with predefined inverses: https://code.jsoftware.com/wiki/Vocabulary/Inverses
But interesting nonetheless.
Earlier quoted context omitted.
If you think APL is less expressive than Matlab, you probably haven't really grasped it, IMO. Having said that, Matlab is optimized for manipulating matrices and replicating the notation of normal linear algebra and has excellent implementations of basically any numerical algorithm that frequently comes up in this . So writing something like chol(X'*X+diag(eig(X))) in an APL will look uglier and quite possibly slower…
> It is completely impossible to implement something like "under" in matlab. I’m a little curious about this. Does J have a notion of the relationship between certain functions and their inverse? What is it that enables “under” in J which makes it impossible in Matlab?
f=:(1+*&2)
f 1 2 3 4
3 5 7 9
(f^:_1)f 1 2 3 4
1 2 3 4
(f^:_1) 1 2 3 4
0 0.5 1 1.5
Earlier quoted context omitted.
> Does J have a notion of the relationship between certain functions and their inverse? Yes. Many built-in words have inverses assigned, and you can assign inverse functions to your own words with :. https://code.jsoftware.com/wiki/Vocabulary/codot EDIT: and here's a table with predefined inverses: https://code.jsoftware.com/wiki/Vocabulary/Inverses
That can definitely be implemented in Matlab with the symbolic math toolbox. But interesting nonetheless.