Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Owl – OCaml Scientific Computing

ocaml.xyz

Post reply on HN
Print view View original

41–50 of 80 posts

Re: Owl – OCaml Scientific Computing

#41

Post by chewbaxxa »

ernst_klim wrote: post #32
chewbaxxa wrote: post #27

Earlier quoted context omitted.

How do you deal with data you haven’t seen before? Importing a large csv for example where a lot of the data is the wrong type or the wrong format.

Like in any other language, write a parser? What do you mean by the wrong format?

Sorry, I meant type inference. In python you could import all the data and deal with types later. Things that aren’t the expected type can be dealt with individually. It’s not pretty but it’s fast and it works.

Have you experienced any problems with static typing in these situations? I appreciate the value of static typing but I’m not sure if it offers substantial benefit when working interactively with data.

“The flexibility of dataframe largely comes from the dynamic typing inherently offered in a language. Due to OCaml’s static type checking, this poses greatest challenges to Owl when I was trying to introduce the similar functionality.”

“To be efficient, Dataframe only takes maximum the first 100 lines in the CSV file for inference. If there are missing values in a column of integer type, it falls back to float value because we can use nan to represent missing values. If the types have been decided based on the first 100 lines, any following lines containing the data of inconsistent type will be dropped.”

http://ocaml.xyz/chapter/dataframe.html

Re: Owl – OCaml Scientific Computing

#43

Post by ernst_klim »

chewbaxxa wrote: post #41
ernst_klim wrote: post #32

Earlier quoted context omitted.

Like in any other language, write a parser? What do you mean by the wrong format?

Sorry, I meant type inference. In python you could import all the data and deal with types later. Things that aren’t the expected type can be dealt with individually. It’s not pretty but it’s fast and it works. Have you experienced any problems with static typing in these situations? I appreciate the value of static typing but I’m not sure if it offers substantial benefit when working interactively with data. “The fl…

> In python you could import all the data and deal with types later.

You can't, you need to know what you are parsing, a number, a complex number, a symbol etc.

> deal with types later

What does it mean? Dynamic typed language is still typed, all the expressions have types.

Just like in Python, you can define types in-place with polymorphic variants and objects, so OCaml would infer their types.

    let instant_complex = object method re = 3.14 method im = 0.0 end
would infer

    instant_complex :
> If there are missing values in a column of integer type, it falls back to float value because we can use nan to represent missing values

Yeah, it's soundness vs completeness. If you want to, you could make your field optional, or made them a subtype of object, which could be None. There is no difference from Python here.

It's just a choice made in favor of soundness and convenience (because using `int option` in case of missing ints would be inconvenient for the most part).

Nobody prohibits Python-like solution in OCaml

    class virtual number = ...

    let none : number = object .. end

    class float = object inherit number ... end
etc. Dynamic typing could be easily replicated within any static language, it's just not why people use static languages, aka soundness.

Re: Owl – OCaml Scientific Computing

#44

Post by srean »

chewbaxxa wrote: post #41
ernst_klim wrote: post #32

Earlier quoted context omitted.

Like in any other language, write a parser? What do you mean by the wrong format?

Sorry, I meant type inference. In python you could import all the data and deal with types later. Things that aren’t the expected type can be dealt with individually. It’s not pretty but it’s fast and it works. Have you experienced any problems with static typing in these situations? I appreciate the value of static typing but I’m not sure if it offers substantial benefit when working interactively with data. “The fl…

In Python (Pandas) you pass the type specification like this when calling read_csv()

---

dtype : Type name or dict of column -> type, optional

Data type for data or columns. E.g. {‘a’: np.float64, ‘b’: np.int32, ‘c’: ‘Int64’}

--- https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

You could of course parse every column as a string and then cast to an appropriate type at run time interactively. For timestamps and such pandas can look at the data to figure out the exact datetime format. Not sure what one loses in OCaml especially if one is working at the repl.

Re: Owl – OCaml Scientific Computing

#46

Post by o09rdk »

seanmcdirmid wrote: post #26
pjmlp wrote: post #17

Earlier quoted context omitted.

Although Julia is a dynamic language, the way it uses type inference and type annotations, you can also achieve a similar experience. Naturally OCaml benefits from almost 20 years existence.

How does Julia’s dynamic typing augment discoverability? Does it magically tie into a code completion engine somehow?

My understanding/experience is that Julia has optional typing. Meaning it's dynamically typed, but supports type annotation that often improves performance and can be used to enforce types (I think).

A lot of Julia code looks statically typed, but if you want to code "pure" dynamically (e.g., for prototyping or just because it's more convenient or better for whatever reason) in style you can. Type annotation is seen as increasing information for the compiler to use, and to increase clarity in specification, but not a necessity.

This is different from other dynamic languages I'm familiar with where type specification and annotation isn't built in to the same extent, and different from static languages that require type specification all the time.

To me, Julia's approach feels the best of the languages I've used. I've grown to like statically typed languages more over time, but there are some situations where it can create huge headaches (e.g., where the type structures of a library, etc. are poorly organized or unclear).

Re: Owl – OCaml Scientific Computing

#47

Post by ddragon »

seanmcdirmid wrote: post #26
pjmlp wrote: post #17

Earlier quoted context omitted.

Although Julia is a dynamic language, the way it uses type inference and type annotations, you can also achieve a similar experience. Naturally OCaml benefits from almost 20 years existence.

How does Julia’s dynamic typing augment discoverability? Does it magically tie into a code completion engine somehow?

It's not Julia's dynamic typing, but Julia's multimethod approach. For each function you don't have one set of arguments that must be generic enough to cover all cases (for example f(number, number)), but you have one visible implementation for each combination of types (f(integer, complex), f(complex, integer), f(T, T) where T Though while Julia has even sum and product types, it's not common for people to go to the level of detail of an ML language (plus you can't dispatch on the values of a struct), and there is no formal way to annotate an implicit interface for now, so each method will usually not be as clear from types alone as Ocaml.

Re: Owl – OCaml Scientific Computing

#48

Post by metta2uall »

srean wrote: post #15
krapht wrote: post #2

Kinda wish they had gone with F#. Would've been less work for them to write nice functional wrappers around existing .NET libraries. Of course, you can't get funding for something that unambitious, so I understand the tack. Still... I look forward to trying this out once more significant work has been done. I'm a big fan of OCaml-like languages, if that wasn't already obvious.

> around existing .NET libraries. Which ones ? A free one analogous to libmath that works with float32 would be a good start. Am serious, would love to know of some .Net libraries for number crunching that one would miss if one were to move out of the .Net platform. Or did you mean non-numeric libraries ?

What about Math.NET?

Re: Owl – OCaml Scientific Computing

#49

Post by ddragon »

o09rdk wrote: post #46
seanmcdirmid wrote: post #26

Earlier quoted context omitted.

How does Julia’s dynamic typing augment discoverability? Does it magically tie into a code completion engine somehow?

My understanding/experience is that Julia has optional typing. Meaning it's dynamically typed, but supports type annotation that often improves performance and can be used to enforce types (I think). A lot of Julia code looks statically typed, but if you want to code "pure" dynamically (e.g., for prototyping or just because it's more convenient or better for whatever reason) in style you can. Type annotation is seen…

Actually, type annotations in Julia do not improve performance (and in some pathological cases can even reduce performance). The Julia JIT compiler will always infer the type at compile-time regardless of annotation and will (almost) always produce the optimal code.

The reason for type annotations are for the multiple dispatch (multimethods), documentation, to deliberately restrict the polymorphism of a function and for the rare times when the compiler will not be able to infer the best type.

Re: Owl – OCaml Scientific Computing

#50

Post by BeetleB »

ernst_klim wrote: post #43
chewbaxxa wrote: post #41

Earlier quoted context omitted.

Sorry, I meant type inference. In python you could import all the data and deal with types later. Things that aren’t the expected type can be dealt with individually. It’s not pretty but it’s fast and it works. Have you experienced any problems with static typing in these situations? I appreciate the value of static typing but I’m not sure if it offers substantial benefit when working interactively with data. “The fl…

> In python you could import all the data and deal with types later. You can't, you need to know what you are parsing, a number, a complex number, a symbol etc. > deal with types later What does it mean? Dynamic typed language is still typed, all the expressions have types. Just like in Python, you can define types in-place with polymorphic variants and objects, so OCaml would infer their types. let instant_complex =…

>You can't, you need to know what you are parsing, a number, a complex number, a symbol etc.

pandas (Python) has the upper hand here. A lot, if not most, real world data will have values of the wrong type interspersed. pandas will still let you read in the table and then deal with these problematic values. For example, reading in the data and then dropping all values that don't conform to the type that is expected could likely be done in 2-3 lines.

But the advantage GP may be speaking of is that you can still do a lot of useful stuff with the data even if you leave the bad values in there.

For all its warts, pandas really is amazing.

Post reply on HN