Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Schema for Clojure(Script) Data Shape Declaration and Validation

blog.getprismatic.com

Post reply on HN
Print view View original

31–40 of 58 posts

Re: Schema for Clojure(Script) Data Shape Declaration and Validation

#31

Post by freshhawk »

danneu wrote: post #23
lkrubner wrote: post #21

Earlier quoted context omitted.

I do this a lot: "Often I stoop to the level of caching my findings in a docstring/comment." I am not ashamed of this. When my co-workers have to read what I wrote, this kind of information makes clear what the function does. If I don't do this, then they will eventually figure it out by running it at the REPL and examining the input and the output, but if I document that in the docstring, I have saved them a lot of…

Yeah, you're right. I would love if every Clojure function came with an example input and output. What I meant to suggest is that my docstring example is something that could be expressed in code.

A friend of mine has been doing similar things with the Swagger documentation spec (https://developers.helloreverb.com/swagger/)

While browsing the docs in a web browser you can fill out some fields for input and see what the output for that input is.

My friend forked it and extracts and lists any example data so you can click to load that example into those fields (my idea!). I tried it out and it's a very nice way to learn/explore a new API, especially testing edge cases where the docs are ambiguous.

It really hits a sweet spot when you have documentation browsing, live execution and example data. I think creating a similar interface combining docstrings, the repl and example data from test fixtures would be a nice tool to have (I find doctests to be one of those things that are better in theory than in practice as they get too long and maintenance is annoying, although perhaps a more literate style in your source code would change that).

Re: Schema for Clojure(Script) Data Shape Declaration and Validation

#33

Post by w01fe »

jamii wrote: post #27

I've been working on a similar idea for replacing types. If you can specify how to take apart a type, name the parts and put them back together you can then get validators, pattern matching, recursive-descent parsing with backtracking, generic traversals, lenses and generators more or less for free. By using simple data-structures to describe the types and compiling them to clojure at the call site you can have first…

Nice! Really looking forward to the release!

Re: Schema for Clojure(Script) Data Shape Declaration and Validation

#35

Post by naiquevin »

krosaen wrote: post #32

pretty cool - I've used something similar for python JSON validation: https://code.google.com/p/jsonvalidator/ but love the idea of something for validating nested data structures as well.

I had come across another schema validation lib for Python on Github a while back (but never tried/used it):

https://github.com/halst/schema

Re: Schema for Clojure(Script) Data Shape Declaration and Validation

#36

Post by w01fe »

jared314 wrote: post #34

This looks like it occupies the same space as core.contracts and core.typed. Is the main benefit, over those two, cljs support?

I think there are a few benefits.

First and foremost, schemas are simple, minimal, easy to read and write, and gracefully extend Clojure's existing type hints. This means that (in my biased opinion) they are significantly better for documentation, which was the primary motivation for developing them.

Second, schemas are data, so it's easy to do more with them beyond documentation. Runtime data validation is one such use, but we can also easily do things like generate core.typed annotations, generate model classes for clients, generate test data, and so on.

Re: Schema for Clojure(Script) Data Shape Declaration and Validation

#37

Post by jwr »

w01fe wrote: post #16
jwr wrote: post #13

Earlier quoted context omitted.

This is a very good idea. And the reason I say that is because in our code base we have developed something similar, feeling the need for it :-). Our stuff is half-baked, though and I never got around to releasing it. What I'd like to see is something similar to what we did — checking more than just types. Here's an example from our code: (conforms example-object [map :from string? :to [map :required-keys {"products"…

Thanks for the feedback! Actually, schema can express arbitrary constraints. Your example translates to schema as: {String {(s/required-key "product") [String] (s/required-key "type") (s/enum "type1" "type2") s/Any s/Any}} ;; allow any other k-v pairs

Interesting! Is the enum automatically a vector/sequence, or is that just an omission in the example?

Looks like I'll be using this library sooner than I thought, thanks!

Re: Schema for Clojure(Script) Data Shape Declaration and Validation

#38

Post by ambrosebs »

w01fe wrote: post #36
jared314 wrote: post #34

This looks like it occupies the same space as core.contracts and core.typed. Is the main benefit, over those two, cljs support?

I think there are a few benefits. First and foremost, schemas are simple , minimal, easy to read and write, and gracefully extend Clojure's existing type hints. This means that (in my biased opinion) they are significantly better for documentation, which was the primary motivation for developing them. Second, schemas are data, so it's easy to do more with them beyond documentation. Runtime data validation is one such…

Author of core.typed here.

I will challenge that schemas are "significantly" better for documentation than core.typed types.

With unions, intersections, heterogeneous maps, parameterised classes, recursive types we can be very expressive.

Here's some examples of the syntax for types: https://github.com/clojure/core.typed/wiki/Types

And some declarative types in action: https://github.com/frenchy64/core.typed-example/blob/master/...

I think core.typed is restrictive enough to allow annotations like the one above, while being opinionated enough to guide the programmer to write clear, comprehensible types.

Re: Schema for Clojure(Script) Data Shape Declaration and Validation

#39

Post by ambrosebs »

filipncs wrote: post #3
aria wrote: post #2

Author here. Happy to answer any questions or comments.

You mention generating core.typed annotations from schemas, to allow for some compile-time checking. I would have thought that this would be extremly hard to automate for non-trivial schemas, or at least that core.typed would have a hard time proving that return values match a schema. Is this only for a subset?

core.typed will have to be extended to understand schemas. core.typed already plays nicely with assertions and branches (via occurrence typing), so we just need information on what type a schema "casts" to. This will not be difficult, it might not even have to live in core.typed's implementation and be extended via multimethods.

One of the problems with gradually annotating an untyped namespace with core.typed is we immediately have to be very accurate with our type annotations.

However if the namespace is completely "schema'ed", we can assume functions simply take `Any` as arguments and rely on Schema to regain type information.

This could be a very nice way to work.

Re: Schema for Clojure(Script) Data Shape Declaration and Validation

#40

Post by ambrosebs »

aria wrote: post #11
drcode wrote: post #7

Earlier quoted context omitted.

Hey, I like what you're doing here, I think I'll use this... it gives you a lot of the benefits of a deep type system while being completely orthogonal to the design of the program and without the cognitive overhead of a complex type system. Also, functional programming is heavily focused on data transformations, which in practice means lots of deeply nested heterogenous data structures... these types of structure ar…

Thanks! Yeah I think a traditional type system wouldn't make sense for Clojure, you need something that can describe data structures more succinctly, which is what we were shooting for

Just so readers don't deduce you're talking about core.typed, core.typed is plenty expressive enough to represent many idiomatic Clojure data structures, and very succinctly.

eg. Heterogeneous keyword maps https://github.com/frenchy64/core.typed-example/blob/master/...

Post reply on HN