Parsing Excel Spreadsheets with Swift's Codable Protocols • Hacker News Board

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#11

Post by scarejunba » Mon, Nov 12, 2018, 7:45 PM UTC

13of40 wrote: post #9

zimablue wrote: post #5

Earlier quoted context omitted.

The advice is that it should only exist in machine-readable form and the reason why Microsoft does it this way is to ensure lock-in. See also MSSQL, for which there exists no complete machine-readable spec ANYWHERE (as far as I could tell when I spent 2 days looking a few years ago).

>lock-in A company that spends millions of dollars employing technical writers to publicly document a format probably isn't conspiring to keep that format secret. Maybe the macaronis aren't the shape you wanted but you got the macaronis.

IIRC it was in response to many government sources requiring open formats (a good instinct) so just because they did it doesn’t mean they wanted to. They may have been forced to, and done the minimum as a result.

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#12

Post by amaccuish » Mon, Nov 12, 2018, 7:47 PM UTC

ergothus wrote: post #3

sheetjs wrote: post #2

First off: awesome work! Correctly reading the data from XLSX is a lot more complex than described or implemented here, mostly because Excel is so robust in reading files and there are many sloppy writers. If you're interested, there's a many-thousand page ECMA-376 specification: https://www.ecma-international.org/publications/standards/Ec... - to correctly get the first worksheet, you actually need to parse the work…

> there's a many-thousand page ECMA-376 Honestly, I'd love to see if there are organizational tips on managing (and using!) a document that large. I feel like technical writing is the closest to coding we get in plain languages, but there are still critical differences. (technical writing is trying to give instruction to a human, while coding is giving instructions to code while giving a lot more context and descript…

> Are there lessons from specs we can learn? Do the good ones have some sort of "concept" section that makes the reading of it easier? Does each subsection do that?

Look at the OpenDocument Format. Does the same as Office Open XML (the microsoft one) in only a few hundred pages.

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#13

Post by aaaaaaaaaab » Mon, Nov 12, 2018, 7:49 PM UTC

oflannabhra wrote: post #6

The Codable protocol in Swift is game-changing. Max Howell (homebrew author) recently started a series of articles[1] in which he used Codable structs as the shared data model in both the backend and frontend of his new app Canopy. Even though communication is through HTTP and JSON, he never even has to touch it. [1] - https://medium.com/@mxcl/server-side-swift-making-canopy-2ed...

Data model != request/response structs

It’s a very bad idea to intermingle the two.

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#14

Post by mpweiher » Mon, Nov 12, 2018, 8:31 PM UTC

oflannabhra wrote: post #6

The Codable protocol in Swift is game-changing. Max Howell (homebrew author) recently started a series of articles[1] in which he used Codable structs as the shared data model in both the backend and frontend of his new app Canopy. Even though communication is through HTTP and JSON, he never even has to touch it. [1] - https://medium.com/@mxcl/server-side-swift-making-canopy-2ed...

> Codable protocol in Swift is game-changing

Which is really weird, considering Objective-C had automatic "activation/passivation" from the beginning (early 80s), and it was pretty trivial to adapt similar mechanisms later.

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#15

Post by ergothus » Mon, Nov 12, 2018, 8:47 PM UTC

edrocks wrote: post #7

ergothus wrote: post #3

Earlier quoted context omitted.

> there's a many-thousand page ECMA-376 Honestly, I'd love to see if there are organizational tips on managing (and using!) a document that large. I feel like technical writing is the closest to coding we get in plain languages, but there are still critical differences. (technical writing is trying to give instruction to a human, while coding is giving instructions to code while giving a lot more context and descript…

I wrote an XLSX(spreadsheet) writer in golang a few years ago and still maintain it. I also deal with a bunch of other several hundred to thousand page docs semi regularly and the best advice I have is to make use of the index, bookmarking pages, and a ton of cmd+f searching for various keywords. I was talking with a lawyer friend the other day who confirmed a good index is really a must for long documents.

In my experience, a good index doesn't just show where any usage is, it also covers where it is significant (bolding, sometimes subcategorizing).

In the big texts (granted, for me these are almost always RPG books, but I've read/used a lot of those) that difference is essential, and the difference between the good ones (e.g. GURPS) and the bad ones (most WW books...the ones that HAVE indexes) is very noticeable and definitely impacts not just ease-of-use, but effective-ness-of-use. Being able to really get/refresh uses of different bits has a direct impact on whether I apply those concepts consistently and correctly or whether I do something that works well enough.

This definitely describes what happens when changing code too, but we don't have that same option for "significant usage". We can get every usage, usage numbers counted by file, and definitions, but not when a use is significant. We rely on tools to get those numbers, because any kind of manually supported index (of code) is doomed to failure, much as most texts fail to have indexes, or at least fail to have very useful ones.

I wonder if that "significant usage" is something we can do something about. What does that even mean? I've been looking at code linguistically a lot and I definitely can see how we can use syntax to better indicate the focus of code (vs trivial but necessary side bits), is there a way to mechanically note that? It would still require coders to write that way, but frankly I feel we need better (or at least more clear) best practices on that front anyway. If we expand our lexicon of constructs, and mechanically make use of it, it becomes testable, reliable, and still more communicative that the current "this feels good to me, therefore I declare the code 'more readable'" approach.

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#16

Post by maxdesiatov » Mon, Nov 12, 2018, 9:08 PM UTC

mpweiher wrote: post #14

oflannabhra wrote: post #6

The Codable protocol in Swift is game-changing. Max Howell (homebrew author) recently started a series of articles[1] in which he used Codable structs as the shared data model in both the backend and frontend of his new app Canopy. Even though communication is through HTTP and JSON, he never even has to touch it. [1] - https://medium.com/@mxcl/server-side-swift-making-canopy-2ed...

> Codable protocol in Swift is game-changing Which is really weird, considering Objective-C had automatic "activation/passivation" from the beginning (early 80s), and it was pretty trivial to adapt similar mechanisms later.

Except that in Objective-C it's a runtime feature, which is by definition slower. Swift's Codable implementation is generated by the compiler or is hand-written with an obvious benefit of a stronger type system.

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#17

Post by conradev » Mon, Nov 12, 2018, 9:29 PM UTC

The one thing I don't love about Swift's Codable is the lack of customizability in the "magic" part: the part where the compiler generates the Encodable/Decodable implementations. Most notably, the compiler can't generate implementations for enums. The only thing that Swift supports customizing without fully implementing the methods for Encodable and Decodable is the name of the keys, using a custom CodingKeys type.

Serde, an equivalent third-party crate in Rust, supports a lot of customization which I find invaluable. It can (de)serialize values like this with ease:

    {"type": "location", "value": {"latitude": 0, "longitude": 0}}

in a very small amount of code:

    #[derive(Serialize, Deserialize)]
    #[serde(tag = "type", content = "value")]
    #[serde(rename_all = "lowercase")]
    enum Value {
        Location { latitude: f64, longitude: f64 },
        String(String),
        ...
    }

Serde also supports customizing serialization on a per-field basis without having to implement the entire protocol, which is nice:

    #[derive(Deserialize, Serialize)]
    struct Record {
        #[serde(with = "chrono::serde::ts_seconds")]
        updated: DateTime
    }

I really hope that Swift has better ways (like the above) to customize Codable in the future. I find myself implementing the protocol myself in 90% of cases, whereas I very rarely have to do that for Serde.

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#18

Post by tzs » Mon, Nov 12, 2018, 10:26 PM UTC

amaccuish wrote: post #12

ergothus wrote: post #3

Earlier quoted context omitted.

> there's a many-thousand page ECMA-376 Honestly, I'd love to see if there are organizational tips on managing (and using!) a document that large. I feel like technical writing is the closest to coding we get in plain languages, but there are still critical differences. (technical writing is trying to give instruction to a human, while coding is giving instructions to code while giving a lot more context and descript…

> Are there lessons from specs we can learn? Do the good ones have some sort of "concept" section that makes the reading of it easier? Does each subsection do that? Look at the OpenDocument Format. Does the same as Office Open XML (the microsoft one) in only a few hundred pages.

ODF is a bit more than a few hundred pages. Here's the page counts for ODF 1.2:

  102 pages, OpenDocument-v1.2-os.pdf
  846 pages, OpenDocument-v1.2-os-part1.pdf
  234 pages, OpenDocument-v1.2-os-part2.pdf
   35 pages, OpenDocument-v1.2-os-part3.pdf

That's 1217 pages.

It is quite a bit smaller than OOXML. OOXML is about 5-6 times the size of ODF. There are three reasons for this.

1. ODF has cleaner and leaner markup. OOXML is uglier and more verbose, leading to more things that the spec has to document..

2. The OOXML spec goes into more detail for a lot of things that the two have in common.

3. The OOXML spec seems to have a lot more introductory or primer material.

The impression I got from looking at both, but not actually trying to use either of them, is that if I had to implement a full featured office suite solely from the spec, with no reference implementations or example documents, it would be easier if I picked ODF, but would probably have better interoperability if I picked OOXML. The strongest impression I got, from both of them, though, is that there is no freaking way I want to implement an office suite!

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#19

Post by ken » Mon, Nov 12, 2018, 10:31 PM UTC

ergothus wrote: post #3

sheetjs wrote: post #2

First off: awesome work! Correctly reading the data from XLSX is a lot more complex than described or implemented here, mostly because Excel is so robust in reading files and there are many sloppy writers. If you're interested, there's a many-thousand page ECMA-376 specification: https://www.ecma-international.org/publications/standards/Ec... - to correctly get the first worksheet, you actually need to parse the work…

> there's a many-thousand page ECMA-376 Honestly, I'd love to see if there are organizational tips on managing (and using!) a document that large. I feel like technical writing is the closest to coding we get in plain languages, but there are still critical differences. (technical writing is trying to give instruction to a human, while coding is giving instructions to code while giving a lot more context and descript…

When I worked at Boeing, technical specs were built in a giant object/hierarchical database. There would be objects for Requirements, and in the design you could link Systems to the Requirements that they implemented. Then you could trace back from the design to see what requirements weren't implemented, or what systems had no apparent purpose, or who edited a particular requirement last.

When they needed a hard copy of The Spec, they could export the hierarchy of requirements (automatically numbered, of course) as HTML or MSWord or whatever.

It was, of course, a giant nearly-unreadable mess, like "Req 1.2.3.4.5: The Foo system shall have a Bar module.", and then 25 more sub-requirements that all start out with "The Foo module's Bar subsystem shall ___" -- and made no sense unless you had just read the previous 30 pages, anyway.

It was not a good system, but it had hints of a good system in it. I think it was a good concept but the UI was terrible and people didn't seem to take much care when working on it.

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#20

Post by maxdesiatov » Mon, Nov 12, 2018, 11:00 PM UTC

conradev wrote: post #17

The one thing I don't love about Swift's Codable is the lack of customizability in the "magic" part: the part where the compiler generates the Encodable/Decodable implementations. Most notably, the compiler can't generate implementations for enums. The only thing that Swift supports customizing without fully implementing the methods for Encodable and Decodable is the name of the keys, using a custom CodingKeys type.…

The main reason for that is lack of hygienic macros in Swift. Currently you can use code generation tools like Sourcery and SwiftGen, but I expect 1st-class meta-programming support to come after Swift 5.0 release. After ABI stability I imagine macros are pretty high on the priority list of the core team.