Parsing Excel Spreadsheets with Swift's Codable Protocols • Hacker News Board

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#21

Post by conradev » Mon, Nov 12, 2018, 11:42 PM UTC

maxdesiatov wrote: post #20

conradev wrote: post #17

The one thing I don't love about Swift's Codable is the lack of customizability in the "magic" part: the part where the compiler generates the Encodable/Decodable implementations. Most notably, the compiler can't generate implementations for enums. The only thing that Swift supports customizing without fully implementing the methods for Encodable and Decodable is the name of the keys, using a custom CodingKeys type.…

The main reason for that is lack of hygienic macros in Swift. Currently you can use code generation tools like Sourcery and SwiftGen, but I expect 1st-class meta-programming support to come after Swift 5.0 release. After ABI stability I imagine macros are pretty high on the priority list of the core team.

They might also go in the opposite direction and add runtime APIs to construct types dynamically. Various initiatives (like the Python interop for TensorFlow) are pulling Swift in that direction:

https://github.com/apple/swift-evolution/blob/master/proposa...

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#22

Post by slavapestov » Tue, Nov 13, 2018, 12:08 AM UTC

maxdesiatov wrote: post #20

conradev wrote: post #17

The one thing I don't love about Swift's Codable is the lack of customizability in the "magic" part: the part where the compiler generates the Encodable/Decodable implementations. Most notably, the compiler can't generate implementations for enums. The only thing that Swift supports customizing without fully implementing the methods for Encodable and Decodable is the name of the keys, using a custom CodingKeys type.…

The main reason for that is lack of hygienic macros in Swift. Currently you can use code generation tools like Sourcery and SwiftGen, but I expect 1st-class meta-programming support to come after Swift 5.0 release. After ABI stability I imagine macros are pretty high on the priority list of the core team.

I have a hard time envisioning full-fledged macros coming to Swift, but moving Equatable/Hashable/Codable synthesis to a library should be possible with more restricted meta-programming capabilites. See Joe Groff's talk for details: https://www.skilled.io/u/swiftsummit/swift-s-reflective-unde...

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#23

Post by wild_preference » Tue, Nov 13, 2018, 2:14 AM UTC

conradev wrote: post #17

The one thing I don't love about Swift's Codable is the lack of customizability in the "magic" part: the part where the compiler generates the Encodable/Decodable implementations. Most notably, the compiler can't generate implementations for enums. The only thing that Swift supports customizing without fully implementing the methods for Encodable and Decodable is the name of the keys, using a custom CodingKeys type.…

Serde offers more out of the box functionality with its annotations, but it's also bit uglier than Swift's Codable when you do need to implement it yourself.

If it wasn't for a blog post that gave some simple examples, I couldn't even figure it out myself.

Possibly not fair to compare them this way, though. No clue what the trade-offs are.

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#24

Post by userbinator » Tue, Nov 13, 2018, 2:33 AM UTC

scarejunba wrote: post #11

13of40 wrote: post #9

Earlier quoted context omitted.

>lock-in A company that spends millions of dollars employing technical writers to publicly document a format probably isn't conspiring to keep that format secret. Maybe the macaronis aren't the shape you wanted but you got the macaronis.

IIRC it was in response to many government sources requiring open formats (a good instinct) so just because they did it doesn’t mean they wanted to. They may have been forced to, and done the minimum as a result.

I believe it was the antitrust lawsuits that forced them too, and it shows; if you look at all the MS protocol/format documents and compare them to something like RFCs and ITU/IEEE/ANSI standards, the MS docs are noticeably harder to read with their verbosity, weird syntax notations and conventions, and almost look as if they were deliberately obfuscated.

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#25

Post by Serow225 » Tue, Nov 13, 2018, 3:41 AM UTC

ken wrote: post #19

ergothus wrote: post #3

Earlier quoted context omitted.

> there's a many-thousand page ECMA-376 Honestly, I'd love to see if there are organizational tips on managing (and using!) a document that large. I feel like technical writing is the closest to coding we get in plain languages, but there are still critical differences. (technical writing is trying to give instruction to a human, while coding is giving instructions to code while giving a lot more context and descript…

When I worked at Boeing, technical specs were built in a giant object/hierarchical database. There would be objects for Requirements, and in the design you could link Systems to the Requirements that they implemented. Then you could trace back from the design to see what requirements weren't implemented, or what systems had no apparent purpose, or who edited a particular requirement last. When they needed a hard copy…

DOORS ?

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#26

Post by saagarjha » Tue, Nov 13, 2018, 4:49 AM UTC

slavapestov wrote: post #22

maxdesiatov wrote: post #20

Earlier quoted context omitted.

The main reason for that is lack of hygienic macros in Swift. Currently you can use code generation tools like Sourcery and SwiftGen, but I expect 1st-class meta-programming support to come after Swift 5.0 release. After ABI stability I imagine macros are pretty high on the priority list of the core team.

I have a hard time envisioning full-fledged macros coming to Swift, but moving Equatable/Hashable/Codable synthesis to a library should be possible with more restricted meta-programming capabilites. See Joe Groff's talk for details: https://www.skilled.io/u/swiftsummit/swift-s-reflective-unde...

> I have a hard time envisioning full-fledged macros coming to Swift, but moving Equatable/Hashable/Codable synthesis to a library should be possible with more restricted meta-programming capabilites.

I've heard many people on the mailing lists talk about wanting to add hygienic macros–so when you're talking about "full-fledged macros", are you talking about unhygienic or hygienic ones?

> See Joe Groff's talk for details

I'm actually curious how much is possible, in terms of reflection capability, both currently and in the future. How close can I get to the dynamism of say the Objective-C runtime? Can I grab a function pointer knowing its mangled name? Can I list every class in a binary by consulting the runtime metadata? Can I "swizzle"?

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#27

Post by marcruser » Tue, Nov 13, 2018, 4:55 AM UTC

sheetjs wrote: post #2

First off: awesome work! Correctly reading the data from XLSX is a lot more complex than described or implemented here, mostly because Excel is so robust in reading files and there are many sloppy writers. If you're interested, there's a many-thousand page ECMA-376 specification: https://www.ecma-international.org/publications/standards/Ec... - to correctly get the first worksheet, you actually need to parse the work…

I laughed when I started reading this comment and then looked at your username. I've used your "js-xlsx" library and stepped through quite a bit of the code. I still can't really understand how you begun to write that library, and I'm curious how you approach reading the Open XML documentation. Do you have a large team of engineers maintaining that codebase?

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#28

Post by slavapestov » Tue, Nov 13, 2018, 5:13 AM UTC

saagarjha wrote: post #26

slavapestov wrote: post #22

Earlier quoted context omitted.

I have a hard time envisioning full-fledged macros coming to Swift, but moving Equatable/Hashable/Codable synthesis to a library should be possible with more restricted meta-programming capabilites. See Joe Groff's talk for details: https://www.skilled.io/u/swiftsummit/swift-s-reflective-unde...

> I have a hard time envisioning full-fledged macros coming to Swift, but moving Equatable/Hashable/Codable synthesis to a library should be possible with more restricted meta-programming capabilites. I've heard many people on the mailing lists talk about wanting to add hygienic macros–so when you're talking about "full-fledged macros", are you talking about unhygienic or hygienic ones? > See Joe Groff's talk for det…

> How close can I get to the dynamism of say the Objective-C runtime?

Well, right now the compiler emits a lot of metadata that is mostly used for runtime generics, dynamic casting, and the (somewhat limited) Mirror type. But these features have enough generality that a surprising amount of stuff has to be encoded. However there’s no nice API for looking at it yet.

> Can I grab a function pointer knowing its mangled name?

That’s just dlsym().

> Can I list every class in a binary by consulting the runtime metadata?

The metadata is there, but there’s no exposed API for doing this.

> Can I "swizzle"?

There’s an experimental thing for this now: https://github.com/apple/swift/pull/20333

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#29

Post by saagarjha » Tue, Nov 13, 2018, 5:33 AM UTC

slavapestov wrote: post #28

saagarjha wrote: post #26

Earlier quoted context omitted.

> I have a hard time envisioning full-fledged macros coming to Swift, but moving Equatable/Hashable/Codable synthesis to a library should be possible with more restricted meta-programming capabilites. I've heard many people on the mailing lists talk about wanting to add hygienic macros–so when you're talking about "full-fledged macros", are you talking about unhygienic or hygienic ones? > See Joe Groff's talk for det…

> How close can I get to the dynamism of say the Objective-C runtime? Well, right now the compiler emits a lot of metadata that is mostly used for runtime generics, dynamic casting, and the (somewhat limited) Mirror type. But these features have enough generality that a surprising amount of stuff has to be encoded. However there’s no nice API for looking at it yet. > Can I grab a function pointer knowing its mangled…

> > Can I grab a function pointer knowing its mangled name?

> That’s just dlsym().

Sorry, I should have been more clear: can I grab a function pointer to an unexported function?

> > Can I "swizzle"?

> There’s an experimental thing for this now: https://github.com/apple/swift/pull/20333

This is interesting; I haven't been really been following the lists recently but I took the time to read the linked thread and the pull request. While the functionality contained in the pull request is interesting, it's not quite the same as swizzling since it's done unconditionally at load time rather than runtime (so it's much more similar to DYLD_INTERPOSE in that sense). While this covers many of the cases when swizzling is necessary, it leaves out a rather important one where the correct method override is selected at runtime, generally conditionally.

Also, as a sidenote, it seems that there is some sort of motivation to have compiler type checks for the replacement, i.e. @_dynamicReplacement(for: bar())–how will this actually work in practice? If I compile a bundle without access to the source of the application I'm going to be loaded into, how would bar() be accessible to the compiler at all?

Re: Parsing Excel Spreadsheets with Swift's Codable Protocols

#30

Post by conradev » Wed, Nov 14, 2018, 3:18 AM UTC

wild_preference wrote: post #23

conradev wrote: post #17

The one thing I don't love about Swift's Codable is the lack of customizability in the "magic" part: the part where the compiler generates the Encodable/Decodable implementations. Most notably, the compiler can't generate implementations for enums. The only thing that Swift supports customizing without fully implementing the methods for Encodable and Decodable is the name of the keys, using a custom CodingKeys type.…

Serde offers more out of the box functionality with its annotations, but it's also bit uglier than Swift's Codable when you do need to implement it yourself. If it wasn't for a blog post that gave some simple examples, I couldn't even figure it out myself. Possibly not fair to compare them this way, though. No clue what the trade-offs are.

That's very fair criticism. Implementing custom serde serializers is painful, but thankfully I only have to do it in rare circumstances.