Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use JSON files as if they are Python modules

github.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Use JSON files as if they are Python modules

#51

Post by alexwebb2 »

aikah wrote: post #14
BinaryIdiot wrote: post #6

As a JavaScript developer for the past 5 years or so, when jumping into Python I certainly miss the simple require( ) which this seems to replicate into Python pretty well! Having said that I don't think that I would use it for serious things since this isn't really the Python way and I would like my code to be most understood by others. Neat though!

JSON stands for JavaScript Object Notation , not Python Object Notation. Furthermore "require" is "proprietary" to nodejs, ES2015 has a different module system and it will be the official js module system.

The require() function was actually part of the old CommonJS standard: http://wiki.commonjs.org/wiki/Modules/1.1

node was originally an implementation of it before eclipsing it. Between the original advocacy by CommonJS and the rising popularity of node, browserify, and Webpack, things like require() leaked out to become the pre-ES2015 de facto standard for importing modules.

require() is so widespread now, and the transform to the ES2015 syntax so trivial, that it's not going away anytime soon.

Re: Use JSON files as if they are Python modules

#52

Post by dr_zoidberg »

orf wrote: post #34
jamesdutc wrote: post #24

The mechanism is called "import hooks": https://docs.python.org/3/reference/import.html#import-hooks https://www.python.org/dev/peps/pep-0302/ You can see the import hook added here: https://github.com/kragniz/json-sempai/blob/master/jsonsempa... Import hooks are a great feature in Python. Take the `import` mechanism in the language and reduce it to its barest theoretical formulation - what does it really do? (Name b…

Import hooks are awesome, one of the cool things you can do is re-write the code as you load it. I made a toy loader that automagically inlines (some) Python function calls[1]. There are also cool projects like MacroPy[2] that do much more extensive things. 1. https://tomforb.es/automatically-inline-python-function-call... 2. https://github.com/lihaoyi/macropy

I read your experiment on inlining and it looks quite interesting. Did you do a benchmark on how it affects (improves?) performance? In my line of work I've found ocasionally places where inlining would've helped (functions that do masking with 64 bit masks, for example, are a mess to "inline by hand" and kill a lot of readability and clarity). Even with the current limitations of your "toy" implementation, it seems like it would help to avoid the costly function calls.

Re: Use JSON files as if they are Python modules

#53

Post by StavrosK »

Excepting the cringemarsh that is "import ", this is actually very useful. This looks very unpythonic:

my_json["this"]["can"]["be"] == "nested"

and is impossible to write defensively and cleanly after the first call:

my_json.get("this", {}).get("no way to not crash here")

Something that could do:

my_json.this.can.be == "nested"

With a default value would be very useful. Does anyone know something like it, or should I whip a module up in an hour?

Re: Use JSON files as if they are Python modules

#54

Post by dadrian »

StavrosK wrote: post #53

Excepting the cringemarsh that is "import ", this is actually very useful. This looks very unpythonic: my_json["this"]["can"]["be"] == "nested" and is impossible to write defensively and cleanly after the first call: my_json.get("this", {}).get("no way to not crash here") Something that could do: my_json.this.can.be == "nested" With a default value would be very useful. Does anyone know something like it, or should I…

I've taken a crack at something similar, I'd love to see what you came up with. I ended up with this super-meh solution: https://github.com/zmap/ztag/blob/master/ztag/transform.py#L...

Re: Use JSON files as if they are Python modules

#55

Post by bdarnell »

jamesdutc wrote: post #24

The mechanism is called "import hooks": https://docs.python.org/3/reference/import.html#import-hooks https://www.python.org/dev/peps/pep-0302/ You can see the import hook added here: https://github.com/kragniz/json-sempai/blob/master/jsonsempa... Import hooks are a great feature in Python. Take the `import` mechanism in the language and reduce it to its barest theoretical formulation - what does it really do? (Name b…

The "loader protocol" (PEP 302) is a related trick that can be used locally, so it only affects imports from packages that opt in. I use this in https://github.com/bdarnell/codegenloader to automagically process protobuf/thrift files and import the generated code.

Re: Use JSON files as if they are Python modules

#56

Post by jimbokun »

orf wrote: post #34
jamesdutc wrote: post #24

The mechanism is called "import hooks": https://docs.python.org/3/reference/import.html#import-hooks https://www.python.org/dev/peps/pep-0302/ You can see the import hook added here: https://github.com/kragniz/json-sempai/blob/master/jsonsempa... Import hooks are a great feature in Python. Take the `import` mechanism in the language and reduce it to its barest theoretical formulation - what does it really do? (Name b…

Import hooks are awesome, one of the cool things you can do is re-write the code as you load it. I made a toy loader that automagically inlines (some) Python function calls[1]. There are also cool projects like MacroPy[2] that do much more extensive things. 1. https://tomforb.es/automatically-inline-python-function-call... 2. https://github.com/lihaoyi/macropy

Sounds a bit like the Ruby-fication of Python.

Re: Use JSON files as if they are Python modules

#57

Post by gh02t »

sametmax wrote: post #33
gh02t wrote: post #25

Earlier quoted context omitted.

Does it respect import paths: no. As for whether or not json-sempai does, I looked through the code and I am pretty sure the answer is yes. Declaring pkg_resources is probably a better way to include static data files in your package but hey radical freedom and all that.

Do you have any good tutorial on how to use pkg_resources ? I use pathlib.Path(__file__).absolute().parent but it's not zip safe. Problem is, I can't wrap my head around pkg_resources and the docs don't help.

Yeah, docs are a bit fuzzy and I'm not exactly sure what the right answer is.

So there are actually two ways to go about it AFAIK and I'm not entirely sure which one is the best. One is to list files in MANIFEST.in and then access it via __file__, but I think this doesn't work if your module gets turned into an egg/wheel (it's for source dists).

On the other hand is using pkg_resources, where you declare the list of files in the package_data argument to setup in your setup.py. You can then get the contents of that file (not its path) by importing the pkg_resources package, which provides a lookup table of loaded resources. This [apparently] works for binary dists but not source dists (the opposite of MANIFEST).

I've used pkg_resources without issue myself, but it's possible I just have been lucky. In any event, here's a few links discussing the differences and also how to use pkg_resources:

http://stackoverflow.com/questions/7522250/how-to-include-pa...

http://blog.codekills.net/2011/07/15/lies,-more-lies-and-pyt...

http://peak.telecommunity.com/DevCenter/PythonEggs#accessing...

If you want an example, here's a package I made that uses pkg_resources (I make no promises that this is the right way to do it, but it works for me):

(setup.py, see line 10) https://github.com/jasonmhite/gefry2/blob/master/setup.py

(the data file I want to include) https://github.com/jasonmhite/gefry2/tree/master/gefry2/data

(where the code loads the data file, see line 67) https://github.com/jasonmhite/gefry2/blob/master/gefry2/mate...

Re: Use JSON files as if they are Python modules

#58

Post by njharman »

amelius wrote: post #49

The enormous downside to this approach is that once you want to load a json file dynamically, e.g. with a filename that comes from a dynamic string, you're stuck and you have to come up with a completely different approach and rewrite your code.

Not correct.

Like many things in Python import is actually just syntactic sugar for a dunder function.

https://docs.python.org/3/library/functions.html#__import__

Re: Use JSON files as if they are Python modules

#59

Post by StavrosK »

dadrian wrote: post #54
StavrosK wrote: post #53

Excepting the cringemarsh that is "import ", this is actually very useful. This looks very unpythonic: my_json["this"]["can"]["be"] == "nested" and is impossible to write defensively and cleanly after the first call: my_json.get("this", {}).get("no way to not crash here") Something that could do: my_json.this.can.be == "nested" With a default value would be very useful. Does anyone know something like it, or should I…

I've taken a crack at something similar, I'd love to see what you came up with. I ended up with this super-meh solution: https://github.com/zmap/ztag/blob/master/ztag/transform.py#L...

What I had in mind sounds pretty close to yours, really. I was thinking of adding a default somewhere along the line (possibly the end), so you could do:

json.structure.that.i.want.to.access.default(True)

Or something like that. I don't really like the default item API I've come up... I'll comment here if I do give it a go, thanks for the snippet!

Re: Use JSON files as if they are Python modules

#60

Post by cjslep »

sarciszewski wrote: post #31
jstoiko wrote: post #8

My favorite is this issue: https://github.com/kragniz/json-sempai/issues/7 > Ever since I saw this I have been unable to sleep. Please fix.

> Works as intended. adds the 'wontfix' label This is gold.

The 'wontfix' label looks more like a yellow than a gold to me.
Post reply on HN