Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Pydb – a lightweight database with Python syntax queries, using ZeroMQ

github.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Show HN: Pydb – a lightweight database with Python syntax queries, using ZeroMQ

#21

Post by daenney »

asrp wrote: post #15
daenney wrote: post #10

What's the purpose/gain of layering ZMQ into this? I read the architecture bit but I'm still unclear as to what benefit this brings. I guess it allows for multiple clients to use the database at the same time? I can see how the queueing thing is useful for writes if you don't want to have to handle more than one at the same time for the sake of complexity, but wouldn't doing this for reading slow things down unnecess…

As you correctly observed, I mainly used ZeroMQ for the fan-in, so I need only consider one request at a time without worrying about chunking, disconnection or other lower (socket) level issues. For speed, the idea is that you could potentially have multiple read-only servers answering queries simultaneous (all taking from the dealer). This isn't fleshed out yet. It possibly involves splitting requests into two queue…

Thanks for the reply, that clears things up!

> I'd be interested in hearing any info about potential slowdowns if you have them.

I figured if you have a central queue that everything needs to go through then you'd also be limited to a single read at a time. But if you can have multiple read-only secondaries then that's unlikely to be an issue.

Re: Show HN: Pydb – a lightweight database with Python syntax queries, using ZeroMQ

#24

Post by asrp »

emehrkay wrote: post #23

Will this code not cause issues? I know that you aren't modifying args or kwargs, in the _run method, but it just seems like a potential point of failure or a python anti-pattern def _run(self, func=None, args=(), kwargs={})

Yes, indeed! Thanks for pointing that out. I actually saw that when I was cleaning this up a bit for release and couldn't make up my mind.

I mean I'm not modifying args or kwargs now but if I did later, I could shoot myself in the foot in a not so obvious way. But on the other hand, I don't know a succinct way to express these default values. I'd probably go with `args=None, kwargs=None` and then `args = args if args else ()`.

Re: Show HN: Pydb – a lightweight database with Python syntax queries, using ZeroMQ

#25

Post by asrp »

ThePhysicist wrote: post #12

I wrote a similar library -BlitzDB- a while ago: https://github.com/adewes/blitzdb It's a pure Python database engine with a MongoDB-like query engine and support for three different backends: File (native), SQL (via SQLAlchemy) and MongoDB. The library transparently translates a large number of MongoDB queries into SQL or its own native storage backend, and when using the SQL backend it can do things that MongoDB ca…

Interesting! I will have to take a deeper look at BlitzDB. I don't know much about MongoDB (so probably wouldn't be a good maintainer) but the sample code looks good. I can't tell how it handles nesting though.

Re: Show HN: Pydb – a lightweight database with Python syntax queries, using ZeroMQ

#26

Post by emehrkay »

asrp wrote: post #24
emehrkay wrote: post #23

Will this code not cause issues? I know that you aren't modifying args or kwargs, in the _run method, but it just seems like a potential point of failure or a python anti-pattern def _run(self, func=None, args=(), kwargs={})

Yes, indeed! Thanks for pointing that out. I actually saw that when I was cleaning this up a bit for release and couldn't make up my mind. I mean I'm not modifying args or kwargs now but if I did later, I could shoot myself in the foot in a not so obvious way. But on the other hand, I don't know a succinct way to express these default values. I'd probably go with `args=None, kwargs=None` and then `args = args if args…

I typically do, but i don't know if it is the most "pythonic"

   def func(list_arg=None, dict_arg=None):
       list_arg = list_arg or []
       dict_arg = dict_arg or {}

Re: Show HN: Pydb – a lightweight database with Python syntax queries, using ZeroMQ

#27

Post by asrp »

asrp wrote: post #14
goerz wrote: post #11

There's already an (albeit deprecated) debugger package called pydb ( http://bashdb.sourceforge.net/pydb/ ). It would be good to choose a different name for this, most importantly because `pip install pydb` is already taken

Thank you! I looked up databases to see if there were obvious name clashes but never thought of debuggers despite using `pdb` a lot. I'm open to suggestions for names if anyone has one. (Although I guess it'd be nice if the top discussion thread wasn't about naming.) EDIT: I just saw you opened an issue on github so I'm happy to take name suggestions there.

I've just changed the project's name to pyzdb. Thanks again!

Re: Show HN: Pydb – a lightweight database with Python syntax queries, using ZeroMQ

#28

Post by moreati »

emehrkay wrote: post #26
asrp wrote: post #24

Earlier quoted context omitted.

Yes, indeed! Thanks for pointing that out. I actually saw that when I was cleaning this up a bit for release and couldn't make up my mind. I mean I'm not modifying args or kwargs now but if I did later, I could shoot myself in the foot in a not so obvious way. But on the other hand, I don't know a succinct way to express these default values. I'd probably go with `args=None, kwargs=None` and then `args = args if args…

I typically do, but i don't know if it is the most "pythonic" def func(list_arg=None, dict_arg=None): list_arg = list_arg or [] dict_arg = dict_arg or {}

It's not ideal. For instance if I as the caller wished to provide an empty dict-like object (e.g. dict_arg=collections.OrderedDict()), then your code would silently ignore it, and use a new dict.

Instead of checking for any object that evaluates to False, you should explicitly check for None, e.g.

   def func(list_arg=None, dict_arg=None):
       if list_arg is None:
           list_arg = []
       ...

Re: Show HN: Pydb – a lightweight database with Python syntax queries, using ZeroMQ

#30

Post by [deleted] »

asrp wrote: post #15
daenney wrote: post #10

What's the purpose/gain of layering ZMQ into this? I read the architecture bit but I'm still unclear as to what benefit this brings. I guess it allows for multiple clients to use the database at the same time? I can see how the queueing thing is useful for writes if you don't want to have to handle more than one at the same time for the sake of complexity, but wouldn't doing this for reading slow things down unnecess…

As you correctly observed, I mainly used ZeroMQ for the fan-in, so I need only consider one request at a time without worrying about chunking, disconnection or other lower (socket) level issues. For speed, the idea is that you could potentially have multiple read-only servers answering queries simultaneous (all taking from the dealer). This isn't fleshed out yet. It possibly involves splitting requests into two queue…

[deleted]
Post reply on HN