Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: File-based cache for slow Python functions

docs.sweep.dev

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Show HN: File-based cache for slow Python functions

#41

Post by williamzeng0 »

Karellen wrote: post #35
wildermuthn wrote: post #9

If you aren’t caching LLM functions during development, then you’re an even greater glutton for punishment than the normal engineer. My local file cache Python decorator also allows the decorator to define the hash manually, either by the decorator’s parameter function call that plucks a value from the cached function params, or by calling a global function from anywhere with any arbitrary value. What’s cool about ca…

I feel like adding an argument to the decorator that labels the "version" of the function would make deliberate cache invalidation more straightforward for cache users.

The version input makes sense, I could also see some developers disliking that ux because of it's verbosity. But to deliberately invalidate you have to make a manual effort in either case.

Re: Show HN: File-based cache for slow Python functions

#42

Post by rmholt »

williamzeng0 wrote: post #32
rmholt wrote: post #6

I have extensively used https://pypi.org/project/diskcache/ . Is there a reason you decided to make an in house solution?

Diskcache works well, we just wanted a dependency free version that we had more control over (easier cache key deletion). I think you'd have to write a custom hashing function for diskcache to use the function source code as a key. I'm also unsure if Diskcache supports ignoring certain fields in the function call.

Good questions! You made me check the docs because those seem like very legitimate issues. So firstly, DiskCache by default just checks the function name, not the source code, but you could hack it to include the source code. I personally usually just deleted the cache if I knew the function meaningfully changed.

And it does support ignoring certain args yes.

Re: Show HN: File-based cache for slow Python functions

#43

Post by mpeg »

williamzeng0 wrote: post #37
mpeg wrote: post #14

I recently wrote a version of this that I use in my projects, some things I do differently that you may or may not care about: - from your code it seems you're not sorting kwargs, I would strongly recommend sorting them so that whether you call f(a=1, b=2) or f(b=2, a=1) the cache key is the same - I use inspect.signature to convert all args to kwargs, this way it doesn't matter how a function gets called, the cache…

Making the __dict__ opt-in makes it a lot more user-friendly at the expense of a little verbosity. That makes sense. These tips make sense, we often use named args in our function calls (not using them has caused so many bugs), but we don't really enforce the order. Copilot doesn't always get it right either. By moving inspect.getsource out of the wrapper, do you mean initializing it when the module is imported? I'm…

Yeah I too try to avoid positional args as much as possible, huge source of bugs and time wasting especially when refactoring code

Re inspect.getsource, I'm not sure if it'd be a huge performance impact, but if it's in the wrapper fn it will get called every time the function gets called, while if it's outside it will be called only when the decorator runs (eg when the module containing the function being decorated is imported).

eg: https://gist.github.com/mpeg/ff1d99fde06f39916b5aaadd76b534f...

EDIT: on a quick test, over 100k function calls, with inspect.getsource inside the wrapper it runs in 2.7s on my Apple M2, and that's not even including the md5 hash, so I suspect this should dramatically improve performance for you

Re: Show HN: File-based cache for slow Python functions

#44

Post by williamzeng0 »

mpeg wrote: post #43
williamzeng0 wrote: post #37

Earlier quoted context omitted.

Making the __dict__ opt-in makes it a lot more user-friendly at the expense of a little verbosity. That makes sense. These tips make sense, we often use named args in our function calls (not using them has caused so many bugs), but we don't really enforce the order. Copilot doesn't always get it right either. By moving inspect.getsource out of the wrapper, do you mean initializing it when the module is imported? I'm…

Yeah I too try to avoid positional args as much as possible, huge source of bugs and time wasting especially when refactoring code Re inspect.getsource, I'm not sure if it'd be a huge performance impact, but if it's in the wrapper fn it will get called every time the function gets called, while if it's outside it will be called only when the decorator runs (eg when the module containing the function being decorated i…

This is great, thanks for the suggestion! We've just updated our docs and code with this change: https://github.com/sweepai/sweep/pull/3332

Re: Show HN: File-based cache for slow Python functions

#45

Post by chlorion »

epr wrote: post #24

def hash_code(code): return hashlib.md5(code.encode()).hexdigest() Be warned. The above function is used as part of the hash. The ostensible purpose is to prevent using cached values of functions who's code has changed, but it does not handle dependencies of that function.

Using md5 for this seems like an odd choice.

Sha1 is a better choice even for non-cryptographic use cases, it's quite a bit faster than md5. Even better would be something like xxhash!

According to a quick bash script I wrote to benchmark the popular hash functions, md5 comes out last compared to sha1, sha256, sha512, and blake2, and by a decent margin!

A good rule of thumb is to never use md5 at all. Not even for non-cryptographic use cases. It's not only broken, but also very slow!

Re: Show HN: File-based cache for slow Python functions

#46

Post by williamzeng0 »

rmholt wrote: post #42
williamzeng0 wrote: post #32

Earlier quoted context omitted.

Diskcache works well, we just wanted a dependency free version that we had more control over (easier cache key deletion). I think you'd have to write a custom hashing function for diskcache to use the function source code as a key. I'm also unsure if Diskcache supports ignoring certain fields in the function call.

Good questions! You made me check the docs because those seem like very legitimate issues. So firstly, DiskCache by default just checks the function name, not the source code, but you could hack it to include the source code. I personally usually just deleted the cache if I knew the function meaningfully changed. And it does support ignoring certain args yes.

Thank you for checking! That works, we could just override the key logic to check the source code (I'm guessing it's easily accessible)

Re: Show HN: File-based cache for slow Python functions

#47

Post by Karellen »

williamzeng0 wrote: post #41
Karellen wrote: post #35

Earlier quoted context omitted.

I feel like adding an argument to the decorator that labels the "version" of the function would make deliberate cache invalidation more straightforward for cache users.

The version input makes sense, I could also see some developers disliking that ux because of it's verbosity. But to deliberately invalidate you have to make a manual effort in either case.

To me, it's more that invalidating the cache without it requires knowledge of implementation details (i.e. where the cache is stored, and how the cache files are named) that ideally shouldn't have to "leak" for the cache to be generally useful.

(Buy hey, we are talking about what is famously one of the hard problems in comp sci, so (respectful) disagreements on how best to do it should be expected :-)

Re: Show HN: File-based cache for slow Python functions

#48

Post by epr »

martinky24 wrote: post #27
epr wrote: post #24

def hash_code(code): return hashlib.md5(code.encode()).hexdigest() Be warned. The above function is used as part of the hash. The ostensible purpose is to prevent using cached values of functions who's code has changed, but it does not handle dependencies of that function.

How do you suggest one might fix that issue? Also pin the cache to a hash of all dependency versions? And then if one minor update And let's say the dependency did change, but it's generally inert (more error handling around edge cases, for example), how do you factor that in? Blow up the whole cache? Your example isn't really a problem with OPs utility, but a specific example of a broader dependency management probl…

> How do you suggest one might fix that issue? Also pin the cache to a hash of all dependency versions?

Pretty much. Recursively collect dependencies by analyzing the AST of the code.

> And then if one minor update And let's say the dependency did change, but it's generally inert (more error handling around edge cases, for example), how do you factor that in? Blow up the whole cache?

You're saying that like it's some kind of ridiculous ask, but yes. The current implementation is already "Blow[ing] up the whole cache" whenever the code for the decorated function is changed anyways. I'd guess that additionally handling dependencies recursively would only modestly increase the rate of "Blow[ing] up the whole cache".

> Your example isn't really a problem with OPs utility...

Whether or not this is a problem in practice obviously depends on your use case. Maybe you don't generally care if functions return the correct result, but many do.

> [This is] a specific example of a broader dependency management problem that affects just about everything.

Dependency resolution is not trivial per se, but it's a pretty common problem. Every single package manager, build system (make), etc. have all solved this.

Re: Show HN: File-based cache for slow Python functions

#49

Post by williamzeng0 »

Karellen wrote: post #47
williamzeng0 wrote: post #41

Earlier quoted context omitted.

The version input makes sense, I could also see some developers disliking that ux because of it's verbosity. But to deliberately invalidate you have to make a manual effort in either case.

To me, it's more that invalidating the cache without it requires knowledge of implementation details (i.e. where the cache is stored, and how the cache files are named) that ideally shouldn't have to "leak" for the cache to be generally useful. (Buy hey, we are talking about what is famously one of the hard problems in comp sci, so (respectful) disagreements on how best to do it should be expected :-)

I appreciate the sentiment! Makes a ton of sense.

Re: Show HN: File-based cache for slow Python functions

#50

Post by williamzeng0 »

chlorion wrote: post #45
epr wrote: post #24

def hash_code(code): return hashlib.md5(code.encode()).hexdigest() Be warned. The above function is used as part of the hash. The ostensible purpose is to prevent using cached values of functions who's code has changed, but it does not handle dependencies of that function.

Using md5 for this seems like an odd choice. Sha1 is a better choice even for non-cryptographic use cases, it's quite a bit faster than md5. Even better would be something like xxhash! According to a quick bash script I wrote to benchmark the popular hash functions, md5 comes out last compared to sha1, sha256, sha512, and blake2, and by a decent margin! A good rule of thumb is to never use md5 at all. Not even for no…

That sounds great, I'm going to see how Sweep does on this issue: https://github.com/sweepai/sweep/issues/3333
Post reply on HN