Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: File-based cache for slow Python functions

docs.sweep.dev

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Show HN: File-based cache for slow Python functions

#31

Post by p10_user »

williamzeng0 wrote: post #4
kapilsinha wrote: post #2

I like the simplicity. I definitely get the payoff for standalone Python scripts, where once the script errors out the memory is cleared. But do you see a similar payoff for Jupyter notebooks (or similar)?

I think the marginal gain would be a lot less for Jupyter notebooks, but I've definitely rerun individual cells and wasted time there before. I think it could help if you forget to save the output of a function within a single cell like this: 1. print(f(x)) # -> check what happened 2. out = f(x) # -> turns out we want to save this, so we have to wait again

FWIW There is a built-in cache system for r markdown documents. I'm not up to speed on their exact implementation but I have found it useful.

https://bookdown.org/yihui/rmarkdown-cookbook/cache.html

Re: Show HN: File-based cache for slow Python functions

#32

Post by williamzeng0 »

rmholt wrote: post #6

I have extensively used https://pypi.org/project/diskcache/ . Is there a reason you decided to make an in house solution?

Diskcache works well, we just wanted a dependency free version that we had more control over (easier cache key deletion). I think you'd have to write a custom hashing function for diskcache to use the function source code as a key.

I'm also unsure if Diskcache supports ignoring certain fields in the function call.

Re: Show HN: File-based cache for slow Python functions

#33

Post by williamzeng0 »

nicwolff wrote: post #28
CGamesPlay wrote: post #26

Earlier quoted context omitted.

I was curious to see an alternative to this, but how is this an alternative? You're saying I can implement my own caching of function calls that invalidates when the arguments or source code change...? These feel like entirely separate layers. Did I miss where diskcache does this stuff?

https://grantjenks.com/docs/diskcache/api.html#diskcache.Cac... Decorator to wrap callable with memoizing function using cache. Repeated calls with the same arguments will lookup result in cache and avoid function evaluation.

It does feel like diskcache is for performance gains in production rather than dev time gains. It seems like it would be bothersome to quickly invalidate the cache in case of a bad write.

Re: Show HN: File-based cache for slow Python functions

#34

Post by williamzeng0 »

wildermuthn wrote: post #9

If you aren’t caching LLM functions during development, then you’re an even greater glutton for punishment than the normal engineer. My local file cache Python decorator also allows the decorator to define the hash manually, either by the decorator’s parameter function call that plucks a value from the cached function params, or by calling a global function from anywhere with any arbitrary value. What’s cool about ca…

100%, invalidation needs to be fast or you're not really saving time. I'm curious about calling a global function, what's the use case for that?

Re: Show HN: File-based cache for slow Python functions

#35

Post by Karellen »

wildermuthn wrote: post #9

If you aren’t caching LLM functions during development, then you’re an even greater glutton for punishment than the normal engineer. My local file cache Python decorator also allows the decorator to define the hash manually, either by the decorator’s parameter function call that plucks a value from the cached function params, or by calling a global function from anywhere with any arbitrary value. What’s cool about ca…

I feel like adding an argument to the decorator that labels the "version" of the function would make deliberate cache invalidation more straightforward for cache users.

Re: Show HN: File-based cache for slow Python functions

#36

Post by [deleted] »

mpeg wrote: post #14

I recently wrote a version of this that I use in my projects, some things I do differently that you may or may not care about: - from your code it seems you're not sorting kwargs, I would strongly recommend sorting them so that whether you call f(a=1, b=2) or f(b=2, a=1) the cache key is the same - I use inspect.signature to convert all args to kwargs, this way it doesn't matter how a function gets called, the cache…

[deleted]

Re: Show HN: File-based cache for slow Python functions

#37

Post by williamzeng0 »

mpeg wrote: post #14

I recently wrote a version of this that I use in my projects, some things I do differently that you may or may not care about: - from your code it seems you're not sorting kwargs, I would strongly recommend sorting them so that whether you call f(a=1, b=2) or f(b=2, a=1) the cache key is the same - I use inspect.signature to convert all args to kwargs, this way it doesn't matter how a function gets called, the cache…

Making the __dict__ opt-in makes it a lot more user-friendly at the expense of a little verbosity. That makes sense.

These tips make sense, we often use named args in our function calls (not using them has caused so many bugs), but we don't really enforce the order. Copilot doesn't always get it right either.

By moving inspect.getsource out of the wrapper, do you mean initializing it when the module is imported? I'm curious how that improves performance.

Re: Show HN: File-based cache for slow Python functions

#38

Post by williamzeng0 »

martinky24 wrote: post #27
epr wrote: post #24

def hash_code(code): return hashlib.md5(code.encode()).hexdigest() Be warned. The above function is used as part of the hash. The ostensible purpose is to prevent using cached values of functions who's code has changed, but it does not handle dependencies of that function.

How do you suggest one might fix that issue? Also pin the cache to a hash of all dependency versions? And then if one minor update And let's say the dependency did change, but it's generally inert (more error handling around edge cases, for example), how do you factor that in? Blow up the whole cache? Your example isn't really a problem with OPs utility, but a specific example of a broader dependency management probl…

+1, we considered traversing the function's dependencies to key the cache on (not just the initial function source code), but decided to leave this in a as a constraint. Otherwise we also blowing up the cache when we didn't want it to happen.

Re: Show HN: File-based cache for slow Python functions

#39

Post by williamzeng0 »

p10_user wrote: post #31
williamzeng0 wrote: post #4

Earlier quoted context omitted.

I think the marginal gain would be a lot less for Jupyter notebooks, but I've definitely rerun individual cells and wasted time there before. I think it could help if you forget to save the output of a function within a single cell like this: 1. print(f(x)) # -> check what happened 2. out = f(x) # -> turns out we want to save this, so we have to wait again

FWIW There is a built-in cache system for r markdown documents. I'm not up to speed on their exact implementation but I have found it useful. https://bookdown.org/yihui/rmarkdown-cookbook/cache.html

file.mtime. (file last modified) is an awesome way to key the cache.

Re: Show HN: File-based cache for slow Python functions

#40

Post by williamzeng0 »

Twirrim wrote: post #13
varispeed wrote: post #10

Earlier quoted context omitted.

How it fares with several millions of cached objects?

Being sqlite backed, it's really fast and threadsafe, the cache is shared safely between all threads or processes. It's a very mature library, too, nice and polished, I've never once experienced a bug with it.

Thread safety is a big issue with ours, we'll run into issues when two different processes attempt to write to the same location, or we'll get a bad read. This is a better solution for large scale workloads.

Ours is more meant for single-process scripts like an LLM workflow.

Post reply on HN