Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Lisp in Production

tech.grammarly.com

Post reply on HN
Print view View original

71–80 of 119 posts

Re: Running Lisp in Production

#71

Post by current_call »

apalmer wrote: post #15
lkrubner wrote: post #11

Good lord, I would go insane if I ran into a bug like this: "We've built an esoteric application (even by Lisp standards), and in the process have hit some limits of our platform. One unexpected thing was heap exhaustion during compilation. We rely heavily on macros, and some of the largest ones expand into thousands of lines of low-level code. It turned out that SBCL compiler implements a lot of optimizations that a…

I dont really understand the downvotes... I agree with you would go nuts. My two cents, if your macro is expanding to thousands of lines of code, your doing something wrong I think. I would expect Macros to expand out to a few lines of code which might have function calls that themselves may contain however many lines of code... but expanding to thousands of lines INLINE via macros seems wrong.

Have you seen any recursive macros?

https://github.com/lispgames/cl-sdl2/blob/ec64831109a17b0dda...

Re: Running Lisp in Production

#72

Post by white-flame »

We deploy distributed, multi-language, centrally Lisp/SBCL servers as well. A few specifics that I'd point out:

Many of SBCL's optimizations are fine grained selectable, using internal SB-* declarations. I know I was at least able to turn off all optimizations for debug/disasm clarity, while specifically enabling tail recursion so that our main loop wouldn't blow up the stack in that build configuration. These aren't in the main documentation; I asked in the #sbcl IRC channel on FreeNode.

You can directly set the size of the nursery with sb-ext:bytes-consed-between-gcs, as opposed to overprovisioning the heap to influence the nursery size. While we've run in the 8-24GB heap ranges depending on deployment, a minimum nursery size of 1GB seems to give us the best performance as well. We're looking at much larger heap sizes now, so who knows what will work best.

While we haven't hit heap exhaustion conditions during compilation, we did hit multi-minute compilation lags for large macros (18,000 LoC from a first-level expansion). That was a reported performance bug in SBCL and has been fixed a while back. Since the Debian upstream for SBCL lags the official releases quite a bit, it's always a manual job to fetch the latest versions, but quite worth it.

Great read, and really familiar. :-)

Re: Running Lisp in Production

#73

Post by vseloved »

white-flame wrote: post #72

We deploy distributed, multi-language, centrally Lisp/SBCL servers as well. A few specifics that I'd point out: Many of SBCL's optimizations are fine grained selectable, using internal SB-* declarations. I know I was at least able to turn off all optimizations for debug/disasm clarity, while specifically enabling tail recursion so that our main loop wouldn't blow up the stack in that build configuration. These aren't…

We played with sb-ext:bytes-consed-between-gcs, but couldn't find the right balance. That's why we were surprised with the result of the oversized heap experiment

Re: Running Lisp in Production

#74

Post by vseloved »

white-flame wrote: post #72

We deploy distributed, multi-language, centrally Lisp/SBCL servers as well. A few specifics that I'd point out: Many of SBCL's optimizations are fine grained selectable, using internal SB-* declarations. I know I was at least able to turn off all optimizations for debug/disasm clarity, while specifically enabling tail recursion so that our main loop wouldn't blow up the stack in that build configuration. These aren't…

Can you also point to the particular version of SBCL that fixed for long-running compilation? We have recently upgraded to one of the latest version, but I think I've missed this change - I'm interested to check it in more detail.

Re: Running Lisp in Production

#76

Post by white-flame »

vseloved wrote: post #73
white-flame wrote: post #72

We deploy distributed, multi-language, centrally Lisp/SBCL servers as well. A few specifics that I'd point out: Many of SBCL's optimizations are fine grained selectable, using internal SB-* declarations. I know I was at least able to turn off all optimizations for debug/disasm clarity, while specifically enabling tail recursion so that our main loop wouldn't blow up the stack in that build configuration. These aren't…

We played with sb-ext:bytes-consed-between-gcs, but couldn't find the right balance. That's why we were surprised with the result of the oversized heap experiment

You might want to do some googling about it, but I also seem to remember that SBCL might decide to use large or small MMU page granularity depending on the size of the heap. That might be the watershed trigger for performance. (or just some random mis-remembered nonsense)

Re: Running Lisp in Production

#77

Post by white-flame »

vseloved wrote: post #74
white-flame wrote: post #72

We deploy distributed, multi-language, centrally Lisp/SBCL servers as well. A few specifics that I'd point out: Many of SBCL's optimizations are fine grained selectable, using internal SB-* declarations. I know I was at least able to turn off all optimizations for debug/disasm clarity, while specifically enabling tail recursion so that our main loop wouldn't blow up the stack in that build configuration. These aren't…

Can you also point to the particular version of SBCL that fixed for long-running compilation? We have recently upgraded to one of the latest version, but I think I've missed this change - I'm interested to check it in more detail.

I think it was back in the 0.54-0.59 version range.

Re: Running Lisp in Production

#78

Post by vseloved »

white-flame wrote: post #77
vseloved wrote: post #74

Earlier quoted context omitted.

Can you also point to the particular version of SBCL that fixed for long-running compilation? We have recently upgraded to one of the latest version, but I think I've missed this change - I'm interested to check it in more detail.

I think it was back in the 0.54-0.59 version range.

that's much older than what we used recently, so it should be a different thing

Re: Running Lisp in Production

#79

Post by rectangletangle »

Semiapies wrote: post #52
lkrubner wrote: post #11

Good lord, I would go insane if I ran into a bug like this: "We've built an esoteric application (even by Lisp standards), and in the process have hit some limits of our platform. One unexpected thing was heap exhaustion during compilation. We rely heavily on macros, and some of the largest ones expand into thousands of lines of low-level code. It turned out that SBCL compiler implements a lot of optimizations that a…

It'd be a Hell of a lot more interesting than the millionth time looking into a bug caused by invalid input into a non-validated form in an ancient CRUD intranet app.

At least those are usually pretty quick to fix.
Post reply on HN