Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

github.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#51

Post by dmitrim »

opticalfiber wrote: post #49

Can you comment on how compatible this is with asyncio-based applications? Looks like an interesting product and I'm going to try it out regardless, but it would be cool to get some clarity since I couldn't find anything in the docs besides this: > Time (blocking call) profiler supports threads and gevent. (from https://stackimpact.com/docs/#getting-started-with-python-pr... )

We haven't tested the whole agent with asyncio applications yet. I guess only CPU profiler was tested during development. We'll do and include it in the docs. For now, if you see any problems, please just open a ticket. Thanks!

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#52

Post by dmitrim »

jedi_stannis wrote: post #26

How would I configure StackImpact to run on my celery workers?

We haven't tested it with celery yet. It looks like it should work. gevent is supported by blocking call profiler, and CPU and memory profilers as well as exception and metric reporting are library independent.

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#53

Post by cuu508 »

mooneater wrote: post #43

I've been trying ways to profile my django code on my dev server. Its using runserver and postgres on virtualbox (ubuntu in ubuntu) and takes 20s to display a page. This is not due to slow db queries, those are quick. strace says its making a huge number of calls to: futex(0xe9d550, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 I tried debug_toolbar, Silk, yet-another-django-profiler, these dont give me insight into where all tha…

runserver as in "manage.py runserver"? An issue I have hit a few times is when the app is making a request to itself, but runserver cannot serve requests concurrently. Say, browser loads request A and waits for response. While processing request A, the python code makes an internal request B and waits for response. This deadlocks because runserver will not start processing B before A has finished. Eventually request A times out, and if it was just some tracking call, the page appears to load, just after a long pause.

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#55

Post by cookiecaper »

metalliqaz wrote: post #3

This is an interesting project. It appears to me, based on the README and the name, that it was primarily intended to profile backend services in a web app. I have a question for hacker news. Does Python still have a lot of momentum in this area? I love Python and use it whenever I can, but I find these days that most web frameworks assume right off the bat that you are using Node. The frontend landscape is so heavil…

IMO having some Node.js experience and React experience are very helpful to get hired into any position that is remotely near the web these days. I have seen several of my clients introduce Node.js and of course everyone is introducing SPAs in some form or another, mostly React. Mozilla is even integrating React into Firefox's UI.

I am personally skeptical of these trends, but my skepticism doesn't change the shift of the industry. My advice would be to get some Node and React experience under your belt so that you can at least discuss it intelligently, and it shouldn't be too much of an impediment moving forward.

Python is in a tough spot for growth, IMO. The new generation of languages have internalized much of what made Python great, while leaving behind a lot of the inadequacies and cruft attached to CPython.

Like you, I will always have a soft spot for Python, but it's getting increasingly difficult to continue to see it as the default choice for new projects (outside of a few specific niches).

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#56

Post by drcongo »

dmitrim wrote: post #9
LaurensBER wrote: post #5

Seems really interesting but a lot of companies are not willing or not able to send this to an unknown un trusted party. Would it be possible to host this on premise? Sentry seems to do quite well with a business model where customers are free to host it on premise. That might be worth a consideration. I for one am interested but for me to become a customer I would first need to be able to trail it on my staging envi…

There is no on-prem offering yet, since there was actually no demand/requests. At least with the Golang agent, which was introduced first. With Python agent we will reprioritise it. Thank you for the feedback! (Disclaimer: I work at StackImpact)

I have a badly behaving Flask app at the moment, so trying it out. As a heads up, some of the links don't work in the table of contents on your documentation page - notably everything indented under "Getting started with Python profiling".

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#57

Post by drcongo »

dmitrim wrote: post #9
LaurensBER wrote: post #5

Seems really interesting but a lot of companies are not willing or not able to send this to an unknown un trusted party. Would it be possible to host this on premise? Sentry seems to do quite well with a business model where customers are free to host it on premise. That might be worth a consideration. I for one am interested but for me to become a customer I would first need to be able to trail it on my staging envi…

There is no on-prem offering yet, since there was actually no demand/requests. At least with the Golang agent, which was introduced first. With Python agent we will reprioritise it. Thank you for the feedback! (Disclaimer: I work at StackImpact)

Also, it feels a little light on documentation and functionality for the Python agent considering it's the same cost as using the Go agent. It's hard to tell if I'm actually going to learn anything about what's causing the memory leaks in my app as a lot of the functionality seems to be Go only.

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#58

Post by viraptor »

rollcat wrote: post #21
galonk wrote: post #11

I got excited but then read that it's some kind of cloud-based web application thing. Is there something like this (show memory use and call times for a Python process) that just runs on my computer to help me profile a long-running Python process?

Yes! It's called profiling and there are many ways to do that. Python has built-in profiling tools (profile, cProfile), there are also whole-system profiling solutions like DTrace (sadly, that is not available on Linux). No fancy GUIs AFAIK, you'd have to RTFM a bit.

> DTrace (sadly, that is not available on Linux)

Sysdig is available though. It even got userspace tracers recently (including Python). Alternatively there's "perf" if you want just the kernel side.

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#59

Post by mooneater »

cuu508 wrote: post #53
mooneater wrote: post #43

I've been trying ways to profile my django code on my dev server. Its using runserver and postgres on virtualbox (ubuntu in ubuntu) and takes 20s to display a page. This is not due to slow db queries, those are quick. strace says its making a huge number of calls to: futex(0xe9d550, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 I tried debug_toolbar, Silk, yet-another-django-profiler, these dont give me insight into where all tha…

runserver as in "manage.py runserver"? An issue I have hit a few times is when the app is making a request to itself, but runserver cannot serve requests concurrently. Say, browser loads request A and waits for response. While processing request A, the python code makes an internal request B and waits for response. This deadlocks because runserver will not start processing B before A has finished. Eventually request…

Interesting. Yes "manage.py runserver". In my case, I have been loading one page at a time using runserver.

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#60

Post by mooneater »

trashcan wrote: post #46
mooneater wrote: post #43

I've been trying ways to profile my django code on my dev server. Its using runserver and postgres on virtualbox (ubuntu in ubuntu) and takes 20s to display a page. This is not due to slow db queries, those are quick. strace says its making a huge number of calls to: futex(0xe9d550, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 I tried debug_toolbar, Silk, yet-another-django-profiler, these dont give me insight into where all tha…

Have you tried the low tech way of adding debug log statements that print how long a function takes to run? Once you know what is slow it should be easier to troubleshoot. Also I would check to see if you have any DNS issues in your virtual environment.

I think I would need to sprinkle django internals with log statements. I would do that as a very last resort but I feel it should be possible to avoid.

I did solve this by removing django debug toolbar though trial and error (and still not sure why it was doing that), but I never found a tool which would discover that problem.

Post reply on HN