Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

github.com

Post reply on HN
Print view View original

21–30 of 72 posts

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#21

Post by rollcat »

galonk wrote: post #11

I got excited but then read that it's some kind of cloud-based web application thing. Is there something like this (show memory use and call times for a Python process) that just runs on my computer to help me profile a long-running Python process?

Yes! It's called profiling and there are many ways to do that. Python has built-in profiling tools (profile, cProfile), there are also whole-system profiling solutions like DTrace (sadly, that is not available on Linux). No fancy GUIs AFAIK, you'd have to RTFM a bit.

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#22

Post by rcarmo »

This interests me a lot because I'm using Azure App Insights (full disclosure: I work at Microsoft) after a couple of years of New Relic and I'm constantly looking for better takes on the "let's instrument this code and profile it remotely" thing, especially around gevent and asyncio (which have their own little challenges).

I've been thinking about building my own using Prometheus as collector/visualiser. Time hasn't been on my side, but eventually...

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#24

Post by Sean1708 »

> The agent overhead is measured to be less than 1% for applications under high load.

Do you have the methodology and data that you used to obtain this figure? Because to be honest I'm quite dubious, especially for an app which is CPU bound.

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#25

Post by alex- »

galonk wrote: post #11

I got excited but then read that it's some kind of cloud-based web application thing. Is there something like this (show memory use and call times for a Python process) that just runs on my computer to help me profile a long-running Python process?

psutil (https://pythonhosted.org/psutil/) is awesome for collecting valuable monitoring information.

pyrasite (http://pyrasite.com/) will let you inject code into a process. This can be used to add monitoring of private internal state etc (if you have no other options).

If you want to have locally hosted graphs then grafana and influx are my current tools of choice.

It is going to be more work than swiping a credit card, but not a crazy amount.

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#27

Post by alex- »

activatedgeek wrote: post #23

Now that this has come up, can somebody explain me how do profilers work? My main concern being regarding the overhead to the process it adds.

PyCon 2017 had a really good talk about debuggers [1] which covered how PEP523 [2] is making debugging python 3.6+ code much faster. I think that a profiler is somewhat similar, however instead of, potentially, stopping execution on each line it is collecting data.

[1] https://www.youtube.com/watch?v=NdObDUbLjdg [2] https://www.python.org/dev/peps/pep-0523/

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#28

Post by fgarnier »

dmitrim wrote: post #17
fgarnier wrote: post #14

Are there any plans for porting this to Android? For example by using Kivy?

Current agent and the dashboard are designed for long-running applications, such as servers or scripts. There are no plans for end user devices yet. But because the agent is pure Python (it just relies on some system specific functionality, such as signalling), it could work with a few tweaks.

Fair enough. One use case I could think of for this is testing games for long periods on Android, and use this tool to find out where the bottleneck is if there is any. Though given that this is designed for long running apps I can see that it wouldn't be useful for quick profiling but rather for production testing (QA stage etc). There's something similar to that already called Gamebench but it isn't as detailed as this, so was keen on knowing whether this tool would make it to Android :)

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#29

Post by dmitrim »

Sean1708 wrote: post #24

> The agent overhead is measured to be less than 1% for applications under high load. Do you have the methodology and data that you used to obtain this figure? Because to be honest I'm quite dubious, especially for an app which is CPU bound.

We are measuring both, individual profiler overhead when active (printed by the agent in debug mode) and total CPU and memory overhead of the app running over long periods of time with and without agent.

Re: Show HN: StackImpact – Python Production Profiler: CPU, Memory, Exceptions

#30

Post by orf »

dmitrim wrote: post #29
Sean1708 wrote: post #24

> The agent overhead is measured to be less than 1% for applications under high load. Do you have the methodology and data that you used to obtain this figure? Because to be honest I'm quite dubious, especially for an app which is CPU bound.

We are measuring both, individual profiler overhead when active (printed by the agent in debug mode) and total CPU and memory overhead of the app running over long periods of time with and without agent.

Are these apps under load? Is there really only a 1% difference when running apache-bench or seige on the applications?
Post reply on HN