Memray: a memory profiler for Python • Hacker News Board

Re: Memray: a memory profiler for Python

#51

Post by sterlinm » Sat, Apr 23, 2022, 1:57 AM UTC

noitpmeder wrote: post #25

Want to chime in to say it's awesome that Bloomberg is releasing this product to the world. I don't think many financial organizations would even consider releasing anything as open source, even if it is as general purpose as this tool. My current one (HFT,MM) wouldn't go for it. It also shows good engineering practices. I'd venture a guess that they are probably one of the better ones in the space. Makes me want to…

> I don't think many financial organizations would even consider releasing anything as open source, even if it is as general purpose as this tool.

Not to discount how great it is that Bloomberg is contributing this to the community, but there are a lot of other interesting open source tools that have been open sourced by by finance firms.

Pandas, for one, was originally developed internally at the hedge fund AQR.

Re: Memray: a memory profiler for Python

#52

Post by sterlinm » Sat, Apr 23, 2022, 2:09 AM UTC

sterlinm wrote: post #51

noitpmeder wrote: post #25

Want to chime in to say it's awesome that Bloomberg is releasing this product to the world. I don't think many financial organizations would even consider releasing anything as open source, even if it is as general purpose as this tool. My current one (HFT,MM) wouldn't go for it. It also shows good engineering practices. I'd venture a guess that they are probably one of the better ones in the space. Makes me want to…

> I don't think many financial organizations would even consider releasing anything as open source, even if it is as general purpose as this tool. Not to discount how great it is that Bloomberg is contributing this to the community, but there are a lot of other interesting open source tools that have been open sourced by by finance firms. Pandas, for one, was originally developed internally at the hedge fund AQR.

Some other interesting projects from financial services firms:

https://perspective.finos.org/ (JP Morgan)

https://github.com/jpmorganchase/regular-table (JP Morgan)

https://github.com/man-group/dtale (Man Numeric)

https://github.com/deshaw/pyflyby (DE Shaw)

Re: Memray: a memory profiler for Python

#53

Post by sterlinm » Sat, Apr 23, 2022, 2:13 AM UTC

erwincoumans wrote: post #8

uniqueuid wrote: post #3

Tangential question: Does anybody have a good recommendation for a profiler that works well with massively async codebases? My experience has been that the concurrent nature of coroutines can make it hard to reason about what's going on at a particular point in time. If you don't know how many things you're awaiting on at a specific moment (and what potential external stuff they may be interacting with), it's not exa…

I like open source Perfetto UI (formerly Google Chrome about::/tracing) Wrote my own timing trace json export for Python and C++, across threads and processes. The docs have some pointers to creating your own traces with the Tracing SDK. See https://ui.perfetto.dev

Looks like you can use magic-trace with a perfetto based UI. https://github.com/janestreet/magic-trace