Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Recording and visualising the 20k system calls it takes to "import seaborn"

blog.mattstuchlik.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Recording and visualising the 20k system calls it takes to "import seaborn"

#31

Post by josephg »

throwaway828 wrote: post #28
josephg wrote: post #21

Earlier quoted context omitted.

Yeah. I recently worked on a small web project being developed at a university. The project is written in flask, and it presents a reasonably simple UI on top of some data living in a mysql database. When I started on the project, page loads often took 10 seconds or more. The web application is used by about 20 people and that was enough to bring their single beefy server to its knees. Someone in NY tried scraping th…

That sounds like some quick kills to be easily made. I use a dev machine that's quite archaic compared to a modern server, a 2nd gen i5 ThinkPad to be precise, that struggles to top 20ms for a request including loading a user and data object, joined tables and all, via ORM from Postgres running locally with a few hundred thousand records in said tables, before touching anything like explicitly adding caching. Check y…

Yeah I spoke in past tense about the performance problems because, as you said, there were an awful lot of easy wins to be made. The site is about 2 orders of magnitude faster now, which is incredibly satisfying.

> Flask's not your problem. You'll have equal or worse woes (if lower level with less hand holding) with anything else.

I’m not so sure about that. It’s hard to run the experiment, but I’ve never seen a nodejs app run anywhere near that slowly. The default-synchronous nature of Python combined with its mediocre performance for straight code magnifies the impact of any bad design choices. At least in a nodejs application your server can happily run many sql queries at the same time, or do other work while it waits for the database. I’m sure sufficiently mediocre web server code can bring nodejs to its knees. But in a decade of working with node, I’ve never seen it done. Certainly not in a web app with only 20 users.

Re: Recording and visualising the 20k system calls it takes to "import seaborn"

#32

Post by josephg »

nonethewiser wrote: post #29
josephg wrote: post #21

Earlier quoted context omitted.

Yeah. I recently worked on a small web project being developed at a university. The project is written in flask, and it presents a reasonably simple UI on top of some data living in a mysql database. When I started on the project, page loads often took 10 seconds or more. The web application is used by about 20 people and that was enough to bring their single beefy server to its knees. Someone in NY tried scraping th…

Im curious what the orm misuse was because Python can obviously handle a lot higher loads than that. Perhaps the orm is to blame for offering some footgun. Or maybe the developer did something impossibly idiotic.

One big problem they had was that the system checked the user’s access permissions on every request. Access control in this application is quite complex, and so the access control code ended up issuing multiple queries and doing a lot of over fetching to do its job. (The classic ORM problem.)

It turned out that this was also happening for all static assets. Oops. And the site is covered in very small images. Double oops.

All told, to load a single page the server was making over 150 sql queries. And because it’s Python, those queries were all issued with blocking code. More than enough to keep the server busy for ages.

Re: Recording and visualising the 20k system calls it takes to "import seaborn"

#34

Post by latency-guy2 »

turtlebits wrote: post #20
albertgoeswoof wrote: post #9

Today I asked a devops engineer to tell me how much time a long (3 seconds avg) api call was spending on database queries, application logic, and network etc. He couldn’t understand the request and instead opened up the azure console and recommended we increase the number cpu cores / memory if performance is an issue. I look at posts like this and cry.

Implementing traces should be done by devs, not the infrastructure team. Devops should implement/support the platform that supports traces.

Disagree, DevOps teams should be looking for resources that are being hit hard unnecessarily and request moves to better solutions when possible.

DevOps teams should be looking at CPU spikes, and should be performing RCAs, and they should be maintaining resources in a healthy state, and they should reject/revert changes and notify problem areas in code by product focused devs.

Product devs, for the most part, are only implementing human lex traces to debug business logic when it arises. Product devs are not equipped with the knowledge to identify system errors that are not "bugs in the code", i.e. they will not be good at telling you why SPROC_LXC1 fails as a result of making a ExcelParserFactoryFactory

Re: Recording and visualising the 20k system calls it takes to "import seaborn"

#35

Post by GuestHNUser »

bb88 wrote: post #15
t8sr wrote: post #12

Earlier quoted context omitted.

Oh absolutely, for some tasks Python is amazing. I use Jupyter notebooks a lot, for example, and the flexibility is an incredible feature. It just worries me when I sometimes see those same Jupyter notebooks running in production, crunching 100s of terabytes of data. Maybe I’m wrong, but I didn’t get the impression everyone realizes exactly how wasteful that is. I guess AWS credits are easy to come by. One thing Goog…

Numpy is competitive with optimized C/C++. So even if it's running in a Jupyter notebook, it's still going to be insanely fast.

> Numpy is competitive with optimized C/C++

Can you cite a source/example for that? I cannot imagine an optimized C program that doesn't blow python with numpy out of the water. Even a poorly written C program is likely to be 2x faster simply because it doesn't have to round trip operations from C to python and back.

Re: Recording and visualising the 20k system calls it takes to "import seaborn"

#36

Post by Spivak »

josephg wrote: post #21
dmwilcox wrote: post #2

I've been writing python for going on 20 years now and while it was a good language to cut my teeth on thus sort of analysis brings only horror. Many thanks to the author for dropping into plain view. I'm going to go back to learning more C and Forth... And shake my fist at passing clouds :)

Yeah. I recently worked on a small web project being developed at a university. The project is written in flask, and it presents a reasonably simple UI on top of some data living in a mysql database. When I started on the project, page loads often took 10 seconds or more. The web application is used by about 20 people and that was enough to bring their single beefy server to its knees. Someone in NY tried scraping th…

I'm so confused by this, Python is really fast. This isn't to say that other languages aren't a lot faster but I can afford to be so ungodly wasteful with CPU bound tasks (on "leaf" programs don't worry I'm not doing this in libraries to be consumed by others) because it literally doesn't matter. The IO to call print(), write a log, or read a file on disk dwarfs the time actually spent running Python code and this is before using the new JIT.

I wouldn't number crunch in Python without something like numpy because you'll pay the cost of Python's dynamism for nothing but a lot of work has gone into making Python's primitives and standard library performant. I steal algorithms from CPython all the time.

Re: Recording and visualising the 20k system calls it takes to "import seaborn"

#37

Post by graemep »

josephg wrote: post #21
dmwilcox wrote: post #2

I've been writing python for going on 20 years now and while it was a good language to cut my teeth on thus sort of analysis brings only horror. Many thanks to the author for dropping into plain view. I'm going to go back to learning more C and Forth... And shake my fist at passing clouds :)

Yeah. I recently worked on a small web project being developed at a university. The project is written in flask, and it presents a reasonably simple UI on top of some data living in a mysql database. When I started on the project, page loads often took 10 seconds or more. The web application is used by about 20 people and that was enough to bring their single beefy server to its knees. Someone in NY tried scraping th…

I have developed a lot of Python based websites (mostly Django), some quite complex, and I have very rarely seen anything that takes seconds to load - sometimes some database queries have been slow. In most cases load time is dominated by loading JS and images.

> The reasons it was slow were all the usual culprits - a misused ORM being the main one.

So, slow queries.

I have not has such bad issues with dependency management either. Not even with old stuff someone else wrote years ago.

Re: Recording and visualising the 20k system calls it takes to "import seaborn"

#38

Post by Jtsummers »

pdonis wrote: post #17
dmwilcox wrote: post #2

I've been writing python for going on 20 years now and while it was a good language to cut my teeth on thus sort of analysis brings only horror. Many thanks to the author for dropping into plain view. I'm going to go back to learning more C and Forth... And shake my fist at passing clouds :)

> I'm going to go back to learning more C and Forth Why would you expect that to decrease the number of syscalls you need? The syscalls are there because the program needs the OS to do things. That need is driven by the application domain, not by the programming language you use.

> The syscalls are there because the program needs the OS to do things.

Maybe some of them are. Many of those syscalls are there because Python (not the core program someone is creating, but rather its platform) needs the OS to do things.

Importing an empty python file takes 28 syscalls (30 measured by their tool, but the last two are closing out the trace not actually related to the import). 29 syscalls if you have any text in it (presumably more for larger files).

The logical equivalent in C for many portions of the import process in Python happen at compile + linker time, not during execution. So while it might not be a pleasant experience to develop, a C equivalent of many Python programs would involve far fewer syscalls at execution time.

Re: Recording and visualising the 20k system calls it takes to "import seaborn"

#39

Post by tomrod »

amluto wrote: post #26
josephg wrote: post #21

Earlier quoted context omitted.

Yeah. I recently worked on a small web project being developed at a university. The project is written in flask, and it presents a reasonably simple UI on top of some data living in a mysql database. When I started on the project, page loads often took 10 seconds or more. The web application is used by about 20 people and that was enough to bring their single beefy server to its knees. Someone in NY tried scraping th…

I once worked on a small web project, at a university, in Python, using WSGI IIRC. It loaded a lot faster than any of the big expensive apps the university had written. Well, there was one exception. The little import statement to import the Oracle database client took maybe 15 seconds. MySQL for the win :) (I would not recommend MySQL for new applications today, although I might recommend it over Oracle…)

Oracle EEE'd MySQL. MariaDB these days. Or postgres along with the rest of the singularity.

Re: Recording and visualising the 20k system calls it takes to "import seaborn"

#40

Post by tomrod »

josephg wrote: post #32
nonethewiser wrote: post #29

Earlier quoted context omitted.

Im curious what the orm misuse was because Python can obviously handle a lot higher loads than that. Perhaps the orm is to blame for offering some footgun. Or maybe the developer did something impossibly idiotic.

One big problem they had was that the system checked the user’s access permissions on every request. Access control in this application is quite complex, and so the access control code ended up issuing multiple queries and doing a lot of over fetching to do its job. (The classic ORM problem.) It turned out that this was also happening for all static assets. Oops. And the site is covered in very small images. Double o…

Woof! That's pretty rough.
Post reply on HN