111–120 of 336 posts
OP, could I ask a question?
You mention 1TB files. Why do you guys at embl not use a database for this sort of stuff? I'd figure that with some proper indexing, I figure you could see pretty decent speedups just from that already.
Earlier quoted context omitted.
That may be the case. However, my point is that we started with a rather direct implementation of a formula in a paper. This was very easy to write but took hours on a test set (which we could extrapolate to taking weeks on real data!). Then, I spent a few hours and ended up with that ugly code that now takes a few seconds (and is dominated by the whole analysis taking several minutes, so it would not be worth it eve…
Right, and my point is that you could probably 1. Have gotten similar performance boosts elsewhere, meaning that you wouldn't have needed to refactor this function in the first place (although the implication of a 10000x speedup means that may not be true, although I can absolutely see the potential for 100x speedups in this code, depending on exactly what the input data is) 2. Its likely that there are much more nat…
2. I don't care about faster at this point. The function is fast enough. Maybe there is some magic incantation of pandas that will be readable and compute the same values, but I will believe it when I see it. What I thought was more idiomatic was much slower.
I think this is more of a case of "the problem does not fit numpy/pandas' structure (because of how the duplicated indices need to be handled), so you end up with ugly code."
> The result is that I find myself doing more and more things in Haskell, which lets me write high-level code with decent performance (still slower than what I get if I go all the way down to C++, but with very good libraries). This strikes me as an odd conclusion to come to if speed was the main motivator.
OP here. Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode. Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me. Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.
Then weigh in the hard realities of some engineering problems. It won't matter that it takes 1% of the time to implement a video decoder in python if it can't deliver decoded frames in a timely manner. It won't matter that the C solution will run 1000x faster if you need a month to develop what should be delivered on Friday.
I'm sorry if this is already covered in the article. I had a brief look before but it won't currently load.
Earlier quoted context omitted.
OP here. Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode. Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me. Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.
Interesting assertion re: TimeToWriteCode, but I think there's TimeToWriteCode vs. TimeToWriteGoodCode. I'm working on my first serious Python project right now, and I find it's super easy to throw together some code that more or less works; but for solid, readable, documented, properly unit-tested code I hope is production-ready, it's not any faster than Perl or Golang. (Sure, if you're a Python expert it's faster f…
Proper unit-testing is also going to take roughly the same time in any language, just because you have to think hard about sensible tests (although I still love mocking/patching in Python, so I'd give it an edge, plus pdb/ipdb for debugging tests is cool). Production-ready also includes deployment, which for anything non-trivial I'd say Golang > Python > Perl.
Finally, if we're talking "serious project", IMO tooling and how that tooling integrates into a CI pipeline are more important than development speed, because as a team or project goes, terrible CI will slow developers more than any language. Although again here I think Python does quite well with decent linting, unit test frameworks, and code coverage options, Golang's opinionated tools are simpler in this respect.
(I enjoyed C# for similar reasons, although I don't think it's kept up w.r.t. tooling - been ages since I used it though.)
The go-to solution for speeding up Python code should always be first to use Cython on critical sections of your Python code and tweak your code using type annotations, at least IMHO.
I have an impression that there are features in Python that give very little programmer productivity but make the language slow. It should be possible to implement a hypothetical FastPython without such features but with great performance gains. Of course it wouldn't be compatible with most of the libraries. I can imagine though that porting most of the libraries to FastPython still would be a manageable task. I wond…
You mean like Cython [1] or RPython [2] from PyPy [3] ? [1] http://cython.org/ [2] https://rpython.readthedocs.io/en/latest/rpython.html [3] http://pypy.org/
Earlier quoted context omitted.
@ the OP - not to sound hostile, but you write code (like in the example here [1]) that is bound to be slow, just from a glance at it. vstacking, munging with pandas indices (and pandas in general), etc; in order for it to be fast, you want pure numpy, with as little allocations happening as possible. I help my coworkers “make things faster” with snippets like this all the time. If you provide me with a self-containe…
That is the _FAST_ version of the code (people keep saying "of course, it's slow", when it's the fast version). Here is an earlier version (intermediate speed): https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e... It's not so easy to post the data to reproduce a real use-case as it's a few Terabytes :) * Here's a simple easy code that is incredibly slow in Python: interesting = set(line.strip() for line in…
with open('interesting.txt') as interesting_file:
interesting = {line.strip() for line in interesting_file}
with open('data.txt') in data_file:
total = sum(int(val) for id, val in map(lambda line: line.split('\t'), data_file) if id in interesting)Earlier quoted context omitted.
Much appreciated! One clarification, since my own language was sloppy. C# did not get open sourced in 2014. That was the date that Microsoft open sourced just about everything (and that's a weasel word there - to my knowledge, it is everything) that wasn't already open source. Things like ADO/ASP/Windows Forms/etc were open sourced back in 2008, along with Microsoft's implementation of their framework libraries, whic…
You wrote "Things like ADO/ASP/Windows Forms/etc were open sourced back in 2008". Could you explain that further? It appears to contradict this ZDNet article from 2014, http://www.zdnet.com/article/microsoft-to-open-source-more-o... , which says: > Microsoft to open source more of .NET, and bring it to Linux, Mac OS X > Microsoft is porting its server-side .NET stack to Linux and Mac OS X, and is making more of that…
You're making a reasonable mistake of confusing .NET Core with the .NET Framework. They're different things. .NET Core is a new development that is not directly compatible with applications relying on the .NET Framework. As you can read on the blog post you linked to, .NET core was announced as open source before it was released -- which was 2015. Its implementation and technologies being open sourced is something altogether different.
Earlier quoted context omitted.
Viper is an interesting approach on speeding up Python. It's developed for MicroPython, which does give them room for breaking changes, but has trade-offs. Arithmetic is much faster, but dictionary lookups take much longer compared to CPython. Viper is a code-emitter from a large subset of Python, and even allows for inline assembly. But it's only for a few architectures at the moment, like ARM and x86.
Viper is called Zerynth today: https://wiki.python.org/moin/Zerynth
Zerynth is a development suite. Notably, it makes use of a VM.
Viper is just one of the code emitters buried inside the MicroPython source code, like here [0]. Notably, it produces native code, not bytecode for a VM.
[0] https://github.com/micropython/micropython/blob/master/py/ob...