Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More Itertools

more-itertools.readthedocs.io

Post reply on HN
Print view View original

31–40 of 51 posts

Re: More Itertools

#31

Post by __mharrison__ »

mabster wrote: post #24
almostgotcaught wrote: post #22

Earlier quoted context omitted.

> Usally, I'd cast my arrays into a pandas DF I promise I mean no offense by this but this is so comically absurd. Like you know it's not a cast right? Ie that you're constructing pandas dataframes. > How should I reason about the tradeoff of using something like this vs pandas/numpy ? For small sizes, operations on native types will be faster than the construction of complex objects.

Also, my grief with DF is they aren't typed (typing module) by column. Maybe that's changed though? It's been a while. The only way to understand what's going on with DF code is to step it in a debugger. I know they can be much faster, but man you pay a maintainability price!

I happen to know a book or two that might help with Pandas.

(Disclaimer: I wrote three of them and spend a good deal of my time helping others level up their Pandas. Spent this morning helping a medical AI company with Pandas.)

Re: More Itertools

#32

Post by benkuykendall »

My favorite function here is more_itertools.one. Especially in something like a unit test, where ValueErrors from unexpected conditions are desirable, we can use it to turn code like

  results = list(get_some_stuff(...))
  assert len(results) = 1
  result = results[0]
into

  result = one(get_some_stuff(...))
I guess you could also use tuple-unpacking:

  result, = get_some_stuff(...)
But the syntax is awkward to unpack a single item. Doesn't that trailing comma just look implausible? (Also I've worked with type-checkers that will complain when a tuple-unpacking could potentially fail, while one has a clear type signatures Iterable[T] -> T.)

Re: More Itertools

#33

Post by screye »

almostgotcaught wrote: post #22
screye wrote: post #19

This looks great. Usally, I'd cast my arrays into a pandas DF and then use the equivalent dataframe operations. To me, pandas and numpy might as well be part of the python stdlib. How should I reason about the tradeoff of using something like this vs pandas/numpy ? Esp. with Numpy 2.0 supporting the string dtype.

> Usally, I'd cast my arrays into a pandas DF I promise I mean no offense by this but this is so comically absurd. Like you know it's not a cast right? Ie that you're constructing pandas dataframes. > How should I reason about the tradeoff of using something like this vs pandas/numpy ? For small sizes, operations on native types will be faster than the construction of complex objects.

No offense taken.

My tasks aren't usually bottlenecked by the df creation operation. To me, the convenience offered by dfs outstrips the compute hit. However, if this is an order of magnitude difference , then it would push me to adopt the more-itertools formulation.

Re: More Itertools

#34

Post by rjmill »

benkuykendall wrote: post #32

My favorite function here is more_itertools.one. Especially in something like a unit test, where ValueErrors from unexpected conditions are desirable, we can use it to turn code like results = list(get_some_stuff(...)) assert len(results) = 1 result = results[0] into result = one(get_some_stuff(...)) I guess you could also use tuple-unpacking: result, = get_some_stuff(...) But the syntax is awkward to unpack a single…

You can also do

  [result] = get_some_stuff(...)

Re: More Itertools

#35

Post by almostgotcaught »

screye wrote: post #33
almostgotcaught wrote: post #22

Earlier quoted context omitted.

> Usally, I'd cast my arrays into a pandas DF I promise I mean no offense by this but this is so comically absurd. Like you know it's not a cast right? Ie that you're constructing pandas dataframes. > How should I reason about the tradeoff of using something like this vs pandas/numpy ? For small sizes, operations on native types will be faster than the construction of complex objects.

No offense taken. My tasks aren't usually bottlenecked by the df creation operation. To me, the convenience offered by dfs outstrips the compute hit. However, if this is an order of magnitude difference , then it would push me to adopt the more-itertools formulation.

> However, if this is an order of magnitude difference , then it would push me to adopt the more-itertools formulation.

My friend it's much worse than a single order magnitude for small inputs

    import time
    import pandas as pd

    ls = list(range(10))

    b = time.monotonic_ns()
    odds = [v for v in ls if v % 2]
    e = time.monotonic_ns() - b
    print(f"{e=}")

    bb = time.monotonic_ns()
    df = pd.DataFrame(ls)
    odds = df[df % 2 == 1]
    ee = time.monotonic_ns() - bb
    print(f"{ee=}")
    print("ratio", ee/e)

    >>> e=1166
    >>> ee=656792
    >>> ratio 563.2864493996569

Re: More Itertools

#36

Post by EdwardDiego »

ericvsmith wrote: post #20
cosmic_quanta wrote: post #15

Earlier quoted context omitted.

It must be possible, because the 'dataclasses' library used to be third-party.

That’s not actually true. While dataclasses to most of its inspiration from attrs, there are many features of attrs that were deliberately not implemented in dataclasses, just so it could “fit” in the stdlib. Or maybe you mean the backport of dataclasses to 3.6 that is available on PyPI? That actually came after dataclasses was added to 3.7. Source: I wrote dataclasses.

> I wrote dataclasses.

Much appreciated!

Re: More Itertools

#40

Post by foretop_yardarm »

almostgotcaught wrote: post #35
screye wrote: post #33

Earlier quoted context omitted.

No offense taken. My tasks aren't usually bottlenecked by the df creation operation. To me, the convenience offered by dfs outstrips the compute hit. However, if this is an order of magnitude difference , then it would push me to adopt the more-itertools formulation.

> However, if this is an order of magnitude difference , then it would push me to adopt the more-itertools formulation. My friend it's much worse than a single order magnitude for small inputs import time import pandas as pd ls = list(range(10)) b = time.monotonic_ns() odds = [v for v in ls if v % 2] e = time.monotonic_ns() - b print(f"{e=}") bb = time.monotonic_ns() df = pd.DataFrame(ls) odds = df[df % 2 == 1] ee =…

my experience is also that numpy and pandas can add 1-2 seconds to python startup time (which is terrible for the testing experience).
Post reply on HN