Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Whenever: Typed and DST-safe datetimes for Python

github.com

Post reply on HN
Print view View original

121–130 of 151 posts

Re: Whenever: Typed and DST-safe datetimes for Python

#121

Post by Gare »

MrJohz wrote: post #104
Gare wrote: post #87

Earlier quoted context omitted.

Fortunately, JS is getting Temporal, which will be great. No such thing on the horizon for Python yet.

Well, there's libraries like this one, which is the point here. And even in JS, Temporal won't be available broadly for a good while yet (it will probably be rolling out in Firefox in a couple of months' time, but in Safari it's still behind a feature flag, and I don't think there's even a feature-flagged implementation for Chrome yet). In the meantime, it makes sense to use a polyfill — again a library. By all means…

I fully agree with that. Just a bit disappointed that Python is not even considering to fix/replace it's problematic datetime library (as far as I know). Excellent third-party library is good, but an excellent standard library is even better!

Re: Whenever: Typed and DST-safe datetimes for Python

#122

Post by akerl_ »

freehorse wrote: post #43
mr_mitm wrote: post #33

Earlier quoted context omitted.

> People can't even be bothered to read docs but will just blindly install a package just because someone was able to package it and upload it to PyPI. That's a straw man argument. No one said "blindly". You can very well carefully consider the pros and cons of adding a dependency and arrive at the conclusion that it makes sense. Many PyPI packages are in the Debian stable repositories, you could use that as an addit…

To be fair, comment^^ actually said choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency" which is what comment^ answers to, which to me actually sounds like that the added dependency comes in place of "reading docs/changelogs carefully". I think it matters a lot how much one can trust a 3rd party library, how much it is used, how much it is maintained etc. Moreo…

How could it possibly be true that libraries that are so commonly used that you consider them “the standard” are so infrequently used as to be a “very small minority” of libraries people actually use?

Re: Whenever: Typed and DST-safe datetimes for Python

#123

Post by burntsushi »

Gare wrote: post #121
MrJohz wrote: post #104

Earlier quoted context omitted.

Well, there's libraries like this one, which is the point here. And even in JS, Temporal won't be available broadly for a good while yet (it will probably be rolling out in Firefox in a couple of months' time, but in Safari it's still behind a feature flag, and I don't think there's even a feature-flagged implementation for Chrome yet). In the meantime, it makes sense to use a polyfill — again a library. By all means…

I fully agree with that. Just a bit disappointed that Python is not even considering to fix/replace it's problematic datetime library (as far as I know). Excellent third-party library is good, but an excellent standard library is even better!

Maybe this is the first step toward that future.

In Temporal's case, it takes significant inspiration from several existing datetime libraries in Javascript. Those were all stepping stones (among others) that led to Temporal.

Re: Whenever: Typed and DST-safe datetimes for Python

#125

Post by iknownothow »

ariebovenberg wrote: post #92
iknownothow wrote: post #86

I've read the link and the GitHub readme page. I'm sure I'm in the top 1% of software devs for the most number of timestamps parsed. [1] DST is not a problem in Python. It's parsing string timestamps. All libraries are bad, including this one, except Pandas. Pandas does great at DST too btw. And I'm not shilling for Pandas either. I'm a Polars user who helicopters Pandas in whenever there's a timestamp that needs to…

Author here. It's indeed a hard problem to parse "All ISO 8601 formats and all its weird mutant children that differ by a tiny bit." Since the ISO standard is so expansive, every library needs to decide for itself what to support. The ISO standard allows all sorts of weird things, like 2-digit years, fractional months, disallowing -00:00 offset, ordinal days, etc. Javascript's big datetime redesign (Temporal) has an…

Thanks for the reply and apologies for the general cynicism. It's not lost on me that it's people like you that build tools that make the work tick. I'm just a loud potential customer and I'm just forwarding the frustration that I have with my own customers onto you :)

Your customers are software devs like me. When we're in control of generating timestamps, we know we must use standard ISO formatting.

However, what do I do when my customers give me access to an S3 bucket with 1 billion timestamps in an arbitrary (yet decipherable) format?

In the GitHub issue you seem to have undergone an evolution from purity to pragmatism. I support this 100%.

What I've also noticed is that you seem to try to find grounding or motivation for "where to draw the line" from what's already been done in Temporal or Python stdlib etc. This is where I'd like to challenge your intuitions and ask you instead to open the flood gates and accept any format that is theoretically sensible under ISO format.

Why? The damage has already been done. Any format you can think of, already exists out there. You just haven't realized it yet.

You know who has accepted this? Pandas devs (I assume, I don't them). The following are legitimate timestamps under Pandas (22.2.x):

* 2025-03-30T (nope, not a typo)

* 2025-03-30T01 (HH)

* 2025-03-30 01 (same as above)

* 2025-03-30 01 (two or more spaces is also acceptable)

In my opinion Pandas doesn't go far enough. Here's an example from real customer data I've seen in the past that Pandas doesn't parse.

* 2025-03-30+00:00 (this is very sensible in my opinion. Unless there's a deeper theoretical regex pattern conflicts with other parts of the ISO format)

Here's an example that isn't decipherable under a flexible ISO interpretation and shouldn't be supported.

* 2025-30-03 (theoretically you can infer that 30 is a day, and 03 is month. BUT you shouldn't accept this. Pandas used to allow such things. I believe they no longer do)

I understand writing these flexible regexes or if-else statements will hurt your benchmarks and will be painful to maintain. Maybe release them under an new call like `parse_best_effort` (or even `youre_welcome`) and document pitfalls and performance degradation. Trust me, I'd rather use a reliable generic but slow parser than spend hours writing a write a god awful regex that I will only use once (I've spent literal weeks writing regexes and fixes in the last decade).

Pandas has been around since 2012 dealing with customer data. They have seen it all and you can learn a lot from them. ISOs and RFCs when it comes to timestamps don't mean squat. If possible try to make Whenever useful rather than fast or pure. I'd rather use a slimmer faster alternative to pandas for parsing Timestamps if one is available but there aren't any at the moment.

If time permits I'll try to compile a non exhaustive list of real world timestamp formats and post in the issue.

Thank you for your work!

P.S. seeing BurntSushi in the GitHub issue gives me imposter syndrome :)

Re: Whenever: Typed and DST-safe datetimes for Python

#126

Post by burntsushi »

iknownothow wrote: post #125
ariebovenberg wrote: post #92

Earlier quoted context omitted.

Author here. It's indeed a hard problem to parse "All ISO 8601 formats and all its weird mutant children that differ by a tiny bit." Since the ISO standard is so expansive, every library needs to decide for itself what to support. The ISO standard allows all sorts of weird things, like 2-digit years, fractional months, disallowing -00:00 offset, ordinal days, etc. Javascript's big datetime redesign (Temporal) has an…

Thanks for the reply and apologies for the general cynicism. It's not lost on me that it's people like you that build tools that make the work tick. I'm just a loud potential customer and I'm just forwarding the frustration that I have with my own customers onto you :) Your customers are software devs like me. When we're in control of generating timestamps, we know we must use standard ISO formatting. However, what d…

Because you pinged me... Jiff also generally follows in Temporal's footsteps here. Your broader point of supporting things beyond the specs (ISO 8601, RFC 3339, RFC 9557, RFC 2822 and so on) has already been absorbed into the Temporal ISO 8601 extensions. And that's what Jiff supports (and presumably, whenever, although I don't know enough about whenever to be absolutely precise in what it supports). So I think the philosophical point has already been conceded by the Temporal project itself. What's left, it seems, is a measure of degree. How far do you go in supporting oddball formats?

I honestly do not know the answer to that question myself. But I wouldn't necessarily look to Pandas as the shining beacon on a hill here. Not because Pandas is doing anything wrong per se, but because it's a totally different domain and use case. On the one hand, you have a general purpose library that needs to consider all of its users for all general purpose datetime use cases. On the other hand, you have a data scienc-y library designed for trying to slurp up and make sense of messy data at scale. There may be things that make sense in the latter that don't in the former.

In particular, a major gap in your reasoning, from what I can see, is that constraints beget better error reporting. I don't know how to precisely weigh error reporting versus flexible parsing, but there ought to be some deliberation there. The more flexible your format, the harder it is to give good error messages when you get invalid data.

Moreover, "flexible parsing" doesn't actually have to be in the datetime library. The task of flexible parsing is not, in and of itself, overtly challenging. It's a tedious task that can be build on top of the foundation of a good datetime library. I grant that this is a bit of a cop-out, but it's part of the calculus when designing ecosystem libraries like this.

Speaking for me personally (in the context of Jiff), something I wouldn't mind so much is adding a dedicated "flexible" parsing mode that one can opt into. But I don't think I'd want to make it the default.

Re: Whenever: Typed and DST-safe datetimes for Python

#127

Post by karlicoss »

Kwpolska wrote: post #100
BiteCode_dev wrote: post #95

Earlier quoted context omitted.

Ah, so you are not using pyQT, numpy, any database driver, pillow or anything using cryptography, then?

For the libraries you listed, the benefits of using a native library are much larger, since they’re wrapping a well-known library that is known to be secure and fully-featured, or since the performance benefits are actually visible in any significant code snippet. But here, there is no Rust library to wrap, and I doubt the performance of a date-time library would have any effect on the performance of virtually all ap…

datetime handling can absolutely be a hot spot, especially if you're parsing or formatting them. Even for relatively simple things like "parse a huge csv file with dates into dataclasses".

In particular, default implementation of datetime in cpython is a C module (with a fallback to pure python one) https://github.com/python/cpython/blob/main/Modules/_datetim...

Not saying it's necessarily justified in case of this library, but if they want to compete with stdlib datetime in terms of performance, some parts will need to be compiled.

Re: Whenever: Typed and DST-safe datetimes for Python

#128

Post by [deleted] »

barbazoo wrote: post #120
Hasnep wrote: post #23

If you've not read the blog post that explains why this library exists I recommend it. It's called "Ten Python datetime pitfalls, and what libraries are (not) doing about it" https://dev.arie.bovenberg.net/blog/python-datetime-pitfalls...

I am a seasoned programmer but whenever I deal with datetime objects I do my best with unit tests and then just hope none of these “edge” cases apply to us. Meaning: I have no idea really how it works under the hood. Now at least there’s an LLM that might spot a bug every now and then so that’s nice.

[deleted]

Re: Whenever: Typed and DST-safe datetimes for Python

#129

Post by [deleted] »

pkkm wrote: post #14
heavenlyblue wrote: post #6

Earlier quoted context omitted.

What do you expect? There are so many ways to handle this behvaiour it's pretty obvious why this is not allowed. Do you take datetime.date and then compare? Do you assume all dates are datetimes at midnight?

I think wesselbindt meant that datetimes should not inherit from dates.

[deleted]

Re: Whenever: Typed and DST-safe datetimes for Python

#130

Post by kelseydh »

A big revelation for me in solving so much timezone insanity came from realising that timezones should be expressed as locations rather than zones.

Avoid general terms like "Pacific Standard Time" and stick to location-specific ones like: "Vancouver/Canada". The latter is how people expect their time to work, and correctly handles whatever quirky choices jurisdictions choose to do with their time.

Post reply on HN