Date parsing performance on iOS (NSDateformatter vs sqlite) • Hacker News Board

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#31

Post by MishraAnurag » Sat, Sep 07, 2013, 2:23 PM UTC

jurre wrote: post #21

0x0 wrote: post #2

I wonder if this could be improved by just using the standard C library strftime(3) instead of going through sqlite?

I was wondering the same thing since that's also what apple recommends in situations like these. This is what I got on the same hardware: strptime_l took 58.803 seconds NSDateFormatter took 107.570 seconds sqlite3 took 7.022 seconds And with MishraAnurag's suggestion of using timegm instead of mktime: strptime_l took 21.656 seconds NSDateFormatter took 108.163 seconds sqlite3 took 7.096 seconds

Are you using mktime to get the unix timestamp? That might be the slower part as opposed to strptime.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#32

Post by jurre » Sat, Sep 07, 2013, 2:26 PM UTC

MishraAnurag wrote: post #31

jurre wrote: post #21

Earlier quoted context omitted.

I was wondering the same thing since that's also what apple recommends in situations like these. This is what I got on the same hardware: strptime_l took 58.803 seconds NSDateFormatter took 107.570 seconds sqlite3 took 7.022 seconds And with MishraAnurag's suggestion of using timegm instead of mktime: strptime_l took 21.656 seconds NSDateFormatter took 108.163 seconds sqlite3 took 7.096 seconds

Are you using mktime to get the unix timestamp? That might be the slower part as opposed to strptime.

I am, source is here: https://gist.github.com/jurre/6475263

My c is quite poor so if you have any suggestions on how to improve I'd love to hear them!

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#33

Post by stevoski » Sat, Sep 07, 2013, 2:32 PM UTC

A class like NSDateFormatter is designed to handle a wide range of date formats. This usual results in sub-optimal performance. If you find it too slow and you have a known, specific date format you should write a specific fast parser.

I did the same with Java's Integer.parseInt(...) method. It is an interesting task to go through.

Now I'll spend the rest of this rainy afternoon playing around with writing a fast ISO date parser :)

Edit: Seems Java's Joda Time library already does parse ISO dates really quickly. 7 seconds for 4 million on my MBP

Edit: A fast custom date parser for ISO dates I just wrote can parse 4 million dates in 150 milliseconds.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#34

Post by mcdee » Sat, Sep 07, 2013, 2:42 PM UTC

pilif wrote: post #3

>Many web services choose to return dates in something other than a unix timestamp (unfortunately) Wrong. That's very fortunate. Unix time stamps have some serious deficiencies as data type for storing time information: for one, they lack precision. One second just might not do it. Then they lack any time zone information. You will never know what a specific time stamp is in. GMT? UTC? Time zone where the server is i…

Actually the ISO format does not give you timezone information. It gives you offset from UTC, from which you cannot infer anything about the timezone in which the date resides.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#35

Post by MishraAnurag » Sat, Sep 07, 2013, 2:59 PM UTC

jurre wrote: post #32

MishraAnurag wrote: post #31

Earlier quoted context omitted.

Are you using mktime to get the unix timestamp? That might be the slower part as opposed to strptime.

I am, source is here: https://gist.github.com/jurre/6475263 My c is quite poor so if you have any suggestions on how to improve I'd love to hear them!

I'd suggest using timegm instead of mktime, or set the TZ environment variable to UTC to ensure all implementations return an identical date. I ran the same tests and found that the strptime was quite fast, but gmtime was taking most of the time. To speed that up, you could borrow SQLite's implementation. Checkout the computeJD function from SQLite's date.c - http://www.sqlite.org/src/doc/trunk/src/date.c

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#36

Post by jurre » Sat, Sep 07, 2013, 3:14 PM UTC

MishraAnurag wrote: post #35

jurre wrote: post #32

Earlier quoted context omitted.

I am, source is here: https://gist.github.com/jurre/6475263 My c is quite poor so if you have any suggestions on how to improve I'd love to hear them!

I'd suggest using timegm instead of mktime, or set the TZ environment variable to UTC to ensure all implementations return an identical date. I ran the same tests and found that the strptime was quite fast, but gmtime was taking most of the time. To speed that up, you could borrow SQLite's implementation. Checkout the computeJD function from SQLite's date.c - http://www.sqlite.org/src/doc/trunk/src/date.c

timegm actually already makes a huge difference, thanks! Might be useful to make a small fast date parsing library based on the sqlite source code.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#37

Post by nly » Sat, Sep 07, 2013, 3:21 PM UTC

thristian wrote: post #20

pilif wrote: post #3

>Many web services choose to return dates in something other than a unix timestamp (unfortunately) Wrong. That's very fortunate. Unix time stamps have some serious deficiencies as data type for storing time information: for one, they lack precision. One second just might not do it. Then they lack any time zone information. You will never know what a specific time stamp is in. GMT? UTC? Time zone where the server is i…

Another good reason to avoid POSIX timestamps: they ignore leap-seconds. Thus, to determine how many (for example) days are between two timestamps, you need an up-to-date database of leap-second insertions. Maybe an error-bar of a few seconds doesn't sound like much, but if that error bar happens to straddle midnight and (like most code) you get a date-stamp by truncation, you could be off by a day. If that error-bar…

It doesn't matter either way. If you wish to account for leap seconds then the device that creates the timestamp needs access to that very same database.

The advantage of ignoring leap-seconds on the recorder is you can map any sufficiently precise monotonic clock to UNIX time with a simple linear equation. Personally I think it makes a lot more sense to keep the complexity contained to the decoder, rather than the encoder where bugs could mean you end up not recording an accurate timestamp to begin with.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#38

Post by greenlakejake » Sat, Sep 07, 2013, 3:27 PM UTC

objclxt wrote: post #4

pilif wrote: post #3

>Many web services choose to return dates in something other than a unix timestamp (unfortunately) Wrong. That's very fortunate. Unix time stamps have some serious deficiencies as data type for storing time information: for one, they lack precision. One second just might not do it. Then they lack any time zone information. You will never know what a specific time stamp is in. GMT? UTC? Time zone where the server is i…

The other major advantage of using ISO 8601 is it's human readable . Very few people are going to be able to look at a Unix timestamp and convert it in their head (...if you can that's a good party trick).

You must go to weird parties.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#39

Post by nly » Sat, Sep 07, 2013, 3:30 PM UTC

colanderman wrote: post #17

rorrr2 wrote: post #9

Earlier quoted context omitted.

Unix timestamps are always UTC (GMT) Quote: > Unix time, or POSIX time, is a system for describing instants in time, defined as the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970

More importantly (the GP missed this as well), Unix timestamps can't convey local time . Local time has UI implications, e.g. the query "is this event on a weekend" is not generally answerable without the time zone.

For historical dates, I'd rather everyone knew how to convert accurately to and from UTC (2 conversions), rather than relying on everyone to have a bug-free and up to date implementation of 2N(N-1) conversions.

That said, the exception for local time, at least in my opinion, is agreeing on dates in the future meant for human interaction (e.g. "I'll meet you at 7 AM local time in Time Square on the 3rd of April 2068"). Here time zone rules may actually change before the date transpires, and you can't be sure of the representation in any other zone or format until closer to the event.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#40

Post by asveikau » Sat, Sep 07, 2013, 4:05 PM UTC

For those million timestamps I am sure that those extra allocations from the statement APIs are not helpful. Since he references the C code sqlite is using (at a quick glance it looks pretty contained) I don't know why he doesn't just include it directly in his project and call it from objc, no statement API needed.

[Edit: I see now that in the test there is only 1 statement object ever created for a test of a million dates. Better than I thought initially. But my guess is the statement object still creates some degree of inefficiency not found in directly calling the C version.]

7 seconds is a long time in CPU terms, I am sure that he can do better.