Date parsing performance on iOS (NSDateformatter vs sqlite) • Hacker News Board

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#21

Post by jurre » Sat, Sep 07, 2013, 1:58 PM UTC

0x0 wrote: post #2

I wonder if this could be improved by just using the standard C library strftime(3) instead of going through sqlite?

I was wondering the same thing since that's also what apple recommends in situations like these. This is what I got on the same hardware:

strptime_l took 58.803 seconds

NSDateFormatter took 107.570 seconds

sqlite3 took 7.022 seconds

And with MishraAnurag's suggestion of using timegm instead of mktime:

strptime_l took 21.656 seconds

NSDateFormatter took 108.163 seconds

sqlite3 took 7.096 seconds

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#22

Post by MishraAnurag » Sat, Sep 07, 2013, 1:58 PM UTC

willvarfar wrote: post #11

Why convert each date with a select if they are putting it into the db anyway? Why not just let sqlite do the conversion as part of the insert statement?

That was to make a fair comparison between the two approaches since NSDateFormatter's dateFromString gives an NSDate, while SQLite was handing back an integer.

But you are right. In production, it makes more sense to let SQLite handle the conversion and insertion in the same statement.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#23

Post by hosay123 » Sat, Sep 07, 2013, 2:01 PM UTC

rorrr2 wrote: post #9

pilif wrote: post #3

>Many web services choose to return dates in something other than a unix timestamp (unfortunately) Wrong. That's very fortunate. Unix time stamps have some serious deficiencies as data type for storing time information: for one, they lack precision. One second just might not do it. Then they lack any time zone information. You will never know what a specific time stamp is in. GMT? UTC? Time zone where the server is i…

Unix timestamps are always UTC (GMT) Quote: > Unix time, or POSIX time, is a system for describing instants in time, defined as the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970

If you've ever moved country, and imported some data from the old country and mixed it with data from your new country, it quickly becomes obvious why preserving source timezone is a deeply useful attribute ("It's a beautifully sunny day! -- me, 04:17hrs").

Or at the very least, preserving the time offset.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#24

Post by coldcode » Sat, Sep 07, 2013, 2:03 PM UTC

jurre wrote: post #21

0x0 wrote: post #2

I wonder if this could be improved by just using the standard C library strftime(3) instead of going through sqlite?

I was wondering the same thing since that's also what apple recommends in situations like these. This is what I got on the same hardware: strptime_l took 58.803 seconds NSDateFormatter took 107.570 seconds sqlite3 took 7.022 seconds And with MishraAnurag's suggestion of using timegm instead of mktime: strptime_l took 21.656 seconds NSDateFormatter took 108.163 seconds sqlite3 took 7.096 seconds

Why not see what sqlite is doing and do something in C yourself that solves the actual problem. It's not surprising that a general purpose Obj-C (or any language) class isn't terribly fast at one specific thing.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#25

Post by dante_dev » Sat, Sep 07, 2013, 2:07 PM UTC

mmm.....can I see the code about NSDateFormatter? because I feel like you're using it wrong. You need to cache somewhere the NSDateFormatter allocation (it is really expensive), reusing the same instance to convert the string to NSDate*.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#26

Post by [deleted] » Sat, Sep 07, 2013, 2:08 PM UTC

justincormack wrote: post #18

mjn wrote: post #7

Earlier quoted context omitted.

If date formatting is a bottleneck for me (it is surprisingly often, because it's very slow in some languages) I typically just run it through the command-line program 'convdate' [1] from crush-tools, which is more or less just a wrapper around strptime+strftime. [1] https://code.google.com/p/crush-tools/wiki/ConvdateUserDocs

If it is a bottleneck shelling out is not a great solution...

[deleted]

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#27

Post by MishraAnurag » Sat, Sep 07, 2013, 2:09 PM UTC

becauseICan wrote: post #15

These two methods start producing different dates after about year 3515.

That's interesting. I'm not sure of the significance of that year or how that relates to the algorithm in Meeus' book. This web page talks about the date algorithms in Meeus' book in some detail but the math is beyond me - http://mysite.verizon.net/aesir_research/date/jdimp.htm. The Julian day conversion algorithm here is the same one used by SQLite.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#28

Post by mjn » Sat, Sep 07, 2013, 2:09 PM UTC

pestaa wrote: post #19

justincormack wrote: post #18

Earlier quoted context omitted.

If it is a bottleneck shelling out is not a great solution...

Shelling out for each piece of data is indeed not great. Shelling out for batch-processing loads of data is on the other hand great.

Yes, and that's the workload convdate is intended for: it batch-converts an entire column of a tab-delimited file. The larger crush-tools suite is intended for unix-style batch processing of tabular data, but fills in some functionality that the classic set of POSIX tools (cut, sort, paste, join, etc.) didn't cover.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#29

Post by MishraAnurag » Sat, Sep 07, 2013, 2:16 PM UTC

dante_dev wrote: post #25

mmm.....can I see the code about NSDateFormatter? because I feel like you're using it wrong. You need to cache somewhere the NSDateFormatter allocation (it is really expensive), reusing the same instance to convert the string to NSDate*.

A link to the source code is present in the article. You can find it here - https://gist.github.com/AnuragMishra/6474321

The NSDateFormatter is already being cached. That was my first suspicion on finding this issue too. We are using one formatter per thread in the production code, but that doesn't apply for the code I've posted since everything is done on the main thread using a single formatter instance.

Re: Date parsing performance on iOS (NSDateformatter vs sqlite)

#30

Post by jurre » Sat, Sep 07, 2013, 2:19 PM UTC

coldcode wrote: post #24

jurre wrote: post #21

Earlier quoted context omitted.

I was wondering the same thing since that's also what apple recommends in situations like these. This is what I got on the same hardware: strptime_l took 58.803 seconds NSDateFormatter took 107.570 seconds sqlite3 took 7.022 seconds And with MishraAnurag's suggestion of using timegm instead of mktime: strptime_l took 21.656 seconds NSDateFormatter took 108.163 seconds sqlite3 took 7.096 seconds

Why not see what sqlite is doing and do something in C yourself that solves the actual problem. It's not surprising that a general purpose Obj-C (or any language) class isn't terribly fast at one specific thing.

Yeah that would probably be the way to go ultimately if you're doing a lot of date parsing, I agree!