\d less efficient than [0-9] • Hacker News Board

Re: \d less efficient than [0-9]

#21

Post by pooriaazimi » Sun, May 19, 2013, 10:24 PM UTC

xonea wrote: post #17

rwmj wrote: post #9

I was a bit surprised that Perl does not seem to be matching Unicode digits. Anyone know why? $ echo '0' | perl -pe 'print "yes: " if m/\d/' yes: 0 $ echo '੧' | perl -pe 'print "yes: " if m/\d/' ੧

You have to tell perl to expect utf8 from stdin (switch -C). $ echo '੧' | perl -C -pe 'print "yes: " if m/\d/' and $ perl -e 'use utf8; print "yes\n" if "੧" =~ m/\d/;' both work :)

`man perlunicode` is chockfull of utf8-related stuff (and it's looong): http://perldoc.perl.org/5.14.0/perlunicode.html

Re: \d less efficient than [0-9]

#22

Post by bellbind » Sun, May 19, 2013, 10:25 PM UTC

ars wrote: post #4

Aurel1us wrote: post #2

Short answer: \d includes all the Unicode characters from http://www.fileformat.info/info/unicode/category/Nd/list.htm

Is that actually a good thing? If I'm using \d to validate numbers (for example to check before string to int conversion, or IP address, phone number, or any other use), other unicode digits are not helpful to me. It's great to support unicode, but I don't think the \d should have been extended this way. Add a \ud or something.

If you use a preg engine you can add the /a modifier which excludes unicode chars from matches.

Re: \d less efficient than [0-9]

#23

Post by fleitz » Sun, May 19, 2013, 10:35 PM UTC

The test code creates a new regex every time, would be interesting to see how it works with a compiled and reused regex.

Re: \d less efficient than [0-9]

#24

Post by deskglass » Sun, May 19, 2013, 10:41 PM UTC

yahelc wrote: post #16

ars wrote: post #5

Earlier quoted context omitted.

Anyone know if this happens in other languages?

Doesn't appear to in JavaScript: "੧".match(/\d/); //null (Incidentally, this may explain the finding from http://stackoverflow.com/a/16622773/172322 , as to why adding the RegexOptions.ECMAScript flag in the C# code eliminates the performance gap)

Nor in python:

print re.match(r'\d','੧')

None

Re: \d less efficient than [0-9]

#25

Post by LawnGnome » Sun, May 19, 2013, 10:43 PM UTC

ars wrote: post #5

wging wrote: post #3

Earlier quoted context omitted.

...at least in C# regexes.

Anyone know if this happens in other languages?

Happens in PHP only if you enable Unicode regex handling via the /u modifier and are running libpcre 8.10 or later (which corresponds to PHP 5.3.4 and later, assuming you're using the bundled libpcre): http://3v4l.org/QD3k0

Re: \d less efficient than [0-9]

#26

Post by joosters » Sun, May 19, 2013, 10:45 PM UTC

The quoted benchmarks all complete in fractions of a second. Not a good sign. They may be reliable results, performed accurately, but why risk it?

IMO you should be running something for much longer, to protect against random short spurious events. e.g. a task reschedule, interrupts, etc could add significant variances. It wouldn't hurt to add a few more zeros to the loop and wait a minute for the results.

Re: \d less efficient than [0-9]

#27

Post by nknighthb » Sun, May 19, 2013, 10:46 PM UTC

xudongz wrote: post #18

ars wrote: post #5

Earlier quoted context omitted.

Anyone know if this happens in other languages?

Not true for Go http://play.golang.org/p/ls96RxJxpz

I would be reluctant to rely on this until the Go documentation is clearer on the intended behavior. Right now it's very poorly specified. The regex doc[1] talks about "same general syntax" as Perl, but points to [2], which doesn't seem to understand what it's saying, describing '\d' in terms of its "Perl" meaning, but then saying that it's [0-9].

[1] http://golang.org/pkg/regexp/

[2] https://code.google.com/p/re2/wiki/Syntax

Re: \d less efficient than [0-9]

#28

Post by wulczer » Sun, May 19, 2013, 10:46 PM UTC

deskglass wrote: post #24

yahelc wrote: post #16

Earlier quoted context omitted.

Doesn't appear to in JavaScript: "੧".match(/\d/); //null (Incidentally, this may explain the finding from http://stackoverflow.com/a/16622773/172322 , as to why adding the RegexOptions.ECMAScript flag in the C# code eliminates the performance gap)

Nor in python: print re.match(r'\d','੧') None

it does when using the re.U flag

  re.match(r'\d', u'੧', re.U)
  

  sys.version
  2.7.3 (default, Mar  4 2013, 14:57:34) \n[GCC 4.7.2]

Re: \d less efficient than [0-9]

#29

Post by joosters » Sun, May 19, 2013, 10:53 PM UTC

fleitz wrote: post #23

The test code creates a new regex every time, would be interesting to see how it works with a compiled and reused regex.

It compiles it once and then matches it against 10000 strings.

Re: \d less efficient than [0-9]

#30

Post by hdragomir » Sun, May 19, 2013, 10:55 PM UTC

0x0 wrote: post #10

I wonder what kind of security vulnerabilities could be looming in validators not expecting non-ascii 0-9 digits and using this regex?

That's exactly what I was thinking!