Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nondeterministic Functions in MySQL (i.e. Rand) Can Surprise You

percona.com

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Nondeterministic Functions in MySQL (i.e. Rand) Can Surprise You

#11

Post by mjevans »

nine_k wrote: post #8

I prefer to prepare the datasets first, and not depend on the order of evaluation (which you may fail to predict correctly): select * from some_table join (select rand() as random_id) on some_table.the_id = random_id This way it is obvious that rand() will be evaluated only once.

I like your query as a portable version of the written function, but I also think that it could fail to yield the results probably intended before that mess was written.

If the intent is actually to select one, and only one, random row from the table than this Stack Overflow entry has a result that is in most cases in the direction I was thinking (LIMIT 1, and have some way of selecting a random entry, remembering that the ID field might be monotonic, but isn't assured to be continuous (have no gaps))

https://stackoverflow.com/questions/19412/how-to-request-a-r...

Unfortunately it seems that it depends on language extensions that are not portable.

EDIT:

It unfortunately also appears that the easiest query scales poorly to large tables as it works by assigning EACH row a random ID and then limiting it to one of those by some order (probably lowest assigned row).

A more effective approach might be a function or procedure based around a pre-recorded 'maximum' ID for the column (assuming it's vaguely monotonic and not a GUID), picking a random result between base and that ID, Verifying it exists, and if it doesn't trying again up to N times before using a fallback procedure. (E.G. after 100 random stabs that fail selecting ~500 records in the range of lastRND +/- 250 and using the ORDER BY rand() or equivalent on that sub-set). If even /that/ fails pick the next or previous record by which is closest.

Re: Nondeterministic Functions in MySQL (i.e. Rand) Can Surprise You

#12

Post by nine_k »

mjevans wrote: post #11
nine_k wrote: post #8

I prefer to prepare the datasets first, and not depend on the order of evaluation (which you may fail to predict correctly): select * from some_table join (select rand() as random_id) on some_table.the_id = random_id This way it is obvious that rand() will be evaluated only once.

I like your query as a portable version of the written function, but I also think that it could fail to yield the results probably intended before that mess was written. If the intent is actually to select one, and only one, random row from the table than this Stack Overflow entry has a result that is in most cases in the direction I was thinking (LIMIT 1, and have some way of selecting a random entry, remembering th…

I think that something like

  select * from the_table 
  offset rand(select count(*) from the_table)
  limit 1;
would do the trick with least bias, though not necessarily very fast.

Re: Nondeterministic Functions in MySQL (i.e. Rand) Can Surprise You

#13

Post by mjevans »

nine_k wrote: post #12
mjevans wrote: post #11

Earlier quoted context omitted.

I like your query as a portable version of the written function, but I also think that it could fail to yield the results probably intended before that mess was written. If the intent is actually to select one, and only one, random row from the table than this Stack Overflow entry has a result that is in most cases in the direction I was thinking (LIMIT 1, and have some way of selecting a random entry, remembering th…

I think that something like select * from the_table offset rand(select count(*) from the_table) limit 1; would do the trick with least bias, though not necessarily very fast.

The offered example query is could return no results in the presence of record gaps / deleted records; also the OFFSET value should probably be computed within a sub-query, not exposed.

A corrected version based on being >= or That was the reason I suggested taking a limited number of stabs at random numbers within the allowed range; my example still has some slight bias (a little in the 'best case' fallback, and a lot in the absolute fallback), but it tries to deliver a truly random result first, many times.

Re: Nondeterministic Functions in MySQL (i.e. Rand) Can Surprise You

#14

Post by Zalastax »

abundant wrote: post #10
paulddraper wrote: post #7

Earlier quoted context omitted.

They say "Functions" though.

Here one would read “that is” as something like “specifically”.

But that's not how one should read "that is". The author should have used "e.g." here, no question. Using "i.e." would imply that there exists only one nondeterministic function in MySQL, which is not the case.

Re: Nondeterministic Functions in MySQL (i.e. Rand) Can Surprise You

#15

Post by cryptonector »

As to SQLite3, the author should have indicated what version they tested, and they should have tested more versions. SQLite3 3.8.3 introduced a flag for marking functions as deterministic (vs. not), and in the version I just tried (3.11.0) it works as expected.

The rule I expect for where clause expression evaluation is: standard short-circuit semantics. That calls to deterministic functions get memoized, or that constant sub-expressions get hoisted out, is a plus -- I do expect GCSE and optimization of deterministic functions from modern RDBMSes. I also expect that functions meant to be used in queries have no side effects -- that they insert/update/delete no rows on any tables.

Post reply on HN