Live data from Hacker News

How to create a 1M record table with a single query

antonz.org

11–20 of 40 posts

Re: How to create a 1M record table with a single query

#11
post #4

If you're running PostgreSQL, you can use the built-in generate_series (1) function like: SELECT id, random() FROM generate_series(0, 1000000) g (id); There seems to be an equivalent in SQLite: https://sqlite.org/series.html [1] https://www.postgresql.org/docs/current/functions-srf.html

Here's how to do something similar in Q (KDB)

    ([]x?x:1000000)
Which gives a table of a million rows like:

    x
    ------
    877095
    265141
    935540
    49015
    ...

Re: How to create a 1M record table with a single query

#12
post #6

Can someone give me a specific use case for this? As "check how a query behaves on a large table" is very vague to me. E.g. I have a table structure but not alot of rows in it, so i go use this to get alot of rows in to check how fast queries get processed?

You want to run a transaction in production and you're unsure about the impact because it might take a lock and block other transactions for some time. In that case you may want to spin up a separate database with fake data and test it there first.

You could create a new database and restore a production backup instead of using fake data but that might not be allowed or require some kind of approval due to rules protecting the privacy of customers or employees.

Re: How to create a 1M record table with a single query

#13
post #4

If you're running PostgreSQL, you can use the built-in generate_series (1) function like: SELECT id, random() FROM generate_series(0, 1000000) g (id); There seems to be an equivalent in SQLite: https://sqlite.org/series.html [1] https://www.postgresql.org/docs/current/functions-srf.html

This can generally fall under the category of “Generate Test Data in SQL” with both generic techniques like Recursive CTEs, as in the OP, and SQL dialect or tool specific options. Search is your friend. The OP also provides a table lookup technique for readable names but doesn’t address other data types such as timestamps or other functions such as MD5(). Other data distribution techniques other than random may also be needed. This post scratches the surface of testing with generated data.

Re: How to create a 1M record table with a single query

#14

> This is not a problem if your DBMS supports SQL recursion A table of which DMBS’s support this would be useful

WITH RECURSIVE is supported in MariaDB 10.2+, MySQL 8.0+, PostgreSQL 8.4+ and SQLite 3.8+. Oracle 11.2+ and SQL Server 2005 support recursive queries, but without the RECURSIVE keyword.

Re: How to create a 1M record table with a single query

#15
post #14

> This is not a problem if your DBMS supports SQL recursion A table of which DMBS’s support this would be useful

WITH RECURSIVE is supported in MariaDB 10.2+, MySQL 8.0+, PostgreSQL 8.4+ and SQLite 3.8+. Oracle 11.2+ and SQL Server 2005 support recursive queries, but without the RECURSIVE keyword.

Before recent versions, in the Postgres implementation CTEs were a wall to predicate push-down which can make them much less efficient than in other DBMSs. Worth noting if you need to support older versions for any reason.

Re: How to create a 1M record table with a single query

#17

> This is not a problem if your DBMS supports SQL recursion A table of which DMBS’s support this would be useful

FWIW Sybase SQL Anywhere has supported it since v10[1] at least. Current version is v17[2].

[1]: http://dcx.sap.com/index.html#1001/en/dbrfen10/rf-select-sta...

[2]: http://dcx.sap.com/index.html#sqla170/en/html/8190aea36ce210...

Re: How to create a 1M record table with a single query

#18

If your DB doesn't support this technique, you might be able to use this rather disgusting technique, which builds an exponentially growing tower of rows by using nested queries with JOIN's. https://stackoverflow.com/a/61169467

Some DBs support "literal tables" which makes this a little less nasty:

    SELECT n FROM (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) AS d(n)
for instance:

        SELECT seq = units.n + tens.n*10 + hundreds.n*100 + thousands.n*1000
          FROM (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) AS units(n)
    CROSS JOIN (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) AS tens(n)
    CROSS JOIN (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) AS hundreds(n)
    CROSS JOIN (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) AS thousands(n)
         WHERE units.n + tens.n*10 + hundreds.n*100 + thousands.n*1000 BETWEEN 1 AND 1337
You can make it neater still by making the sub-table with a CTE:

    WITH Digits AS (SELECT n FROM (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) AS d(n))
            SELECT seq = units.n + tens.n*10 + hundreds.n*100 + thousands.n*1000
              FROM Digits AS units
        CROSS JOIN Digits AS tens
        CROSS JOIN Digits AS hundreds
        CROSS JOIN Digits AS thousands
             WHERE units.n + tens.n*10 + hundreds.n*100 + thousands.n*1000 BETWEEN 1 AND 1337
or

    WITH Digits AS (SELECT n FROM (VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) AS d(n))
    , Thousands AS (SELECT seq = units.n + tens.n*10 + hundreds.n*100 + thousands.n*1000
              FROM Digits AS units
        CROSS JOIN Digits AS tens
        CROSS JOIN Digits AS hundreds
        CROSS JOIN Digits AS thousands
        )
    SELECT seq FROM thousands ORDER BY seq
     WHERE units.n + tens.n*10 + hundreds.n*100 + thousands.n*1000 BETWEEN 1 AND 1337
though at that point you probably have the recursive option so you'd need to test to see which is more efficient. As a further hack you can combine this and the recursive CTE to extend the number of rows you can return when recursion limits would otherwise be a problem.

You can also use simple window functions to make a sequence from a system table that you know will have a few thousand rows:

    SELECT n=ROW_NUMBER() OVER (ORDER BY object_id) FROM sys.all_columns
Or if you just need a few thousand rows to hang randomly generated data from you don't even need the function.

All a bit hacky compared to some DBs that have sequence support, but useful for generating test data or numbers tables where that isn't supported.

Re: How to create a 1M record table with a single query

#19
post #14

Earlier quoted context omitted.

WITH RECURSIVE is supported in MariaDB 10.2+, MySQL 8.0+, PostgreSQL 8.4+ and SQLite 3.8+. Oracle 11.2+ and SQL Server 2005 support recursive queries, but without the RECURSIVE keyword.

Before recent versions, in the Postgres implementation CTEs were a wall to predicate push-down which can make them much less efficient than in other DBMSs. Worth noting if you need to support older versions for any reason.

Yeah, but recursive CTEs are a different beast and still do not support pushdown. I am not sure if any database supports pushdown into recursive CTEs.

Re: How to create a 1M record table with a single query

#20
post #11
post #4

If you're running PostgreSQL, you can use the built-in generate_series (1) function like: SELECT id, random() FROM generate_series(0, 1000000) g (id); There seems to be an equivalent in SQLite: https://sqlite.org/series.html [1] https://www.postgresql.org/docs/current/functions-srf.html

Here's how to do something similar in Q (KDB) ([]x?x:1000000) Which gives a table of a million rows like: x ------ 877095 265141 935540 49015 ...

Nothing to do with the article, SQL, or a DB, but I can't help wanting to add even just a bit when I see array languaes mentioned. So, in J it's just:

    ?~1000000
or, if you don't need elements to be unique:

    ?$~1000000
Though it's just an array, not a persistent table - I don't know much about Jd :(
Post reply on HN