Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alenka: GPU database engine

github.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Alenka: GPU database engine

#41

Post by paulmd »

rotten wrote: post #20
keknaut wrote: post #5

Earlier quoted context omitted.

why?

We don't really call pandas a database either. It looks like a data processing tool/library. "real" databases have integrated persistence models, as well as discussions and design tradeoffs regarding ACID transactions and scalability. There are query planners and indexes, constraints (type, value, foreign key) and other logic that can help enforce business rules around the data. There are triggers and embedded functi…

That's a matter of perspective. I don't think this is too different in concept from, say, Bigtable, which is billed by Google as a database.

Re: Alenka: GPU database engine

#42

Post by gshulegaard »

tmostak wrote: post #28
nrjdhsbsid wrote: post #12

I keep hearing the promise of GPU databases but they don't seem to be terribly useful for most real world workloads. It reminds me of the big hoopla for GPU h264 encoders. When they came out everyone realized the quality was worse and not much faster. Some things don't lend themselves to parallel processing, notably anything linear like transactions. I mean yeah the GPU can sort a hundred billion items a second but h…

Not every database is a transactional database. The precise goal of databases like Redshift, Vertica, MemSQL, SAP HANA, Exasol and Impala are to be able to crunch multi-billion row datasets as fast as possible. Indexes often don't help with analytic queries as frequently good chunks of tables need to be scanned. You might apply a limit at the end of the query but that doesn't mean you don't need to scan billions of r…

> Not every database is a transactional database.

...

> The precise goal of databases like Redshift, ...

Huh? It was my understanding that Redshift was a fork of PostgreSQL and is therefore not only transactional but also relational.

Simple Google search confirms transactions in Redshift: http://docs.aws.amazon.com/redshift/latest/dg/r_BEGIN.html

Re: Alenka: GPU database engine

#43

Post by Tuna-Fish »

floatboth wrote: post #36
Tuna-Fish wrote: post #24

Earlier quoted context omitted.

The fixed-function encoders are now less than half as good (that is, they require double the bitrate for same quality) as x264. Anyone who is serious about streaming gets a 8-core CPU or maybe a second computer to do the encoding.

No way it's double. 1.5x maybe. And they have H.265 now (in Polaris cards). Not everybody can afford that setup, and I think even an 8-core will struggle with 4K…

When x264 is properly optimized for video games, having the hardware encoders require only double the bitrate for similar results is probably optimistic, if anything. Especially in DOTA/LOL, x264 is just leagues apart in dealing with the semi-static backgrounds. You can tell the people who use the hw encoders because the stream looks like mush, even at high quality settings.

The target for most streamers is Twitch 1080p H.264, because that's the platform where the money is made. 4K is not needed or useful, and H.265 won't help simply because twitch won't stream it.

Re: Alenka: GPU database engine

#44

Post by gshulegaard »

arnon wrote: post #30
nrjdhsbsid wrote: post #12

I keep hearing the promise of GPU databases but they don't seem to be terribly useful for most real world workloads. It reminds me of the big hoopla for GPU h264 encoders. When they came out everyone realized the quality was worse and not much faster. Some things don't lend themselves to parallel processing, notably anything linear like transactions. I mean yeah the GPU can sort a hundred billion items a second but h…

You're thinking about transactional databases, and you're right. Transactional databases will probably not benefit hugely from a GPU. That's not saying it's impossible, but probably not worth the effort. However, there are so many types of databases around. Lambda architectures are all the rage now - you keep one database for your transactionals, and another for analytics. Analytics are huge, in the multi-billions of…

> Transactional databases will probably not benefit hugely from a GPU.

What about parallel queries over a Restriction-Union normalized data model?

I think the benefits would be similar in nature to columnar stores as you note:

> GPUs tend to lend themselves well to analytics, contrary to transactions. Specifically, columnar databases. When the columns are all of the same data type, and the data locality is high, GPUs perform /very/ well.

Re: Alenka: GPU database engine

#45

Post by floatboth »

Tuna-Fish wrote: post #43
floatboth wrote: post #36

Earlier quoted context omitted.

No way it's double. 1.5x maybe. And they have H.265 now (in Polaris cards). Not everybody can afford that setup, and I think even an 8-core will struggle with 4K…

When x264 is properly optimized for video games, having the hardware encoders require only double the bitrate for similar results is probably optimistic, if anything. Especially in DOTA/LOL, x264 is just leagues apart in dealing with the semi-static backgrounds. You can tell the people who use the hw encoders because the stream looks like mush, even at high quality settings. The target for most streamers is Twitch 10…

4K is needed and useful for recording videos, not streaming.

Re: Alenka: GPU database engine

#46

Post by dgudkov »

gshulegaard wrote: post #42
tmostak wrote: post #28

Earlier quoted context omitted.

Not every database is a transactional database. The precise goal of databases like Redshift, Vertica, MemSQL, SAP HANA, Exasol and Impala are to be able to crunch multi-billion row datasets as fast as possible. Indexes often don't help with analytic queries as frequently good chunks of tables need to be scanned. You might apply a limit at the end of the query but that doesn't mean you don't need to scan billions of r…

> Not every database is a transactional database. ... > The precise goal of databases like Redshift , ... Huh? It was my understanding that Redshift was a fork of PostgreSQL and is therefore not only transactional but also relational. Simple Google search confirms transactions in Redshift: http://docs.aws.amazon.com/redshift/latest/dg/r_BEGIN.html

You're confused by the word transactional. The parent meant transactional in the sense that it's oriented for transactional systems (OLTP). Redshift and other columnar DBMSs are designed for analytic workloads, rather than transactional. Nevertheless, they are relational and support database transactions.

Re: Alenka: GPU database engine

#47

Post by paulddraper »

gshulegaard wrote: post #42
tmostak wrote: post #28

Earlier quoted context omitted.

Not every database is a transactional database. The precise goal of databases like Redshift, Vertica, MemSQL, SAP HANA, Exasol and Impala are to be able to crunch multi-billion row datasets as fast as possible. Indexes often don't help with analytic queries as frequently good chunks of tables need to be scanned. You might apply a limit at the end of the query but that doesn't mean you don't need to scan billions of r…

> Not every database is a transactional database. ... > The precise goal of databases like Redshift , ... Huh? It was my understanding that Redshift was a fork of PostgreSQL and is therefore not only transactional but also relational. Simple Google search confirms transactions in Redshift: http://docs.aws.amazon.com/redshift/latest/dg/r_BEGIN.html

Yes, Redshift is a fork of PostgreSQL.

No, it is not transactional.

Yes, it is relational.

Re: Alenka: GPU database engine

#48

Post by gshulegaard »

dgudkov wrote: post #46
gshulegaard wrote: post #42

Earlier quoted context omitted.

> Not every database is a transactional database. ... > The precise goal of databases like Redshift , ... Huh? It was my understanding that Redshift was a fork of PostgreSQL and is therefore not only transactional but also relational. Simple Google search confirms transactions in Redshift: http://docs.aws.amazon.com/redshift/latest/dg/r_BEGIN.html

You're confused by the word transactional . The parent meant transactional in the sense that it's oriented for transactional systems (OLTP). Redshift and other columnar DBMSs are designed for analytic workloads, rather than transactional. Nevertheless, they are relational and support database transactions.

So I guess the parent's use of transactional was flawed then?

I am also confused by the implication that OLTP and OLAP are mutually exclusive. It is my understanding that they are not...

Re: Alenka: GPU database engine

#49

Post by tmostak »

gshulegaard wrote: post #48
dgudkov wrote: post #46

Earlier quoted context omitted.

You're confused by the word transactional . The parent meant transactional in the sense that it's oriented for transactional systems (OLTP). Redshift and other columnar DBMSs are designed for analytic workloads, rather than transactional. Nevertheless, they are relational and support database transactions.

So I guess the parent's use of transactional was flawed then? I am also confused by the implication that OLTP and OLAP are mutually exclusive. It is my understanding that they are not...

People in the database space use the term "transactional database" loosely to refer to databases optimized for handling simultaneous inserts, deletes and updates at rates often measured in thousands per second - the kind that you would use to say power an airline ticketing system or inventory management for a retailer.

Just because a database can support transactions does not mean that it is geared for transactional workloads, and hence would likely not be called transactions.

OLTP+OLAP is called HTAP. Its an active area of research but to my knowledge there is still no silver bullet there - systems that do it often store data in both row and columnar format to ensure they are optimized for both, with associated overhead.

Re: Alenka: GPU database engine

#50

Post by gshulegaard »

tmostak wrote: post #49
gshulegaard wrote: post #48

Earlier quoted context omitted.

So I guess the parent's use of transactional was flawed then? I am also confused by the implication that OLTP and OLAP are mutually exclusive. It is my understanding that they are not...

People in the database space use the term "transactional database" loosely to refer to databases optimized for handling simultaneous inserts, deletes and updates at rates often measured in thousands per second - the kind that you would use to say power an airline ticketing system or inventory management for a retailer. Just because a database can support transactions does not mean that it is geared for transactional…

I guess I have always thought of a transaction as an ACID-compliant concept and "transactional database" to refer to a DB that supports transactions[1].

To that end, I always thought of OLTP vs. OLAP model and engine optimizations to be more or less orthogonal to whether or not a DB is transactional. I would even suggest that the inclusion of transactions in Redshift as justification for my seemingly unconventional view:

> Some PostgreSQL features that are suited to smaller-scale OLTP processing, such as secondary indexes and efficient single-row data manipulation operations, have been omitted to improve performance.[2]

But who knows, maybe I am barking up the wrong tree so-to-speak.

[1] https://en.wikipedia.org/wiki/Database_transaction#Transacti...

[2] http://docs.aws.amazon.com/redshift/latest/dg/c_redshift-and...

Post reply on HN