Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oracle embraces SQLite; wraps it around BerkeleyDB as SQL API

oracle.com

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Oracle embraces SQLite; wraps it around BerkeleyDB as SQL API

#11

Post by dlsspy »

hackermom wrote: post #7

I have nothing but good things to say about SQLite. Small. No dependencies. Lightning fast. Public domain. I use it exclusively for all things databases where the customer for whatever reason does not demand one of the other players. I'm glad to see a major actor in the DB industry picking up on it.

Most importantly: The best tested software out there: http://www.sqlite.org/testing.html

"As of version 3.6.23 (all statistics in the report are against that release of SQLite), the SQLite library consists of approximately 67.2 KSLOC of C code. (KSLOC means thousands of "Source Lines Of Code" or, in other words, lines of code excluding blank lines and comments.) By comparison, the project has 679 times as much test code and test scripts - 45678.3 KSLOC."

Re: Oracle embraces SQLite; wraps it around BerkeleyDB as SQL API

#12

Post by elblanco »

This post just generated a tremendous amount of discussion in my company. Not just for the SQLite wrapper, but also about the Berkeley XML DB. Anyone have experience using Berkeley for medium-to-large scale data stores (1-100GB?)? How's the concurrency? What's the licensing model? Is there a way we can use this without having to give away our source?

Re: Oracle embraces SQLite; wraps it around BerkeleyDB as SQL API

#13

Post by gojomo »

elblanco wrote: post #12

This post just generated a tremendous amount of discussion in my company. Not just for the SQLite wrapper, but also about the Berkeley XML DB. Anyone have experience using Berkeley for medium-to-large scale data stores (1-100GB?)? How's the concurrency? What's the licensing model? Is there a way we can use this without having to give away our source?

I don't have experience with the Berkeley XML DB. I have used the BerkeleyDB 'Java Edition' (a different codebase and on-disk format) for multi-GB spill-to-disk queues and maps/sets.

The BerkeleyDB license is roughly: free to include in open-source projects, pay if you want to redistribute as part of a closed-source product. So like with the GPL, using strictly inside your company might not be considered 'distribution'. But also, if you are selling software licenses their bundling rates might not be bad compared to your unit price. Of course, do your own license analysis before business use.

Re: Oracle embraces SQLite; wraps it around BerkeleyDB as SQL API

#15

Post by dlsspy »

neilc wrote: post #9
hackermom wrote: post #8

Earlier quoted context omitted.

SQLite outperforms it by far. That could also weigh in a little :)

Do you have any benchmarks to support that claim?

I do. I wrote a tool to help me understand what I could do on various storage tiers running as safely as possible. Here's one of my results from linode:

http://skitch.com/dlsspy/nh2qb/kvtest-results-on-linode

Here's the code: http://github.com/dustin/kvtest

Depending on what you're doing and how safe you want it, you can adjust inputs to select a different winner.

Re: Oracle embraces SQLite; wraps it around BerkeleyDB as SQL API

#16

Post by gojomo »

dlsspy wrote: post #15
neilc wrote: post #9

Earlier quoted context omitted.

Do you have any benchmarks to support that claim?

I do. I wrote a tool to help me understand what I could do on various storage tiers running as safely as possible. Here's one of my results from linode: http://skitch.com/dlsspy/nh2qb/kvtest-results-on-linode Here's the code: http://github.com/dustin/kvtest Depending on what you're doing and how safe you want it, you can adjust inputs to select a different winner.

What does the 'auditable' qualifier (which seems to make the difference between SQLite beating BDB, or BDB beating SQLite) mean?

Re: Oracle embraces SQLite; wraps it around BerkeleyDB as SQL API

#17

Post by dlsspy »

gojomo wrote: post #16
dlsspy wrote: post #15

Earlier quoted context omitted.

I do. I wrote a tool to help me understand what I could do on various storage tiers running as safely as possible. Here's one of my results from linode: http://skitch.com/dlsspy/nh2qb/kvtest-results-on-linode Here's the code: http://github.com/dustin/kvtest Depending on what you're doing and how safe you want it, you can adjust inputs to select a different winner.

What does the 'auditable' qualifier (which seems to make the difference between SQLite beating BDB, or BDB beating SQLite) mean?

auditable means every revision of every change was saved in a fashion that allows you to go back in time and what-not.

Specifically, it means this: http://github.com/dustin/kvtest/blob/master/sqlite-base.cc#L...

Re: Oracle embraces SQLite; wraps it around BerkeleyDB as SQL API

#19

Post by strlen »

elblanco wrote: post #12

This post just generated a tremendous amount of discussion in my company. Not just for the SQLite wrapper, but also about the Berkeley XML DB. Anyone have experience using Berkeley for medium-to-large scale data stores (1-100GB?)? How's the concurrency? What's the licensing model? Is there a way we can use this without having to give away our source?

Project Voldemort uses BerkeleyDB Java Edition as one of the storage engines (the most frequently used one). Java Edition differs from the C version in being log-structured (very fast writes, at the cost of somewhat slower reads). In one deployment, we're able to store ~130 gb per node data (each node having 32 gb of memory, 18gb of it going to JVM heap, 10gb of the JVM heap going to BDB cache), while having average read latency times (which include the overhead of decoding a request packet) of I should add, this is also while using a single bdb environment for all partitions that belong to a node -- note the most efficient way to store this data (bdb je 4.0 does support sharing a cache between multiple environments, so we might move to separate environment per partition model). We've also built the distribution layer ourselves, rather than use the built-in HA features, but if I recall correctly, the first version of Amazon SimpleDB (prior to Erlang re-write) was built with BDB-JE HA (not to be confused with Amazon Dynamo, which is built on regular BDB-JE and provides its own distribution mechanisms).

Overall, we've found that BDB-JE performs best when there's 1:5 ratio between memory and data set size. With JE (as opposed to classic BDB) the issues are: Java implies a certain vertical scalability limit (due to GC pauses with very large heap sizes, however the GC does get better with every release), log-structured B+Tree is great for writes, but might mean more seeks being made for reads (especially if you were to use range query functionality).

Licensing might be an issue if you're shipping a commercial product to customer site. It doesn't matter if it's used on your own servers or in an open source project (and unlike GPL, the BerkeleyDB license is compatible with Apache licensed projects). The licensing also doesn't apply to Perl/Ruby/Python/other dynamic language bindings for BerkeleyDB, you're free to package them (and then request the customer to install BerkeleyDB themselves, much like you would with MySQL).

In short, it's a great solution if you fit its use case. I should also add that interacting with the Sleepycat people online (e.g., on the forums) also makes you forget they're a part of Oracle.

Re: Oracle embraces SQLite; wraps it around BerkeleyDB as SQL API

#20

Post by psaccounts »

gojomo wrote: post #13
elblanco wrote: post #12

This post just generated a tremendous amount of discussion in my company. Not just for the SQLite wrapper, but also about the Berkeley XML DB. Anyone have experience using Berkeley for medium-to-large scale data stores (1-100GB?)? How's the concurrency? What's the licensing model? Is there a way we can use this without having to give away our source?

I don't have experience with the Berkeley XML DB. I have used the BerkeleyDB 'Java Edition' (a different codebase and on-disk format) for multi-GB spill-to-disk queues and maps/sets. The BerkeleyDB license is roughly: free to include in open-source projects, pay if you want to redistribute as part of a closed-source product. So like with the GPL, using strictly inside your company might not be considered 'distributio…

What would be the licensing model if the product is open source, but is NOT free (i.e., customers are charged for an open-source product in which BDB is used)?
Post reply on HN