Live data from Hacker News

Why SQLite Uses Bytecode

sqlite.org

181–190 of 231 posts

Re: Why SQLite Uses Bytecode

#181
post #106

Earlier quoted context omitted.

> I was advocating for it for decades, but everyone dismisses it with “you don’t know better than rdbms”. That’s almost a religion. Which is simply not true. Query optimization is done heuristically, for the simple reason that you usually need to run the query to get the information required for "perfect" optimization. If the RDBMS really knew better, it wouldn't offer query hints.

Postgres doesn't offer query hints. ;)

[deleted]

Re: Why SQLite Uses Bytecode

#182

Earlier quoted context omitted.

4. lots of small building blocks of static machine code precompiled/shipped with DB software binary, later iterated & looped through based on the query plan the optimizer came up with. Oracle does this with their columnar/vector/SIMD processing In-Memory Database option (it's not like LLVM as it doesn't compile/link/rearrange the existing binary building blocks, just jumps & loops through the existing ones in the req…

That’s a really cool idea! Is there any writeups or articles about this in more detail?

Sounds like 'copy & patch compilation' is a close cousin.

Re: Why SQLite Uses Bytecode

#183
post #48
post #43

Earlier quoted context omitted.

the b5000 was one of the first non -virtual stack machines, but its instruction set isn't any more of a virtual machine than the 8088's or the pdp-10's. there were a number of interpreted stack languages in the 50s, though not nearly as many as there would be later

When one does digital archeology is it quite common to see Assembly referred to as bytecode, when the CPUs are actually interpreters written in microcode. Another example, all the Xerox PARC workstations, which loaded the respective interpreter (Smalltalk, Interlisp, Mesa, Mesa/Cedar) into the CPU as first step during the boot process.

i have never seen assembly, or the code generated from it, referred to as 'bytecode' unless there was no physical hardware or microcode that could interpret it — except in the case of smalltalk, whose bytecode was, as you say, interpreted by microcode on the alto and the d-machines. but possibly your archeology has delved into cultures mine has not? i'd be very interested in reading more, if you happen to have any links handy

Re: Why SQLite Uses Bytecode

#184
post #118

Earlier quoted context omitted.

I love how the example SQL query uses abbreviated columns that sound like runes: select -- count(*) a.mandt, b.rldnr, a.bukrs, a.gjahr, a.belnr, b.docln, a.buzei -- * from BSEG as a innerjoin ACDOCA as b on b.rclnt = a.mandt and b.rbukrs = a.bukrs and b.gjahr = a.gjahr and b.belnr = a.belnr where a.mandt = '715' and b.rldnr = '0L' and b.docln = '000001' and a.buzei = '001' and a.gjahr = '2018' --and a.gjahr = '2017'…

yeah, these accounting table field names have not changed since around 30 years, which is not necessarily a bad thing... for a human readable view of this you can look e.g. here https://docsfortec.com/sap/S4/tables/ACDOCA

They're a bit of a bilingual mess.

Did they start out with German table names and decide to give new tables English names at some point?

Re: Why SQLite Uses Bytecode

#185

I think most people associate bytecode VMs / interpreters with general-purpose programming languages, but it's a surprisingly useful concept in other contexts. Sometimes bytecode VMs appear in unexpected places! A few that I'm aware of: - eBPF, an extension mechanism in the Linux kernel - DWARF expression language, with interpreters in debuggers like GDB and LLDB - The RAR file format includes a bytecode encoding for…

Yep, Greenspun's tenth rule:

Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.

Re: Why SQLite Uses Bytecode

#186
post #13

Earlier quoted context omitted.

VMs really can be. They have a long history in code portability. In the old days it really wasn't uncommon to use an approach centered around some interpretable byte code running in a vm, where the reusable vm was all that needed porting to different architectures and operating systems. This all happened well before Java. It was really big in gaming, Zork, Sierra games, LucasArts games, and even a few more "action" g…

Don't know about Flashback, but Another World famously was VM based.

I might be wrong, but I've read on other project pages that it is. https://github.com/sylefeb/a5k

Re: Why SQLite Uses Bytecode

#187

I think most people associate bytecode VMs / interpreters with general-purpose programming languages, but it's a surprisingly useful concept in other contexts. Sometimes bytecode VMs appear in unexpected places! A few that I'm aware of: - eBPF, an extension mechanism in the Linux kernel - DWARF expression language, with interpreters in debuggers like GDB and LLDB - The RAR file format includes a bytecode encoding for…

The original TeX Fonts stored their metrics in TFM (short for TeX font metrics) files, which contains a bytecode interpreter for calculating ligatures and kerning between characters. I learned about that when I tried reading the files myself. From what I can tell, modern fonts using OpenType just have tables to accomplish something similar now, in the form of the GSUB and GPOS tables? Documentation for the TFM format…

TrueType (and OpenType, which is an evolution of TT) absolutely includes a bytecode instruction set: https://developer.apple.com/fonts/TrueType-Reference-Manual/...

Re: Why SQLite Uses Bytecode

#188
post #98

Earlier quoted context omitted.

Typically you get XML for showplan because it can represent the tree structure better.

You get XML for estimated execution plans if you use SET SHOWPLAN_XML ON. You get the actual execution plan XML along with your results if you use SET STATISTICS XML. You can see cached execution plans in the sys.dm_exec_query_plan dynamic management view. The estimated/actual execution plan feature in SQL Server Management Studio was changed to use XML in SQL Server 2005. The SQL Server team are only adding new info…

I havent used SQL server much since the 2008 days but it used to happen quite often that the estimated and actual execution plan differed a lot. I think the estimated part also includes how many rows get returned from a subquery and that informs which joining algorithm is used.

I second your longing for something more detailed than explain query plan in Sqlite though.

In fact I complained about it here and some sqlite developer pointed me to these resources:

https://news.ycombinator.com/item?id=30405275

I've never actually tried yet. Queries are still too fast to do all that:)

Re: Why SQLite Uses Bytecode

#189

SQLite's design docs were the first time I had seen a database use a virtual machine instead of walking a tree. I later noticed VMs in libraries, embedded DSLs, and other applications outside of large general-purpose programming languages. That really drove home for me that VMs could be anywhere and were often a useful step in handling a user's expressions.

MonetDB is another database that uses a VM [1], using an instruction set called the MonetDB Assembly Language (MAL). I believe it pioneered the technique [2]. The VM is basically designed to express the query as vectorized functions on columns.

[1] https://stratos.seas.harvard.edu/files/MonetDebull2012.pdf

[2] https://www.researchgate.net/publication/220538804_Database_...

Re: Why SQLite Uses Bytecode

#190
post #151

Earlier quoted context omitted.

I don’t doubt the author, but what is it that makes rendering a tree of objects to a table a difficult problem? Is that not what browsers do when they render a table element?

Quote from the article: "A tree-of-objects representation is more difficult to publish in a human-readable form. The objects that comprise the tree tend to all be very different, and thus it is tricky to come up with a consistent and simple table representation with which to display the objects. Any any such table representation that you do come up with would almost certainly have more than six columns, probably many…

Hey, this is excellent. Thanks for the thorough response.
Post reply on HN