Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

35% Faster Than The Filesystem (2017)

sqlite.org

Post reply on HN
Print view View original

131–140 of 166 posts

Re: 35% Faster Than The Filesystem (2017)

#131

Post by pdimitar »

pedrocr wrote: post #129
pdimitar wrote: post #127

Earlier quoted context omitted.

Well, sqlite is not recommended for write-heavy workflows anyway. I have the same impressions as your parent commenter: passing around an sqlite connection handle is thread-safe and always will be faster than application-level locking. If you do however have a write-heavy workflow then it's likely time to replace sqlite.

> I have the same impressions as your parent commenter: passing around an sqlite connection handle is thread-safe and always will be faster than application-level locking. I'm not debating that. It's just that that's not the current bottleneck so no point in dealing with the extra complexity. The current solution can saturate my SSD with low CPU usage. > If you do however have a write-heavy workflow then it's likely…

I did a research about a month ago and was very disappointed. Basically, outside BerkeleyDB and Firebase semi-embedded mode (since it is still not fully embedded) we're on our own.

From then on, you can just use BoltDB or its successor(s) if you don't care about SQL. Which I do, and you seem to as well.

At this point I am thinking of developing an Elixir wrapper for sqlite that can also do a best effort strong-typing and be done with it. It seems sqlite made many developers complacent, but then again, the raging consumerism and micro-management economy we live in worldwide does not exactly encourage tinkering and OSS contributions.

Re: 35% Faster Than The Filesystem (2017)

#132

Post by ptx »

zzzcpan wrote: post #119
ahachete wrote: post #106

Earlier quoted context omitted.

Backups on databases like PostgreSQL are strongly consistent, regardless of where you take it from (master or replica). Postgres replication is strictly sequential wrt transaction commit order. As such, splitting the database may incur in significant consistency issues that a backup doesn't incur into. I believe this splitting technique is not a good one except for potentially narrow use cases.

Strong consistency is not what you think then. You can only do stale reads from backups.

I thought ahachete meant that a database backup is atomic, i e. it will only contain fully completed transactions. The problem with data split across databases then is that transactions don't span multiple databases, so you can't get an atomic snapshot of data spanning multiple databases.

Are you saying that even a single-database backup is not atomic?

Re: 35% Faster Than The Filesystem (2017)

#133

Post by pedrocr »

pdimitar wrote: post #131
pedrocr wrote: post #129

Earlier quoted context omitted.

> I have the same impressions as your parent commenter: passing around an sqlite connection handle is thread-safe and always will be faster than application-level locking. I'm not debating that. It's just that that's not the current bottleneck so no point in dealing with the extra complexity. The current solution can saturate my SSD with low CPU usage. > If you do however have a write-heavy workflow then it's likely…

I did a research about a month ago and was very disappointed. Basically, outside BerkeleyDB and Firebase semi-embedded mode (since it is still not fully embedded) we're on our own. From then on, you can just use BoltDB or its successor(s) if you don't care about SQL. Which I do, and you seem to as well. At this point I am thinking of developing an Elixir wrapper for sqlite that can also do a best effort strong-typing…

I've been loosely following this:

https://github.com/spacejam/sled

Since I'm writing in rust it might be a good fit. It's non-SQL but does have some nice features. I would need to figure out how to adapt all the syncer features to that model though. Right now I should just focus on finishing the merging code to be able to release a 1.0 some time soon.

Re: 35% Faster Than The Filesystem (2017)

#134

Post by pdimitar »

pedrocr wrote: post #133
pdimitar wrote: post #131

Earlier quoted context omitted.

I did a research about a month ago and was very disappointed. Basically, outside BerkeleyDB and Firebase semi-embedded mode (since it is still not fully embedded) we're on our own. From then on, you can just use BoltDB or its successor(s) if you don't care about SQL. Which I do, and you seem to as well. At this point I am thinking of developing an Elixir wrapper for sqlite that can also do a best effort strong-typing…

I've been loosely following this: https://github.com/spacejam/sled Since I'm writing in rust it might be a good fit. It's non-SQL but does have some nice features. I would need to figure out how to adapt all the syncer features to that model though. Right now I should just focus on finishing the merging code to be able to release a 1.0 some time soon.

Not to be a complete cynic but IMO you should just use any of the pretty well-done NoSQL K/V stores if you don't mind ditching SQL.

There's plenty that are very well done, BoltDB successors and FoundationDB included.

BTW I've heard very good things about the widely used sqlite Rust libraries (sorry, don't remember the names).

Re: 35% Faster Than The Filesystem (2017)

#135

Post by ImNotTheNSA »

notyourday wrote: post #126
otterley wrote: post #124

Earlier quoted context omitted.

The interviewer didn't like the answer because it didn't answer the implicit question, which is how well you can write your own software. We know that you can load a table into SQLite and have it do the heavy lifting for you. But we're often not looking for software _users_ as much as we're looking for _engineers_. When we interview people, the answers we want often aren't real-world answers because we're trying to g…

> But we're often not looking for software _users_ as much as we're looking for _engineers_. What the applicant illustrated is that he is a much better engineer than the interviewer and the interviewer did not like it. Good software engineering is about leveraging existing robust tools in a new way, not about re-inventing a wheel while making is square. For me that would have been an insta-hire.

I disagree. Applicant should have done more to understand the requirements for the task. That’s a massive part of software engineering, and you would be surprised how much time software engineers will spend solving the wrong problem. He didn’t understand that part of the problem included rolling a novel solution or restricted use of a database. Almost a daily problem for software engineers is extracting this information from customers or systems engineers.

> Good software engineering is about leveraging existing robust tools in a new way, not about re-inventing a wheel while making is square.

Use of external libraries or solutions is often very restrictive (if not banned) in my field. We can use existing internal or known verified solutions. Not everything is so black and white, and not all options are always on the table. They rarely ever are.

I don’t think really think “leveraging existing tools” is a defining part of software engineering... though it certainly is part of it. Software engineering is much more than just coming up with solutions.

Re: 35% Faster Than The Filesystem (2017)

#136

Post by esmi »

otterley wrote: post #124
realshowbiz wrote: post #122

I remember a job interview where the take home project was to parse a log file and search/sort by various top attributes as fast as possible. I thought about writing something from scratch, but I knew that whatever I could optimize in an hour or two would be nothing compared to more robust tools that already existed. So I used sqlite to ingest the log once, and query it in various ways to produce the requested report…

The interviewer didn't like the answer because it didn't answer the implicit question, which is how well you can write your own software. We know that you can load a table into SQLite and have it do the heavy lifting for you. But we're often not looking for software _users_ as much as we're looking for _engineers_. When we interview people, the answers we want often aren't real-world answers because we're trying to g…

If that’s the case then the restrictions should’ve been given in the problem statement and the interviewer should’ve been able to answer why the candidate’s answer was rejected.

Re: 35% Faster Than The Filesystem (2017)

#137

Post by realshowbiz »

otterley wrote: post #124
realshowbiz wrote: post #122

I remember a job interview where the take home project was to parse a log file and search/sort by various top attributes as fast as possible. I thought about writing something from scratch, but I knew that whatever I could optimize in an hour or two would be nothing compared to more robust tools that already existed. So I used sqlite to ingest the log once, and query it in various ways to produce the requested report…

The interviewer didn't like the answer because it didn't answer the implicit question, which is how well you can write your own software. We know that you can load a table into SQLite and have it do the heavy lifting for you. But we're often not looking for software _users_ as much as we're looking for _engineers_. When we interview people, the answers we want often aren't real-world answers because we're trying to g…

Yeah, although imagine if carpenters were interviewed/judged based on the quality by which they could create hammers and drills from scratch.

But it was a good thing overall. I showed me signs that the company has an elevated degree of not invented here culture, and that they aren’t very clear when requesting/scoping work.

https://en.m.wikipedia.org/wiki/Not_invented_here

Re: 35% Faster Than The Filesystem (2017)

#138

Post by bmurphy1976 »

Spivak wrote: post #77
bmurphy1976 wrote: post #14

Earlier quoted context omitted.

>Really the only downside I ever found — at the scale I was operating at — was that it bloats the database backups. Easily solved by having two separate databases. One for dynamic content, one for static files, which is probably good practice regardless.

Doesn't that introduce the desync problem again? Is there something that can enforce consistency across the two databases?

I'm assuming the static and dynamic content don't need to be transactionally integrated. If they do, you could still solve it with two databases, but you'd use one database for the "current" data and another database to store "historical" data.

Also, remember the GP said "small to mid-size."

Re: 35% Faster Than The Filesystem (2017)

#139

Post by pedrocr »

pdimitar wrote: post #134
pedrocr wrote: post #133

Earlier quoted context omitted.

I've been loosely following this: https://github.com/spacejam/sled Since I'm writing in rust it might be a good fit. It's non-SQL but does have some nice features. I would need to figure out how to adapt all the syncer features to that model though. Right now I should just focus on finishing the merging code to be able to release a 1.0 some time soon.

Not to be a complete cynic but IMO you should just use any of the pretty well-done NoSQL K/V stores if you don't mind ditching SQL. There's plenty that are very well done, BoltDB successors and FoundationDB included. BTW I've heard very good things about the widely used sqlite Rust libraries (sorry, don't remember the names).

>Not to be a complete cynic but IMO you should just use any of the pretty well-done NoSQL K/V stores if you don't mind ditching SQL.

My thought process is that the current solution works so I'd only change to a K/V store with the extra complexity if that got me a better API and more of the stack in Rust. Consuming something in Go seems like a pain, while also adding a GC runtime just for this.

>BTW I've heard very good things about the widely used sqlite Rust libraries (sorry, don't remember the names).

I'm using rusqlite, seems to work fine.

Re: 35% Faster Than The Filesystem (2017)

#140

Post by b3b0p »

realshowbiz wrote: post #122

I remember a job interview where the take home project was to parse a log file and search/sort by various top attributes as fast as possible. I thought about writing something from scratch, but I knew that whatever I could optimize in an hour or two would be nothing compared to more robust tools that already existed. So I used sqlite to ingest the log once, and query it in various ways to produce the requested report…

I have come across cases like this in a couple interviews. What I did was do both. In those cases I often got an either an onsite and/or offer immediately.
Post reply on HN