Now that people are considering NOSQL will more people consider no-DB • Hacker News Board

Re: Now that people are considering NOSQL will more people consider no-DB

#51

Post by yesbabyyes » Wed, Aug 31, 2011, 4:19 PM UTC

wvenable wrote: post #38

71104 wrote: post #13

answering directly to the subject: i do hope so. SQL too often introduces only a layer of complexity between the server-side application and the storage, while most of times an application could be designed to just use the filesystem, which is a database on its own by the way: it's a big, usually efficient, lookup table that maps keys (file paths) to values (file contents). why store passwords through SQL when a serv…

> each file named with the username and containing his password (without any file format, just the password, possibly hashed or encrypted)? Because your advertisers want to know how many users signed up last month, last six months, and last year. When you only consider one use-case for your data, it's easy to consider using NoSQL or the file system to store your data but in doing so you fail to imagine all the other…

ctime, mtime, atime.

Re: Now that people are considering NOSQL will more people consider no-DB

#52

Post by anamax » Wed, Aug 31, 2011, 4:20 PM UTC

wpietri wrote: post #36

guygurari wrote: post #24

In many applications, data outlives code. This is certainly the case in enterprise applications, where data can sometimes migrate across several generations of an application. Data may also be more valuable to the organization than the code that processes it. While I'm no fan of databases, one obvious advantage is that they provide direct access to the data in a standard way that is decoupled from the specific applic…

The goal is not to replace databases altogether. The goal is to solve some particular problems very well. Last time I used this approach, for example, we mirrored a bunch of data in a traditional SQL store for reporting and ad-hoc querying, things that databases are great at. In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, sch…

> In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, schema improvements become nearly impossible.

It's unclear that multiple applications with direct access to said data make schema improvements any easier.

The obvious solution, copying the data for applications that are using the new schema, pretty much guarantees that one or more of the copies are wrong.

> I also think data integrity is easier to maintain with a system like this. SQL constraints don't allow me to express nearly as much about data integrity as I can in code. Sure, I could use stored procedures, but if I'm going to write code somewhere, I'd rather it be in my app.

How do you guarantee that all of the apps that touch that data use the current version of said code?

Code normalization is as important as data normalization.

Re: Now that people are considering NOSQL will more people consider no-DB

#53

Post by yesbabyyes » Wed, Aug 31, 2011, 4:22 PM UTC

I use Redis for this. I imagine it would be possible to create certain js objects that automatically persist to Redis.

Re: Now that people are considering NOSQL will more people consider no-DB

#54

Post by alexro » Wed, Aug 31, 2011, 4:27 PM UTC

jcromartie wrote: post #50

guygurari wrote: post #24

In many applications, data outlives code. This is certainly the case in enterprise applications, where data can sometimes migrate across several generations of an application. Data may also be more valuable to the organization than the code that processes it. While I'm no fan of databases, one obvious advantage is that they provide direct access to the data in a standard way that is decoupled from the specific applic…

When people set out to design a SQL database, they usually end up updating and deleting records. This is bad because it destroys history, and nothing that you can add to your SQL architecture will fix it at a fundamental level. By basing your system on a journaled event stream, you start with a foundation of complete history retention, and you can build exactly the sort of reporting views you need at any time (say, b…

You can have a design where your previous version of a record gets automatically copied into another table along with the timestamp of the operation. Then you can slice this history however you want. All with no additional app code.

But I wouldn't write off the noDB approach for various transitional data, or data that isn't mean to live long anyway, like tweets.

Re: Now that people are considering NOSQL will more people consider no-DB

#55

Post by overshard » Wed, Aug 31, 2011, 4:35 PM UTC

Any time you have someone who is not a programmer who wants to maintain code, you will have a DB. And this almost all the time.

Re: Now that people are considering NOSQL will more people consider no-DB

#56

Post by wvenable » Wed, Aug 31, 2011, 4:37 PM UTC

yesbabyyes wrote: post #51

wvenable wrote: post #38

Earlier quoted context omitted.

> each file named with the username and containing his password (without any file format, just the password, possibly hashed or encrypted)? Because your advertisers want to know how many users signed up last month, last six months, and last year. When you only consider one use-case for your data, it's easy to consider using NoSQL or the file system to store your data but in doing so you fail to imagine all the other…

ctime, mtime, atime.

Add one other criteria to that -- say location -- and it's already useless.

Re: Now that people are considering NOSQL will more people consider no-DB

#57

Post by moheeb » Wed, Aug 31, 2011, 4:46 PM UTC

wpietri wrote: post #49

eftpotrm wrote: post #47

Earlier quoted context omitted.

I also think data integrity is easier to maintain with a system like this. SQL constraints don't allow me to express nearly as much about data integrity as I can in code. Sure, I could use stored procedures, but if I'm going to write code somewhere, I'd rather it be in my app. Surely I could just as easily say 'Sure, I could use a data access layer in my app but if I'm writing a multi-app database, I'd rather the dat…

Yes, you can definitely do it either way. Years ago as a demo a friend built the heart of a financial exchange in stored procedures. It was very fast, and very reliable. But the same is true about the LMAX system that Fowler describes. Personally, though, I'd much rather do my important coding in a real programming language. Better tools, more libraries, bigger communities, and no vendor lock-in.

Ever hear of ANSI SQL?

Re: Now that people are considering NOSQL will more people consider no-DB

#58

Post by eftpotrm » Wed, Aug 31, 2011, 4:46 PM UTC

wpietri wrote: post #49

eftpotrm wrote: post #47

Earlier quoted context omitted.

I also think data integrity is easier to maintain with a system like this. SQL constraints don't allow me to express nearly as much about data integrity as I can in code. Sure, I could use stored procedures, but if I'm going to write code somewhere, I'd rather it be in my app. Surely I could just as easily say 'Sure, I could use a data access layer in my app but if I'm writing a multi-app database, I'd rather the dat…

Yes, you can definitely do it either way. Years ago as a demo a friend built the heart of a financial exchange in stored procedures. It was very fast, and very reliable. But the same is true about the LMAX system that Fowler describes. Personally, though, I'd much rather do my important coding in a real programming language. Better tools, more libraries, bigger communities, and no vendor lock-in.

Hmmm.

Modern SQL dialects are Turing complete and frankly pretty rich dialects. I know MS SQL Server best so can't speak in detail for others, but the community around that is certainly very substantial. Library support, well, doesn't work quite the same way (yet!) but there's plenty of libraries of code samples available for adapting. Vendor lock-in is a tricky one; by the time you've got to a certain scale of application I tend to think you're programming as much to the API (whether it's the provider's standard API or your own specific API layered over the underlying platform) as to the official 'language'; lock-in can creep up surprisingly easily. Facebook avoided vendor lock-in by writing in open PHP and have since had to write their own PHP compiler to get the performance they needed from the solution they were locked in to.

A former employer used to bulk process EDI order lines in very large quantities. Deduplicating them, dynamically rebatching them according to what was available and what wasn't, updating orders with newer product where customer had specified 'this or better', cross-referencing against multi million row datasets of cataloguing and tagging information to identify how to handle the item. It was a monster; I hate to think about the volumes of data that touched each batch, and with processing orders it absolutely had to have transactional integrity. And yet, written in SQL and running on a very average commodity server, it was actually very fast. The data never left the server until it was ready to do so and all the loads stayed happily internal. The implications of trying to implement it on a NoDB solution - the volumes of data being passed around, the amount of data specific library code the DBMS provides but the underlying language doesn't which would need reading..... It's not pretty.

I don't maintain SQL is the perfect language for everything, that's patently silly. But I do maintain it's a lot more powerful (and with good performance and reliability) than it's given credit for on some very complex operations, and that a lot the reasons people prefer to work in alternatives boil down to lack of understanding. A little learning of what a modern DBMS is capable of can reap huge rewards of saving work in the 'real programming language', as you put it.

Re: Now that people are considering NOSQL will more people consider no-DB

#59

Post by pnathan » Wed, Aug 31, 2011, 4:51 PM UTC

71104 wrote: post #13

answering directly to the subject: i do hope so. SQL too often introduces only a layer of complexity between the server-side application and the storage, while most of times an application could be designed to just use the filesystem, which is a database on its own by the way: it's a big, usually efficient, lookup table that maps keys (file paths) to values (file contents). why store passwords through SQL when a serv…

Well -

I have written several systems that used that sort of approach. Pretty soon you realize you just reimplemented SELECT, and you did a buggy, half-baked job of it.

If you have money to burn on speed & reliability, dropping the database is a good idea. Otherwise, I simply have written too many half-baked hardwired select queries to recommend it.

At this point, I'd rather do some kind of in-memory SQLite with a persistent MySQL/PgSQL backend.

Re: Now that people are considering NOSQL will more people consider no-DB

#60

Post by matwood » Wed, Aug 31, 2011, 4:59 PM UTC

jcromartie wrote: post #50

guygurari wrote: post #24

In many applications, data outlives code. This is certainly the case in enterprise applications, where data can sometimes migrate across several generations of an application. Data may also be more valuable to the organization than the code that processes it. While I'm no fan of databases, one obvious advantage is that they provide direct access to the data in a standard way that is decoupled from the specific applic…

When people set out to design a SQL database, they usually end up updating and deleting records. This is bad because it destroys history, and nothing that you can add to your SQL architecture will fix it at a fundamental level. By basing your system on a journaled event stream, you start with a foundation of complete history retention, and you can build exactly the sort of reporting views you need at any time (say, b…

When people set out to design a data driven application, they usually end up updating and deleting records.

FTFY...

It's not hard to build history into a SQL table design. You can even store events in a...wait for it... SQL database. I have built numerous systems backed by SQL databases that have complete history retention. Answering questions like 'who had id 'X' on this date 3 years ago' are easily solvable with basic standard sql.

I certainly don't believe SQL databases are perfect or the tool for every job, but in many cases they work just fine until you get into very large datasets. Admittedly, I only deal with databases in the 100s of GB range so I have yet to personally run into the scaling problems that a Google or Facebook have and the SQL backed systems I have built work just fine.