Live data from Hacker News

Jepsen Disputes MongoDB's Data Consistency Claims

infoq.com

371–380 of 416 posts

Re: Jepsen Disputes MongoDB's Data Consistency Claims

#371

Earlier quoted context omitted.

RabbitMQ stores your data, right? Then it's a database! That's pretty much all it takes. A flat file, memory-store, SQL DB, Document store, any of them can be databases if that's where you stick your data! But also no, RabbitMQ and Kafka and the like are clearly message buses and though they might also technically qualify as a DB it would be a poor descriptor.

Ah I see, we are going with “well technically it stores something therefore it is database joke”. Now I’m fully onboard :) Back when I worked in LA my CTO used to joke that most places use Microsoft Outlook as a database and Excel as BI tool.

Excel can be an excellent source of new line-of-business applications.

Many of my employer's applications started out as a shared spreadsheet or Access database.

Our development team worked with the users and built a web application to solve the same problem.

This approach has a lot of advantages:

* The market exists and has an incumbent. There's a lower risk of a write-off.

* The users are open to process changes. You still have to migrate people off of the spreadsheet, though.

* It's easy to add value with reporting, error checking, concurrent access, and access control.

* You can import the existing data to make the transition easier. This will require a lot of data cleaning.

Edited to add the following text from another post.

You can cover most of the requirements with a set of fixed fields.

The last 10% to 20% of the use cases requires custom reports and custom fields.

Users should be able to define their own reports and run them without your involvement.

They should also be able to define custom field types with validation, data entry support, etc.

If your web application has these two features and other advantages then you should be able to replace Excel.

Re: Jepsen Disputes MongoDB's Data Consistency Claims

#372
post #242

Earlier quoted context omitted.

That is documented as a feature. https://www.sqlite.org/different.html under "Manifest typing". SQLite does not use column typing except in integer primary keys.

Fair enough, but just because you call a bug a feature doesn’t mean it’s not a bug. I think most people have realised weak typing is not a good idea in programming languages. It’s especially not a good idea in databases.

Most people have not in fact realized weak typing is not a good idea. I myself vastly prefer strongly typed languages and think they are superior. However there are a huge number of people I work with and know professionally who prefer dynamically typed languages. Weak versus strong typing is a highly subjective opinion. Each one has different costs and benefits and which camp you land in depends in large part on what you value personally.

Re: Jepsen Disputes MongoDB's Data Consistency Claims

#373
post #297
post #64

Earlier quoted context omitted.

There was a time when I advocated for MongoDB with the usual caveats. The ability to easily store and index complex data was of great value. And then in 2015 October, within a week of each other, SQLite and MySQL both learned how to index on expressions and store JSON (SQLite 3.9 2015-10-14, MySQL 5.7 2015-10-21). PostgreSQL added jsonb the year prior in 9.4. At that moment the value of MongoDB for me diminished grea…

Why is storing json in a database important to you? Whenever I see json fields in PostgreSQL/MySQL, I know I'm most likely in for inconsistent data and a world of pain.

Inconsistent json data is exactly why you use a json data type or database.

I know that you, dear developer, would never produce inconsistent data. But lots of other developers do.

It is often the case that you will need to query that inconsistent json data, but either the pain is too great, or the value too low, to normalize that data. Thus, you dump it as is into a json field or database.

Re: Jepsen Disputes MongoDB's Data Consistency Claims

#374
post #328

Earlier quoted context omitted.

Curiously, I just read this: https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-q...

Expensify aren't really scaling SQLite in the way that people would expect. To say it's scaling SQLite is not exactly wrong, but probably gives the wrong impression. The users of their database likely wouldn't see it as SQLite, and they don't use the stock SQLite code. They have their own layer on top that happens to use SQLite as the storage format on disk[1]. This layer means they aren't using full SQLite at the ap…

Agreed, its not a vanilla SQLite but can give you a rough idea of the quality of SQLite codebase.

Re: Jepsen Disputes MongoDB's Data Consistency Claims

#375

Earlier quoted context omitted.

Sends emails when scanned vs returned record ratio is greater than a threshold. Not quite sure how something like this would be different for MySQL/postgres. If an index is missing for you query pattern, wouldn't you create an index in MySQL/postgres?

> If an index is missing for you query pattern, wouldn't you create an index in MySQL/postgres? No. I’d consider adding an index. An index is not free, it comes at a cost and that cost may well be higher than the costs of not having that index. For example, if a reporting query that runs once every few hours is lacking an index, the cost of updating that index on every write (and the disk space/memory used) may well…

Agreed, but a notification about a potential performance issue is not forcing anyone to add an index.

Re: Jepsen Disputes MongoDB's Data Consistency Claims

#376

Earlier quoted context omitted.

> If an index is missing for you query pattern, wouldn't you create an index in MySQL/postgres? No. I’d consider adding an index. An index is not free, it comes at a cost and that cost may well be higher than the costs of not having that index. For example, if a reporting query that runs once every few hours is lacking an index, the cost of updating that index on every write (and the disk space/memory used) may well…

Exactly this. If you connect to your database and run a couple un-indexed queries to explore your data, any query which takes over 100ms will appear for the next 30 days in the "Performance Advisor" tab and it will offer you indexes to create targeted at that query. Just based on a query you did manually one time. Just to note, this is referring to the features for a hosted databased on cloud.mongodb.com, and not som…

I prefer this being shown in the slow queries over not showing it at all , from my experience from people running bad queries manually on production MySQL instances and starting a domino effect.

I guess to each their own.

Re: Jepsen Disputes MongoDB's Data Consistency Claims

#377

Earlier quoted context omitted.

> but there is no MongoDB use in Facebook hasn't been for years Are you sure?

Until very recently I worked there, and at once point I dug quite deeply into the company to see if I could find if it was still in use. I couldn't find documents referencing its present existence, packages to install it, or anything else, so I'm pretty certain that it's not in use.

There are three VIPs actively being used to load balance MongoDB traffic as of this moment.

The docs you likely found on the wiki are dated, but MongoDB is definitely being used in Enterprise Engineering.

Source: I'm currently on the EE traffic team.

It's difficult to make accurate blanket assertions about large technical organizations, which Facebook certainly is.

I am amused by the downvotes that my previous comments received.

Re: Jepsen Disputes MongoDB's Data Consistency Claims

#378
post #362

Earlier quoted context omitted.

I’m not sure if I can even understand what you are talking about or why you said that. If you have pid,{name:’val’, others...}. And an index of name with a million John and one Jane. Good luck getting fast results.

> If you have pid,{name:’val’, others...}. And an index of name with a million John and one Jane. Good luck getting fast results. Fast results: https://dbfiddle.uk/?rdbms=postgres_10&fiddle=dd9370966b1528... . Postgres does actually keep statistics on json columns, but if you've got a functional index on the table and the query uses it then it doesn't matter if there is one "jane" and a million "johns". You're lookin…

No it doesn’t. But what do I know running several clusters in production for a $100 million business? Please read up on the subject before arguing.

Re: Jepsen Disputes MongoDB's Data Consistency Claims

#379
post #320
post #185

Earlier quoted context omitted.

Here is an anecdote. I had to work on a tool that shows what's wrong with an assembly line: missing parts, delays, etc... So that management can take corrective action. Typical "BI" stuff but in a more industrial setting. The company went all out on new technologies. Web front-end, responsive design, "big data", distributed computing, etc... My job was to use PySpark to extract indicators from a variety of data sourc…

Well in these situations, the implicit ask of your company (I've been there myself) is to basically rebuild excel but replace some of the power/flexibility of excel for safety and to remove the risk of error away from front end users (aka move the risk to the back end developers) Unfortunately which specific features of Excel are acceptable to remove are unknown until you have already way over invested into the proje…

I posted elsewhere[0] in this thread about my employer's successful practice of replacing shared spreadsheets with web applications.

Here is another option that we use instead of CSV import.

Our applications support custom reports and custom fields.

Users can define new reports and run them on demand.

They can also define custom field types with validation, data entry support, etc.

This combination provides some of the extensibility of Excel while retaining the advantages of an application.

Edited for wording changes.

[0] https://news.ycombinator.com/item?id=23292374

Re: Jepsen Disputes MongoDB's Data Consistency Claims

#380
post #362

Earlier quoted context omitted.

> If you have pid,{name:’val’, others...}. And an index of name with a million John and one Jane. Good luck getting fast results. Fast results: https://dbfiddle.uk/?rdbms=postgres_10&fiddle=dd9370966b1528... . Postgres does actually keep statistics on json columns, but if you've got a functional index on the table and the query uses it then it doesn't matter if there is one "jane" and a million "johns". You're lookin…

No it doesn’t. But what do I know running several clusters in production for a $100 million business? Please read up on the subject before arguing.

[deleted]
Post reply on HN