Json-Base – Database built as JSON files • Hacker News Board

Re: Json-Base – Database built as JSON files

#161

Post by imtringued » Fri, Jul 03, 2020, 8:57 AM UTC

dnautics wrote: post #99

kbenson wrote: post #84

Earlier quoted context omitted.

> If it met the needs of the company, particularly to the desired level at the times these features were requested, I don't think there's a valid critique of the developer's architecture beyond iT's NoT DoNe CoRrEcTlY. I'm pretty sure the cascading series of "his next problem" sentences implies that there were plenty of problems with the architecture that weren't identified ahead of time, and they had to encounter an…

I would say operator friendliness is actually the best reason to roll your own (was clearly not the case here). If you have a system that is less complex, because it meets your use case only and not the competing use cases of every damn engineering outfit that can pay overpaid and underqualified devs to commit to an open sourced codebase, and as a result requires less labor to manage (for example, not using kubernete…

Rolling something yourself has the benefit that you don't need to teach yourself how the system works because you built it but once you want someone else to join the team this fires back because that person needs to be taught how your database works. If the database in question was widely used then spending time on learning a new database would be worth the investment but if it is only used internally then it's just a waste of time.

Re: Json-Base – Database built as JSON files

#162

Post by globular-toast » Fri, Jul 03, 2020, 9:03 AM UTC

What a stupid idea. SQLite exists and you should use it. I took over a codebase at work that contained an ad hoc implementation of something like this and it was surely the most unprofessional thing I'd ever seen. What are they teaching kids in university these days?

Re: Json-Base – Database built as JSON files

#163

Post by elpescado » Fri, Jul 03, 2020, 9:08 AM UTC

smt88 wrote: post #93

This is both more effort and a worse outcome than just using Postgres with JSONB data types.

...or even just storing JSON in files.

Re: Json-Base – Database built as JSON files

#164

Post by imtringued » Fri, Jul 03, 2020, 9:15 AM UTC

treeman79 wrote: post #68

bvinc wrote: post #16

Someone at my old company basically did this and put it into production. The first problem he encountered was that multiple connections couldn't both be using the database at a time without clobbering each other. "No problem," he thought, this is a good use case for micro services. A service sitting on top would ensure that there was only one operation being performed at a time. Next, his problem was that the databas…

Many many years ago In college, SQL sounded hard. So I built my own database in PHP. Enough said

As long as the stakes are low everything is fine.

Re: Json-Base – Database built as JSON files

#165

Post by trollied » Fri, Jul 03, 2020, 9:33 AM UTC

Just learn SQL. Please!

Re: Json-Base – Database built as JSON files

#166

Post by toyg » Fri, Jul 03, 2020, 9:45 AM UTC

mbreese wrote: post #135

jarym wrote: post #31

What ever you try to do with JSON has probably been tried before with XML. Including XML databases. Now I’ll accept that XML databases has their use (especially if it involved storing and transforming third-party XML) but I can’t think of any good use for this when there’s SO many better options.

Having lived through XML databases and build systems, it’s sad/funny/interesting(?) to see this all play out again. This has all happened before, and it will all happen again.

Same here! I tried pretty hard at the time to work with XML databases. In the end, SQL is just more practical in most circumstances, and easier to reason about. The same will likely happen with this sort of effort.

Re: Json-Base – Database built as JSON files

#167

Post by slifin » Fri, Jul 03, 2020, 9:48 AM UTC

Nesting in your primary data store is an anti-pattern for information workloads where you need to ask questions against your data

Consider:

    {
      "name": "Sam",
      "age": 12,
      "friends" [
      {
        "name": "Tom",
        "age": 15
      }
      ]
   }

Attributes like name and age are properties of a person entity, when placed in a JSON hierarchy something else is happening, the one dimensional relationship the things have between each other is also being saved into the structure

That's dangerous because relationships should be formed on read, not on write, otherwise you concrete all future reads towards whatever it was on write, and if you're particularly sloppy the data gets duplicated which is even worse

The solution is to normalise your data store and use relational algebra to reify relationships at runtime

The problem with mainstream databases is they don't force normalisation, automatic indexing and pulling off attribute level normalisation is unworkable performance wise, so in most teams this doesn't work but this idea does work if you want to try this out learn Datomic

Re: Json-Base – Database built as JSON files

#168

Post by raxxorrax » Fri, Jul 03, 2020, 10:11 AM UTC

boynamedsue wrote: post #52

bvinc wrote: post #16

Someone at my old company basically did this and put it into production. The first problem he encountered was that multiple connections couldn't both be using the database at a time without clobbering each other. "No problem," he thought, this is a good use case for micro services. A service sitting on top would ensure that there was only one operation being performed at a time. Next, his problem was that the databas…

It's easy to get a little laugh from this, but congrats to the guy for exploring. Now he knows first-hand the inordinate challenge and can describe it in detail, but more importantly avoid these hard-learned patterns later.

Every database probably hits similar problems in development. But it is probably better to just use sqlite if you need it to help out.

Re: Json-Base – Database built as JSON files

#169

Post by marci » Fri, Jul 03, 2020, 10:14 AM UTC

jfkebwjsbx wrote: post #153

marci wrote: post #140

Earlier quoted context omitted.

I understand, it was mostly or the-in-the-future-maybe-have-a-web-function-to-sync. I also suggested Couch because outside of western countries, you seldom find laptops or desktops (outside of cities), but smartphones with a recent browser are ubiquitous, so if it worked on IE7 it would run anywhere, even in the most remote area with no/crappy network. And the first time I used couch, my programming "knowledge" was v…

> And the first time I used couch, my programming "knowledge" was very basic HTML (no JS). That probably explains a lot: you have a hammer and everything looks like a nail. CouchDB makes no sense whatsoever for the requirements described.

> you have a hammer and everything looks like a nail.

Actually, that comment made me see things differently on a project that had me scratch my head for the last couple of weeks, so thanks a lot!

Don't know if it's related but I tend to feature creep.

Re: Json-Base – Database built as JSON files

#170

Post by agumonkey » Fri, Jul 03, 2020, 11:06 AM UTC

bvinc wrote: post #49

jamil7 wrote: post #36

Earlier quoted context omitted.

Funny read, why did nobody stop him?

I tried. I held a meeting to talk about the code. I found the problems hard to predict and hard to describe. It was decided that after the meeting he would work more on making his code less hacky and more production ready. But the real answer is that our team was very siloed. No one knew what anyone else was doing. The other problem was that he was actually solving real world problems, and he was a very high performe…

Tech debt 201