Could be good for things that are often read and rarely written. As soon as multiple people try to update the same file though just use an established db
Json-Base – Database built as JSON files
111–120 of 189 posts
Re: Json-Base – Database built as JSON files
#112Someone at my old company basically did this and put it into production. The first problem he encountered was that multiple connections couldn't both be using the database at a time without clobbering each other. "No problem," he thought, this is a good use case for micro services. A service sitting on top would ensure that there was only one operation being performed at a time. Next, his problem was that the databas…
> Next, his problem was that the database would get corrupt sometimes when something bad happened in the middle of writing the file. I'm not sure i understand how this can happen... unless you try to update JSON in-place (which is a very bad idea for any text-based format), what you do is encode/write the entire JSON from scratch. So either the file is written properly or it isn't written. Honestly from the entire me…
1. A cosmic ray storm turned all ASCII charactees into ECBDIC
2. Lightning struck the 12 V feed and upped the voltage to 10 MV, turning all 0 and 1’s into 6’s
3. Someone spilled a New England Pale Ale on the server
4. The process was assinated by the mysterious killer only known from his modus operandi of leaving OOM written in blood across the syslog
5. Birds nested within the server and fed all the SATA cables to their babies
Seriously though, disk writes aren’t atomic.
Re: Json-Base – Database built as JSON files
#113Earlier quoted context omitted.
I tried. I held a meeting to talk about the code. I found the problems hard to predict and hard to describe. It was decided that after the meeting he would work more on making his code less hacky and more production ready. But the real answer is that our team was very siloed. No one knew what anyone else was doing. The other problem was that he was actually solving real world problems, and he was a very high performe…
It's unfortunate that in this industry, on a lot of teams, "high performer" means "sloppy coder who lets his co-workers finish their project." The problems he encountered with his dumbass solution were EASILY foreseen by an even noob coder. What did he "get done"? How did writing his own shitty version of a database add value to the company? He is good at finishing his own pointless tasks quickly, maybe, but if I was…
If he was in your team it would be your fault. I think you just need a healthy balance of senior/juniors on the same codebase.
As OP said they were siloed from each other and he definitely needed some mentorship. I've seen devs like that turn to incredible coders just after a couple of months of pair programming.
Re: Json-Base – Database built as JSON files
#114You definitely shouldn't use this in production.
Looking at the code there's:
- race conditions everywhere.
- bad and inconsistent formatting, which doesn't help with the
- huge if-else monstrosities.
- Also uses synchronous IO and asynchronous IO randomly.
- Uses try-catch liberally, doesn't check the caught errors, and just re-tries blindly forever in some cases.
If you do any parallel updates/inserts/removals with this "database" you're pretty much guaranteed to lose data. Updates are essentially: 1. read table, 2. make changes, 3. save table. Which at least would work if it was all synchronous.
I know this is going to sound harsh, but building databases is hard for even the most experienced coders, and whoever wrote this is clearly at the other end of that spectrum.
https://github.com/Devs-Garden/jsonbase/blob/master/tables.j...
Re: Json-Base – Database built as JSON files
#115Didn't everyone do this at some point? Sort of like everyone that started C++ in the 90's rolled their own string class. I remember doing this before "JSON" hadn't yet found its acronym (shaking rake ... get off my lawn!)
Re: Json-Base – Database built as JSON files
#116Earlier quoted context omitted.
> If it met the needs of the company, particularly to the desired level at the times these features were requested, I don't think there's a valid critique of the developer's architecture beyond iT's NoT DoNe CoRrEcTlY. I'm pretty sure the cascading series of "his next problem" sentences implies that there were plenty of problems with the architecture that weren't identified ahead of time, and they had to encounter an…
I would say operator friendliness is actually the best reason to roll your own (was clearly not the case here). If you have a system that is less complex, because it meets your use case only and not the competing use cases of every damn engineering outfit that can pay overpaid and underqualified devs to commit to an open sourced codebase, and as a result requires less labor to manage (for example, not using kubernete…
Sure, but there are different levels of "roll your own". Mysql or Pestresql + a text field and a microservice front end for access control and JSON validation (if you don't want to use the included components from those respective projects that handle those for you) is easier and friendlier most of the time than a microservice on top of sqlite on a local disk, which is friendlier than replacing sqlite with BDB, which is probably friendlier than rolling your own storage format.
Once you've abstacted it to a service, your API is what you and your client (should) care about, and many of the arguments for more specialized implementations no longer apply. Personally, I think the only reason I would go with something like sqlite instead of Postgres/Mysql behind a microservice is if I was baking the date into it with each release, so the sqlite data files are shipped with the version released. Even then, I'm not sure there's any reason I would do anything other than sqlite though. Even if I had need of lots of JSON files, I would probably have my build procedure process them into an sqlite file I tested and shipped with, if only because I would then avoid having to deal with all the problems this guy encountered by trying to make his own database.
Re: Json-Base – Database built as JSON files
#117Someone at my old company basically did this and put it into production. The first problem he encountered was that multiple connections couldn't both be using the database at a time without clobbering each other. "No problem," he thought, this is a good use case for micro services. A service sitting on top would ensure that there was only one operation being performed at a time. Next, his problem was that the databas…
> Next, his problem was that the database would get corrupt sometimes when something bad happened in the middle of writing the file. I'm not sure i understand how this can happen... unless you try to update JSON in-place (which is a very bad idea for any text-based format), what you do is encode/write the entire JSON from scratch. So either the file is written properly or it isn't written. Honestly from the entire me…
Here, have a read. Warning: if you haven’t done low-level development, you might walk away wondering if your entire life has been a lie.
Re: Json-Base – Database built as JSON files
#118Earlier quoted context omitted.
It's unfortunate that in this industry, on a lot of teams, "high performer" means "sloppy coder who lets his co-workers finish their project." The problems he encountered with his dumbass solution were EASILY foreseen by an even noob coder. What did he "get done"? How did writing his own shitty version of a database add value to the company? He is good at finishing his own pointless tasks quickly, maybe, but if I was…
If he was in your team it would be your fault. I think you just need a healthy balance of senior/juniors on the same codebase. As OP said they were siloed from each other and he definitely needed some mentorship. I've seen devs like that turn to incredible coders just after a couple of months of pair programming.
Yeah that is a great point, I would have never let it get to this point.
That is 100% a pet peeve of mine: Places that hire perfectly capable jr. engineers and then fail to give them the support they need.
Re: Json-Base – Database built as JSON files
#119Someone at my old company basically did this and put it into production. The first problem he encountered was that multiple connections couldn't both be using the database at a time without clobbering each other. "No problem," he thought, this is a good use case for micro services. A service sitting on top would ensure that there was only one operation being performed at a time. Next, his problem was that the databas…
How did you go about porting the database code to something more sane? (just assuming you did)
I imagine if this database system is contained well enough, it shouldn't be so difficult to swap its internals with something else. Especially if it's all just JSON-like.
Re: Json-Base – Database built as JSON files
#120You definitely shouldn't use this in production. Looking at the code there's: - race conditions everywhere. - bad and inconsistent formatting, which doesn't help with the - huge if-else monstrosities. - Also uses synchronous IO and asynchronous IO randomly. - Uses try-catch liberally, doesn't check the caught errors, and just re-tries blindly forever in some cases. If you do any parallel updates/inserts/removals with…
[deleted]