Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LokiJS – In-memory JavaScript Datastore with Persistence

lokijs.org

Post reply on HN
Print view View original

21–30 of 31 posts

Re: LokiJS – In-memory JavaScript Datastore with Persistence

#21

Post by joeminichino »

marknadal wrote: post #18
andrewaylett wrote: post #2

While "Performance over everything" is obviously hyperbole, I hope they really don't mean it and actually prioritise correctness: that it apparently does actually store and retrieve data suggests that at some level, at least, they do.

(Full disclosure, I work at a database company) Their performance is pretty impressive, but you can get over 30X their speed (30M ops/sec) by caching - see https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... . And this can be done without compromising the storage or integrity of the data. The Observable pattern lets you update caches while you perform writes so the data stays correct, reflecting what is o…

this is actually something we are playing with :) we have a couple of drafts in the pipe but you are welcome to bring some wisdom over to LokiJS - it's free for all to use and for all to contribute, even if it's just on a theoretical level. Cheers

Re: LokiJS – In-memory JavaScript Datastore with Persistence

#22

Post by joeminichino »

egeozcan wrote: post #3

> In-Browser NoSQL db with syncing and persisting Is the syncing part as straightforward as Pouch[1]? I couldn't find anything about syncing on its docs. If a db comes with an easy offline-first and sync-when-possible model with granular permissions, it will be very interesting. Current solutions have (IMHO unacceptable) fixed dependencies on the backend or the architecture. [1]: https://pouchdb.com/

syncing isn't as straight forward mainly because not all DBs have replication protocols like CouchDB does. However, LokiJS provides a Changes API which serializes changes from a set point in time so you can easily process those.

Re: LokiJS – In-memory JavaScript Datastore with Persistence

#23

Post by k__ »

cwmma wrote: post #15
k__ wrote: post #6

Earlier quoted context omitted.

I always wondered what the use-cases for PouchDB are. Who wants to replicate their "whole" database locally?

there is a forthcoming couch plugin that does filtered replication by user

Any more info? Is it meant for security related stuff? I read the filters aren't good for this ATM.

Re: LokiJS – In-memory JavaScript Datastore with Persistence

#24

Post by k__ »

egeozcan wrote: post #13
k__ wrote: post #6

Earlier quoted context omitted.

I always wondered what the use-cases for PouchDB are. Who wants to replicate their "whole" database locally?

You can filter what you are syncing or create a db per user and declare interactions with foreign data in their own instance, which will be executed according to permissions in the backend. What I actually want is MeteorJS with neither MongoDB or the JavaScript backend, or PouchDB without Couch, or Jaydata + ASP.NET WebAPI with more open source compatibility...

Have you tried Horizon?

Re: LokiJS – In-memory JavaScript Datastore with Persistence

#25

Post by marknadal »

joeminichino wrote: post #21
marknadal wrote: post #18

Earlier quoted context omitted.

(Full disclosure, I work at a database company) Their performance is pretty impressive, but you can get over 30X their speed (30M ops/sec) by caching - see https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... . And this can be done without compromising the storage or integrity of the data. The Observable pattern lets you update caches while you perform writes so the data stays correct, reflecting what is o…

this is actually something we are playing with :) we have a couple of drafts in the pipe but you are welcome to bring some wisdom over to LokiJS - it's free for all to use and for all to contribute, even if it's just on a theoretical level. Cheers

Nice, very excited to hear!

Honestly the wisdom I have has just been playing around with benchmark.js on V8 and noticing the Chrome team doesn't do as they claim (although latest Chrome Canary we saw major improvements), while FireFox is a different beast. You've probably done about just as much of this as me.

What we found was that every function call in javascript significantly decreased performance (even if it did nothing) at a quazi logarithmic rate. So on a theoretical level if you're able to reuse function results that are deterministic you can save a ton of overhead. However IDK how micro-optimizing you want to get as it can sacrifice code legibility.

Would love to chat more if you have time, I'll see if I can find your email and send you a message. Mine is (no spaces): a q u i v a @ google's email provider.

Re: LokiJS – In-memory JavaScript Datastore with Persistence

#26

Post by egeozcan »

k__ wrote: post #24
egeozcan wrote: post #13

Earlier quoted context omitted.

You can filter what you are syncing or create a db per user and declare interactions with foreign data in their own instance, which will be executed according to permissions in the backend. What I actually want is MeteorJS with neither MongoDB or the JavaScript backend, or PouchDB without Couch, or Jaydata + ASP.NET WebAPI with more open source compatibility...

Have you tried Horizon?

Yes, but it's still WIP and you need to go all-in on JavaScript IIRC. I don't dislike JavaScript but would like to have options.

Re: LokiJS – In-memory JavaScript Datastore with Persistence

#27

Post by thinkloop »

V8 has some quirky memory mechanics. Object pointers, and similar, are subject to a vague hard limit, loosely defined by `--max-old-space-size` and other related flags, while the objects themselves can extend beyond it. For example, if you had 1 object that contained 15gb text, that would likely be OK, however if you had 1 billion objects, storing 1 character each, that would likely lead to a crash - even if the process memory is below limits. Relatedly the shape and nature of the data matters (nesting, types, etc.), limiting the ability to use the full memory available in various ways. I imagine a lot of it has to do with GC, but I don't think that's the full story.

Have you noticed such quirks building loki? Do you imagine being able to make full use of, say, a 64gb ram server? Is this something being taken into account and optimized in the project somehow? @joeminichino

Re: LokiJS – In-memory JavaScript Datastore with Persistence

#28

Post by ldehaan »

I used this on a internal project for a little while, it's easy to use, and the promises integration is useful, however at high volumes it did very poorly.

I was storing time series data.

I ended up moving to redis which has worked very well as long as I make sure it has enough memory for the date range I am keeping.

I hope this project keeps at it and gets a handle on performance under heavy load.

Re: LokiJS – In-memory JavaScript Datastore with Persistence

#29

Post by joeminichino »

ldehaan wrote: post #28

I used this on a internal project for a little while, it's easy to use, and the promises integration is useful, however at high volumes it did very poorly. I was storing time series data. I ended up moving to redis which has worked very well as long as I make sure it has enough memory for the date range I am keeping. I hope this project keeps at it and gets a handle on performance under heavy load.

hey @ldehaan i'd love to hear more about the shortcomings of Loki, i never had a real-life need for really high volumes so I would greatly appreciate some insights in order to improve it.
Post reply on HN