91–100 of 121 posts
Earlier quoted context omitted.
Yes I like that, much more accurate descriptor than "log", though it is more verbose. In the end it is really just nitpicking I guess, but the discoverability of calling these systems "log services" is very low compared to what they are really capable of.
Somehow I associate "journal" with temporary/transitional data, but that's probably just me thinking of journaling file-systems.
Earlier quoted context omitted.
Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]" Seems reasonable, right? Except "[3] Kafka addressed these durability concerns in version 0.8" So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "…
Or even "we are ready to pay 3 people for a year to build that, if we don't and give you the money instead what can you do ?". The software industry has a real problem contributing to open source, the stuff, you know, allowing them to make money in the first place.
Earlier quoted context omitted.
The classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_. So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good. There are other operational issues people h…
Then configure it using IP addresses. I've been using Zookeeper like this for 4 years in public EC2, swapping machines in and out and it always worked. What's the problem.
Since I'm on AWS EC2, I want to try this:
- Write the logs to local SSD, asynchronously
so as not hold back the http request.
- Have a separate cron job that loops through
the log directory and scoops up all the files.
- The job will then stuff those files into a Kinesis Firehose.
AFAIK, Kinesis Firehose does not require any capacity provisioning,
unlike the Kinesis Streams, so I'm set "for life" (up to 5MB/second)
- The firehose will accumulate the logs and put them into S3.
Hurray unlimited storage!
- S3 will trigger a Lambda.
- Lambda will parse through the log from S3, pull out
interesting properties (IP address, user id, session id, etc) and
stuff them into a DynamoDb table.
- If I need to see data from one user/ip/session I will use DynamoDb
to find the right S3 blobs.
- If I need to reprocess the logs to extract a new piece
of data that I did not foresee earlier, I can run a
map-reduce task
Any opinions? I don't really want to use a SaaS log service because gigabytes per day.
Earlier quoted context omitted.
Would also be interested to hear more about the "deployment nightmares with Zookeeper". For us, it has to be one of the most stable pieces of 3rd party server software we run.
General things: when your session fails to migrate, when log compaction can't keep up with writes, when logs fill up your disk (ZK used to require external cron jobs to prune logs), when watcher notifications arrive late, when the leader's GC exceeds its heartbeat timeout, and so on. Lots of things can and do go wrong with Zookeeper. I suspect it depends on the use case, but building a Zookeeper dependency into any s…
Earlier quoted context omitted.
Mind sharing a bit more?
The classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_. So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good. There are other operational issues people h…
Earlier quoted context omitted.
Java at first is great. Once you get in to large and complex loads, the GC tuning becomes load specific. While tuning GC can be fun (for the first dozen times), the bigger problem happens when your load shifts and your GC tuning becomes subtly wrong. Also even IF you are tuned properly, there is a world of 99-percentile you'll never get to. Additionally, the difference between openJDK and Oracle JDK becomes something…
I find that 'jps' works as a good replacement for pgrep.
The point is you cant use binary name to make your way around anymore. The binary name is 'java'. Standard unix tools just don't work. Yes there are work arounds, but over time things end up being just a little more complex than they should be.
Which means if someone is comparing zookeeper and etcd, well etcd wins major points for being unixy and easy to deploy. Copy 1 binary, done. ZK loses major points here. Gotta make sure the JVM is installed, but do you need the OpenJDK or the Oracle one? if the latter, well apt-get and yum are less helpful.
It's all just little globs of annoying details that add up to be a small pain. Nothing horrible, but if you could make a choice to avoid that, why not?
Basically I guess what I'm saying is there is probably a market for replacing all the Javay distsys stuff with Rust/Go versions. I mean look at etcd!
No, actually Java is a bane to the database world. Cassandra doesn't work, and Hadoop is a complete waste of hosts for most companies (hence the move to Spark.)
We detached this subthread from https://news.ycombinator.com/item?id=11669189 and marked it off-topic.
Speaking of logs, I want to put some logging in place for my web server. I log every single request with extensive details, so I can debug things later if needed. It's several gigabytes per day now, so I can no longer just dump it on disk as I did for the last couple of years. Since I'm on AWS EC2, I want to try this: - Write the logs to local SSD, asynchronously so as not hold back the http request. - Have a separat…
Earlier quoted context omitted.
'plus systems software in Java', mind sharing an explanation?
Java at first is great. Once you get in to large and complex loads, the GC tuning becomes load specific. While tuning GC can be fun (for the first dozen times), the bigger problem happens when your load shifts and your GC tuning becomes subtly wrong. Also even IF you are tuned properly, there is a world of 99-percentile you'll never get to. Additionally, the difference between openJDK and Oracle JDK becomes something…