111–120 of 121 posts
Wonder how it compares to Facebook Scribe?
This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware. On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing di…
If you read the CS papers surrounding distributed systems, you will often see the notion of a 'journal' or a 'log', meaning an append-only structure, which typically contains numerous agreed-upon facts.
Earlier quoted context omitted.
Rumors of Scala's dominance at Twitter are slightly exaggerated. While it's true that the non-revenue-related backend is almost all Scala, on the ads eng side, it's almost 100% Java. Which language a new project is written in has more to do with who's writing it than anything else.
Actually I just wondered cause of Finangle. I thought that their projects uses the RPC service heavily.
Earlier quoted context omitted.
General things: when your session fails to migrate, when log compaction can't keep up with writes, when logs fill up your disk (ZK used to require external cron jobs to prune logs), when watcher notifications arrive late, when the leader's GC exceeds its heartbeat timeout, and so on. Lots of things can and do go wrong with Zookeeper. I suspect it depends on the use case, but building a Zookeeper dependency into any s…
As an operator I do feel that pain but the alternative of every piece of software that needs distributed coordination reimplementing it, likely poorly, it isn't great either.
Ideally, these kinds systems should be built such that the distributed coordination piece is pluggable and the implementation can be chosen based on deployment concerns.
Earlier quoted context omitted.
Then configure it using IP addresses. I've been using Zookeeper like this for 4 years in public EC2, swapping machines in and out and it always worked. What's the problem.
How is this an improvement over using DNS? When you swap in a new EC2 instance it has different IP addresses from the old instance. So now not only do you have to restart your other Zookeeper servers in the cluster, you have to update their configuration then restart them.
Speaking of logs, I want to put some logging in place for my web server. I log every single request with extensive details, so I can debug things later if needed. It's several gigabytes per day now, so I can no longer just dump it on disk as I did for the last couple of years. Since I'm on AWS EC2, I want to try this: - Write the logs to local SSD, asynchronously so as not hold back the http request. - Have a separat…
I've heard some bad things about Kinesis in general. Why not have your cron job just put the logs onto S3 directly? I've used Lambda a bit. The debugging process can be a pain, since you're forced the upload a ZIP file, and if your code times out Lambda doesn't give you any traceback to indicate what happened. There's also a maximum run time of each Lambda invocation, which I believe is 5 minutes. Is there a chance y…
For Kinesis, I planned to use Firehose, not Streams (the latter have to be provisioned, which I was hoping to avoid). The firehose could put data into S3 for me, and S3 would trigger lambda. However I just realized that S3 will only make 3 attempts to invoke the Lambda, so that pretty much rules out this part of my design - the data will not get lost, but it will not get indexed either. I may run map/reduce later, but I don't want to be dependent on doing that to pick the loose ends.
These are just server logs, they don't affect business continuity. Still I wouldn't want to be sitting there and wondering "is this user having connectivity problems, or did I just lose a pile of logs?".
I could probably put the files directly into S3, got carried away stacking my AWS features together. :) I'd need to be more careful with batching, so as not to create batches too small or too large. Perfectly doable, though Kinesis Firehose already does that for me. Plus in case of the EC2 instance death I will lose the current batch with the hand-rolled solution, but not with Firehose.
So I guess I should just put a batch of data directly to S3 and send an SQS message to make sure that it's indexed properly, then delete the local files.
I really don't want to pick up another thing that I have to understand and manage. Like ELK. Someone who knows ELK will probably have no problem managing it, but my head is full with business domain problems.
Speaking of logs, I want to put some logging in place for my web server. I log every single request with extensive details, so I can debug things later if needed. It's several gigabytes per day now, so I can no longer just dump it on disk as I did for the last couple of years. Since I'm on AWS EC2, I want to try this: - Write the logs to local SSD, asynchronously so as not hold back the http request. - Have a separat…
We just went through an extensive ecosystem survey and reached an identical design, which we're currently implementing. So far, so good! AWS has aws-kinesis-agent, which you can deploy for the log aggregation bit, which is very easy to use.
Earlier quoted context omitted.
Seriously this seems like a common pattern in open source: [big company] could just improve [X] but instead builds something from the ground up.
Sometimes you need to let talented engineers build things from the ground up, because it's good for them, makes them happy, and stops them from going to work somewhere else. Keeping people with the skills to solve these types of problems around and happy is also great for recruiting and for helping your less capable engineers learn and grow.
actually, trying to let people do work you don't need done, in order to keep them happy, is a pretty rookie manager mistake.
If they aren't passionate about it, and you can't persuade them to do the things you need doing, they aren't the right person for the job.
That is always true, even if they were the right person in the past.
Your goal in that case should be to try find stuff the company needs done that they want to do, and push them to work on that. But if you find nothing, ...
Earlier quoted context omitted.
Java has garbage collection and is about two times slower than C
For good C, which is hard to write - for me and other mortal humans. On the other hand, as a mortal, I can write a-grade-above-code-that-an-idiot-would-write-just code in Java at about 10 times the speed I can write dire-useless-risible C code. The comparison is pointless though, good modern languages like Rust and Julia are developing and LLVM is enabling further development.
Earlier quoted context omitted.
Java has garbage collection and is about two times slower than C
C dont have garbage collection. In some cases Java can outperform C.