What? No bird related name for this project?!
Bird projects got outlawed during the interregnum.
Twitter open-sources a high-performance replicated log service
101–110 of 121 posts
Re: Twitter open-sources a high-performance replicated log service
#102100% Java. I'm wondering. Why doesn't this has any Scala?
Rumors of Scala's dominance at Twitter are slightly exaggerated. While it's true that the non-revenue-related backend is almost all Scala, on the ads eng side, it's almost 100% Java. Which language a new project is written in has more to do with who's writing it than anything else.
[deleted]
Re: Twitter open-sources a high-performance replicated log service
#103Speaking of logs, I want to put some logging in place for my web server. I log every single request with extensive details, so I can debug things later if needed. It's several gigabytes per day now, so I can no longer just dump it on disk as I did for the last couple of years. Since I'm on AWS EC2, I want to try this: - Write the logs to local SSD, asynchronously so as not hold back the http request. - Have a separat…
I've heard some bad things about Kinesis in general. Why not have your cron job just put the logs onto S3 directly?
I've used Lambda a bit. The debugging process can be a pain, since you're forced the upload a ZIP file, and if your code times out Lambda doesn't give you any traceback to indicate what happened. There's also a maximum run time of each Lambda invocation, which I believe is 5 minutes. Is there a chance your parsing may run longer than that? Also, what will you do if you upload some bad code and the parsing fails? Will it be the end of the world if you lose data while you fix the parsing?
Oh, I see you plan on doing map-reduce to re-parse the logs, so maybe that part isn't as big a deal.
You could also consider doing something like rsyslog -> db-of-choice while also rotating the files off to S3 for long-term map-reduces. This is all to ignore the obvious ELK cluster solution, which will give you good data visualization and investigation options, but may be more of a headache to set up and maintain than you are looking for.
Anyway, those are my thoughts. Hope they help.
Re: Twitter open-sources a high-performance replicated log service
#104People talk about Zookeeper as a negative, but in a quorum it's been one of the most stable and reliable pieces of software I've deployed, despite being a bit frustrating to set up / configure. Netflix's Exhibitor[1] is an indispensable addition to it. Also, once you're on the Big Data™ train, a lot of things like to plug into Zookeeper, so it becomes more of a convenience. Kafka, and presumably DL, are at their most…
I found Zookeeper to be pretty easy to learn about and deploy. I also found it got a semi-bad reputation with others because they did not want to learn anything. Two major examples being:
* attempting thousands (tens of thousands?) of simultaneous writes across data centres/continents at the same instant and then saying it was slow. Subscribing all clients to updates on all parts of the tree. The architecture had been grown by people who didn't understand the guarantees and constraints.
* calling it unreliable after arbitrarily moving nodes around without changing connect strings and generally mis-configuring it. Essentially, pointing clients at machines that no longer contained nodes and blaming ZK for this not working.
It could be easier to setup. People often don't want to think about such things at all. But it's also not the hardest, and I found it to be very resilient to node and network failures when deployed correctly.
Re: Twitter open-sources a high-performance replicated log service
#105100% Java. I'm wondering. Why doesn't this has any Scala?
Rumors of Scala's dominance at Twitter are slightly exaggerated. While it's true that the non-revenue-related backend is almost all Scala, on the ads eng side, it's almost 100% Java. Which language a new project is written in has more to do with who's writing it than anything else.
Actually I just wondered cause of Finangle. I thought that their projects uses the RPC service heavily.
Re: Twitter open-sources a high-performance replicated log service
#106Earlier quoted context omitted.
Mind sharing a bit more?
The classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_. So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good. There are other operational issues people h…
This behavior should be fixed in Zk 3.4.7+ and 3.5.0+
Re: Twitter open-sources a high-performance replicated log service
#107Earlier quoted context omitted.
I find that 'jps' works as a good replacement for pgrep.
sure, but jps isnt pkill. The point is you cant use binary name to make your way around anymore. The binary name is 'java'. Standard unix tools just don't work. Yes there are work arounds, but over time things end up being just a little more complex than they should be. Which means if someone is comparing zookeeper and etcd, well etcd wins major points for being unixy and easy to deploy. Copy 1 binary, done. ZK loses…
> Basically I guess what I'm saying is there is probably a market for replacing all the Javay distsys stuff with Rust/Go versions. I mean look at etcd!
Or in the other direction. If your main system runs on the JVM and your sysadmins are used to the JVM tools then having a piece of infrastructure that's just another .jar is wonderful, and C/Rust/Go/Ruby/etc. infrastructure elicits groans. Mixing platforms will always be harder than a common platform. So the infrastructure market depends on where you think the future of applications is.
Re: Twitter open-sources a high-performance replicated log service
#108Earlier quoted context omitted.
Or even "we are ready to pay 3 people for a year to build that, if we don't and give you the money instead what can you do ?". The software industry has a real problem contributing to open source, the stuff, you know, allowing them to make money in the first place.
They did open source their solution. It sounds like you are arguing against parallel development. That happens all the time in open source.
Contributing to other open source projects is very different than to open source your project. NIH is one of the worst problem of open source : companies are spending a lot of money to dev new software instead of improving existing one.
Re: Twitter open-sources a high-performance replicated log service
#109People talk about Zookeeper as a negative, but in a quorum it's been one of the most stable and reliable pieces of software I've deployed, despite being a bit frustrating to set up / configure. Netflix's Exhibitor[1] is an indispensable addition to it. Also, once you're on the Big Data™ train, a lot of things like to plug into Zookeeper, so it becomes more of a convenience. Kafka, and presumably DL, are at their most…
I found Zookeeper to be pretty easy to learn about and deploy. I also found it got a semi-bad reputation with others because they did not want to learn anything. Two major examples being: * attempting thousands (tens of thousands?) of simultaneous writes across data centres/continents at the same instant and then saying it was slow. Subscribing all clients to updates on all parts of the tree. The architecture had bee…
Agreed, some of the problems I encountered were because I did not carefully read the documentation. Some are because a lot of tooling has become easier to deploy in the years since Zookeeper was released, so the bar has been raised.
Which is just to say there's room for improvement. A dependency on Zookeeper is fine if you've already got a configured cluster, and a cognitive speed bump if not.
Re: Twitter open-sources a high-performance replicated log service
#110This looks potentially fantastic. If I could beg one wish from the developers of this (and almost every other project anywhere near this space), though, it would be one tiny piece of documentation: What's your unique ID scheme? Let's say I'm willing to believe[1] that you've got Durable and Consistent down, once messages make it committed in to the system. What's the story for messages on their way in? My application…
> Could the service have committed more messages than it acknowledged?
Yes of course. It may have committed records but not had time to to ACK success.
> Or many less than I've sent?
Yes again. However it will NEVER ACK success for writes which it hasn't committed.
So the persistence story is pretty straightforward (I'll see if we can update docs if this is missing).
If you want to avoid duplication on the write side, you would have to retrieve the last log record-- with id x-- and start adding records again from id x+1.
Fencing + locking combine to provide efficient exclusive access and the fat client (for now) guarantees write ordering.