91–100 of 109 posts
Earlier quoted context omitted.
No, it is just pass through, they just did a crappy job of handling concurrency.
Maybe they should hire somebody smart like you.
They DID do a crappy job at concurrency and it's fair enough to admit it. It's not personal sometimes it takes a few iterations to get to the right solution.
Earlier quoted context omitted.
the only reason to write something like that in Go is if you're looking to impress a recruiter at (Go). Advise: pick a better tool for your problem.
This is terrible advice. Golang may have a flavor-of-the-week status in some people's minds, but it's a language that really deserves more credit. It is really easy to program and learn, and it does a lot of stuff other languages rely on third party programs for. With each release, nagging issues (namely around garbage collection) are getting resolved, but it's more than production ready. To ignore Golang right now w…
Except Go provides limited value for somebody who already knows .net or a jvm language. And that's a huge chunk of the market.
And the key advantage that Go offers to business is that they can easily hire from a pool of experienced programmers and have them learn Go with little downtime.
Earlier quoted context omitted.
Maybe they should hire somebody smart like you.
To be snide, maybe they should, I am an expert in this topic this is what I do all day every day. They're doing development without understanding how computers work, where the bottlenecks are, or what the maximum theoretical throughput for the use-case is. They ended up with something slightly better than the horrible situation they were in, and are celebrating a inefficient solution as a technical triumph.
Would something like NodeJs+clusters (or any evented IO framework) be a better fit (considering the clusters are stateless, and don't have to talk to eachother)?
If we're talking concurrency/parallelism, would you prefer JVM+Threads/Erlang+Actors over Go? Thanks.
Earlier quoted context omitted.
Maybe they should hire somebody smart like you.
Where did this come from ? They DID do a crappy job at concurrency and it's fair enough to admit it. It's not personal sometimes it takes a few iterations to get to the right solution.
There are two other solutions that spring to mind, which might require quite a bit less code: 1) Take the original code, do the upload exactly in place in the original request (not even spawning a goroutine). However: protect the upload with a semaphore which only allows N-in-flight. My reasoning is, well, if the system operates with low latency when operating nominally, blocking the incoming request isn't too painfu…
Re: Queue: Wouldn't the queue involve locking lest two workers end up trying to work on the same request? To be completely concurrent, I guess one could use a lock-free data structure instead (or implement one on top of something like RocksDB)?
[0] http://ferd.ca/queues-don-t-fix-overload.html
[0] http://engineering.voxer.com/2013/09/16/backpressure-in-node...
Earlier quoted context omitted.
To be snide, maybe they should, I am an expert in this topic this is what I do all day every day. They're doing development without understanding how computers work, where the bottlenecks are, or what the maximum theoretical throughput for the use-case is. They ended up with something slightly better than the horrible situation they were in, and are celebrating a inefficient solution as a technical triumph.
> inefficient solution as a technical triumph They were able to solve their problem in a single process balanced over 4 boxes without ever having to hire someone like you, despite your expertise. Could they have increased throughput? Absolutely. It would have involved a different architecture with more complexity & time, and it also would have relied on skills beyond what was immediately available. I'm guessing their…
Well, Go's runtime allocates 8KB (last I recall) of growable stack space per goroutine. Assuming that their first solution was deployed on the same instance type as their final solution: c4.large (3.75 GB), then they could handle at most ~470,000 outstanding gorountines; assuming that all RAM is used for only gorountines, which is not realistic of course. So their server fell over once it exhausted memory.
This type of memory exhaustion isn't a problem with evented I/O. You have a single thread that responds to async events related to the I/O you're performing.
So, due to the limitations of Go's runtime, they settled upon a worker-pool that allows at most MAX_WORKERS outstanding requests to S3. Not the most efficient solution for this problem. But it works for their use case, for now, and that's what truly matters.
Earlier quoted context omitted.
> inefficient solution as a technical triumph They were able to solve their problem in a single process balanced over 4 boxes without ever having to hire someone like you, despite your expertise. Could they have increased throughput? Absolutely. It would have involved a different architecture with more complexity & time, and it also would have relied on skills beyond what was immediately available. I'm guessing their…
This type of simple I/O bound pass-through problem lends itself extremely well to evented I/O. Conceptually, their first solution was closest to mimicking the benefits of evented I/O, given how Go's runtime works. When a goroutine submits a blocking I/O request, it will yield to another goroutine and wake up later when it can work with the data. So what happened with their first solution? Well, Go's runtime allocates…
Without really knowing the company's needs, I am relying on this paragraph from the post:
While working on a piece of our anonymous telemetry and analytics system, our goal was to be able to handle a large amount of POST requests from millions of endpoints. The web handler would receive a JSON document that may contain a collection of many payloads that needed to be written to Amazon S3, in order for our map-reduce systems to later operate on this data.
Knowing this, I would build it differently.
1. Clients post to S3 Directly 2. Lambda -> Overload business logic, private data, cleanup, spam control etc... 3. Prepare files (64M) for Hadoop 4. Hadoop
There's no reason to have that proxy in the middle, Amazon S3 will handle those millions of requests with no real trouble, I wouldn't throw machines on this process.
Does anybody here have any experience with Go's garbage collection pauses with large stack sizes? I've got a scala app that regularly consumes about 48G of ram, and I'm very happy with the response times during heavy loads like this, but the P99.5 is abysmal because of garbage collection. I've tried tuning it, but it doesn't seem like anything I do helps. I'll probably end up using an Azul JVM but I'm curious how oth…
But I only maintain EE apps on at most 10gb heaps, so I'm not hugely experienced with tuning this. All I can recommend is heap size and CMS thresholds (no G1 exp) set so that at steady state you don't end up hitting full GCs.