How We Went from 30 Servers to 2: Go • Hacker News Board

Re: How We Went from 30 Servers to 2: Go

#451

Post by daderam » Wed, Mar 13, 2013, 5:28 PM UTC

Will be great if you can throw some light on "where" and in "what kind of work load" Ruby did not do well? I wonder why Ruby had hogged CPU? As I understand there are 4 kinds of workloads in your system (please correct me if I am wrong)

1. Handling several concurrent client connections. * How many concurrent clients are we talking about here? * Use of blocking sockets will require a separate thread for each connected client? Was Ruby not able to support many threads? * Non-Blocking sockets will require lesser number of threads. Note: I am from a Java background and Java's NIO scales well in handling volumes of concurrent sockets)

2. Handling volumes of data. (Data mining/aggregation) This is an IO/CPU intensive work load. * I dont think you would had done data-processing in ruby. Please correct me if I am wrong. * Scaling the "periodic" process to mine data can be efficiently done by choosing an appropriate storage system with efficient indexes. (MySQL/Postgres/Columnar-Storage/Cassandra etc etc) * Did you run a database? Did it run as a separate tier in a different server?

3. A queue/scheduling framework which would trigger the data-processing in periodic intervals(like google app-engine cron service) * How many "cron-entries" do you have in your system?

4. A CPU intensive algorithm which parses the incoming client's data before persisting in the database. * The incoming data from the client may require CPU intensive processing. (Example: parsing the data, eliminating duplicates etc etc). Did the CPU shoot up because of this?

Will be great if you can throw some light on the volume/scale of the processing involved. Would like to know what kind of work load is "Go" better than Ruby. * Should I consider "Go" if I have lots of String (stream) processing to do? * Should I consider "Go" for handling several concurrent sockets?

Would appreciate any help in understanding this.

Re: How We Went from 30 Servers to 2: Go

#452

Post by kragen » Wed, Mar 13, 2013, 6:04 PM UTC

Shamharoth wrote: post #393

qznc wrote: post #287

Earlier quoted context omitted.

That is somewhat orthogonal. Lisp is better at abstraction than Agda or Coq or Isabelle or any of those ML/Haskell theorem proofers. To maintain the theme: C if you are terrified about performance Lisp if you are terrified about boilerplate Agda if you are terrified about correctness If you are terrified about all of these, then welcome to the world of engineering.

If you are terrified about all these, there is ADA.

Ada gives you neither the performance of C nor the abstraction- and boilerplate-removing power of Lisp nor the provability of Agda. This is why it is not used.

Re: How We Went from 30 Servers to 2: Go

#453

Post by kragen » Wed, Mar 13, 2013, 6:06 PM UTC

pjmlp wrote: post #400

qznc wrote: post #363

Earlier quoted context omitted.

Go does take the last decade of language design by Pike, Thompson, and Griesemer into account. Mostly Pike as I am not sure if the other two guys even did design some language in this time. Personally, I consider gofmt the biggest achievement of Go, if it manages to make that mainstream. While there are equivalent tools for C they are not widely used.

You mean by designing a language that is basically Alef from Plan9(1992) with a few changes? Yeah, really actual.

The English term for "actual" is "up-to-date". The English word "actual" means "not imaginary".

Re: How We Went from 30 Servers to 2: Go

#454

Post by Shamharoth » Wed, Mar 13, 2013, 6:12 PM UTC

kragen wrote: post #452

Shamharoth wrote: post #393

Earlier quoted context omitted.

If you are terrified about all these, there is ADA.

Ada gives you neither the performance of C nor the abstraction- and boilerplate-removing power of Lisp nor the provability of Agda. This is why it is not used.

The performance is pretty close, actually (far more than Go, for example) : http://benchmarksgame.alioth.debian.org/u32/benchmark.php?te...

And yes, it is not used, only in some obscure and low profile projects : https://www.adacore.com/customers

Re: How We Went from 30 Servers to 2: Go

#455

Post by sluu99 » Wed, Mar 13, 2013, 6:32 PM UTC

trungonnews wrote: post #260

yannk wrote: post #248

Earlier quoted context omitted.

tl;dr: I tried Go, but it wasn't like Ruby, so now I'm trying Scala.

Ha. Is there a high performance, compiled, and concurrent friendly language that looks like Ruby? Please share. :)

Here you go: http://cobra-language.com/

Re: How We Went from 30 Servers to 2: Go

#456

Post by pixelbath » Wed, Mar 13, 2013, 6:32 PM UTC

kamaal wrote: post #239

kyllo wrote: post #74

Earlier quoted context omitted.

This is why Joel Spolsky correctly observed that the replacement of C/C++ and functional languages with Java in university CS curricula is a tragedy. If you don't understand pointers or recursion, you are not a polyglot and you cannot pick up just any language in a matter of weeks. The "[language] programmer" (where [language] usually = Java or .NET) trend is a misguided attempt by industry to commoditize programmer…

The problem complicates further with the fact that IDE fever grips you very early on while using a language like Java. When I started with eclipse + Java, within hours my perception of Java was that it was really more of 'fill up the blanks' rather than programming.

This became clear to me when suggesting using a language other than that supported by Visual Studio to one of my previous teams. Their reply of, "Can't do it; no IntelliSense" shocked me completely.

Re: How We Went from 30 Servers to 2: Go

#457

Post by justin66 » Wed, Mar 13, 2013, 7:09 PM UTC

pnathan wrote: post #425

nkohari wrote: post #158

Earlier quoted context omitted.

> I'm very sympathetic to your tech-on-tech-on-tech-on-tech hogwashery promulgated by open sourcers & vendors prosteltyzing their wonderful solutions and the other acculturation factors that permits developers approaches other than trying & seeing & exploring- Are you studying for the SATs, or just trying to make your argument sound more compelling through the use of unnecessarily complex words?

On a forum like HN, where the audience spends a great deal of time reading and engaging in self-improvement, choosing words that are not simplistic should be fine.

I'm going to go out on a limb and guess that even in a community where "acculturation factors" is something people actually say (crit lit? sociology?) it's something that causes people to play that community's equivalent of buzzword bingo.

Re: How We Went from 30 Servers to 2: Go

#458

Post by masklinn » Wed, Mar 13, 2013, 7:55 PM UTC

qznc wrote: post #291

cpprototypes wrote: post #108

Earlier quoted context omitted.

I agree, but I think the age of the language and community is another factor. Java, Python, C++, these are all old languages with decades of history and habits. Newer things like Node.js and Go have no history, no baggage to learn or avoid. I think starting in something with such a clean slate is somewhat easier because there is less ecosystem to learn.

I once heard a quote like this "programming languages are frozen knowledge about software engineering". (Anybody got a clue who said something like this?) New languages usually improve upon older languages by making certain errors impossible by design. For example, Go allows pointers, but not pointer arithmetic. With C we learned pointer arithmetic is often harmful (though sometimes necessary), so Java removed pointe…

> On the other hand, we will find the pitfalls and dark corners of Go over time, but currently we do not know them enough. Go will acquire its own conventions, but we do not know all of them yet. This is the risk of a clean slate.

That's bullshit, Go is not a "clean slate", and a number of design issues of Go are known and have been known from the first release regardless of its designer's refusal to acknowledge them. We know pervasive nullable types are a source of errors, we know shared-memory concurrency is an error-prone default, we know a lack of generics makes userland code painful and generics are hard to retrofit in an established language (and even Go's designers know it, why do you think they build special-case generic collections in the interpreter?), we know allowing implicitly ignoring errors is a bad idea and making it easier to ignore than handle errors also is. These are not recent issues, they're well known and there are a number of possible strategies for handling them.

And Go's worst sin, to me: we know that foisting complexity and repetitiveness upon the user leads to forgetting, and forgetting leads to mistakes. And that's exactly Go's approach to errors, resources management and shared structures mutability. Human error is something you can very reliably bet on, human infallibility... not so much.

Re: How We Went from 30 Servers to 2: Go

#459

Post by steveklabnik » Wed, Mar 13, 2013, 8:26 PM UTC

trungonnews wrote: post #260

yannk wrote: post #248

Earlier quoted context omitted.

tl;dr: I tried Go, but it wasn't like Ruby, so now I'm trying Scala.

Ha. Is there a high performance, compiled, and concurrent friendly language that looks like Ruby? Please share. :)

I personally am looking very heavily into Rust, but if you haven't seen Elixir, you should.

Re: How We Went from 30 Servers to 2: Go

#460

Post by rektide » Wed, Mar 13, 2013, 8:27 PM UTC

mrgreenfur wrote: post #432

porker wrote: post #335

Earlier quoted context omitted.

That's a very narrow assumption as to why someone would express their opinion. The OP might be sharing their experience (in what was a humorous way) purely to be sociable in this discussion... Not everyone is out to convince others.

Righto, he could be here to make himself feel good by using words he doesn't usually get to.

Dare I call into question what you are here for? How do you feel about your contribution- attempting to heap more shame- plays for the whole audience? Is what you are doing of value?