Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What I Wish I Had Known Before Scaling Uber [video]

youtube.com

Post reply on HN
Print view View original

271–280 of 284 posts

Re: What I Wish I Had Known Before Scaling Uber [video]

#271

Post by maxlamb »

fail2fail2ban wrote: post #89
maxlamb wrote: post #84

Earlier quoted context omitted.

Although companies like Snowflake are bridging the gap between the two camps, with products that can natively ingest JSON and Avro, and makes it queryable via SQL allowing for joins etc.

It's possible to use SQL to retrieve data from a text file; that's not the same as using SQL backed by an actual relational database engine (and a well designed schema) that has been perfected over many decades.

Snowflake does have a relational database engine, it was actually founded by two Oracle engineers (who had worked at Oracle 15+ years)

Re: What I Wish I Had Known Before Scaling Uber [video]

#272

Post by barrkel »

iamleppert wrote: post #270
willejs wrote: post #245

Earlier quoted context omitted.

Saying something like '...they have very poor engineers working for them.' is pretty unfounded, and naive. It's easy to say that having 1700 services is overkill from our point of view, but we don't know the complete architecture, problems being solved and environment that they operate in. One thing to possibly consider is that once you have set up tooling for a platform; logging, request tracing, alerting, orchestra…

This wasn't a derivation from the repo count, it was in response to the debug part of the talk where he was talking about fan out. Some app was iterating through a list of things and for each thing was making a network request, instead of figuring out what it needed up front. It's either complete laziness or incompetence, probably the later. I assume with any company that has that many services their culture is partl…

I agree that it is utterly stupid. However, it is almost certainly a political / interpersonal / interteam issue. Very probably, the person making the calls couldn't or didn't convince the service they were calling that a bulk access mechanism was required.

It probably doesn't help that canonical REST as usually applied to APIs doesn't really have a single well-known pattern for this.

Re: What I Wish I Had Known Before Scaling Uber [video]

#273

Post by barrkel »

threeseed wrote: post #187
agentultra wrote: post #184

Earlier quoted context omitted.

> I have no idea why people keep thinking microservices is all about scalability. It's an aspect. It's often the beginning of a micro-service migration story in talks I've heard. > Microservices is all about taking a big problem and breaking it down into smaller components ... and putting the network between them. It's all well and good but the tradeoffs are not obvious there either. Most engineers I know who claim t…

You seem to be repeating these weird myths that have no basis in reality. You don't need to be an expert in distributed systems to use microservices. It's literally replacing a function call with an RPC call. That's it. If you want to make tracing easier you tag the user request with an ID and pass it through your API calls or use something like Zipkin. But needing formal verification in order to test your architectu…

RPC vs PC means introduces a greatly increased probability of random failure. If you're trying to do something transactional, you're in a different world of pain.

Re: What I Wish I Had Known Before Scaling Uber [video]

#274

Post by aurelianito »

dozzie wrote: post #269
aurelianito wrote: post #266

Earlier quoted context omitted.

Yes, but now we actually try to automate in a systematic way; as opposed to a single admin hacking some custom shell scripts. That's what the dev in devop mean. And that's new, at least it was not a common activity in the XX century.

If by "systematic way" you mean a group of programmers clueless about system administration hacking together some custom Ansible scripts, then I wouldn't call it progress. Sysadmins have tools to automate their work for a long, long time (cfengine, bcfg2, even Puppet and Chef predates DevOps hype). DevOps didn't bring anything new to the table.

Is not about doing it right or wrong. Is about the automation that is reused cross-project and the fact that a lot of things that used to require a sysadmin now we automated its job away. If developers implement it correctly or poorly is a different issue.

It is also not about hype (or not). I do agree that the name is posterior to the beginning of the practice, but it is the name that we have.

Re: What I Wish I Had Known Before Scaling Uber [video]

#275

Post by dozzie »

aurelianito wrote: post #274
dozzie wrote: post #269

Earlier quoted context omitted.

If by "systematic way" you mean a group of programmers clueless about system administration hacking together some custom Ansible scripts, then I wouldn't call it progress. Sysadmins have tools to automate their work for a long, long time (cfengine, bcfg2, even Puppet and Chef predates DevOps hype). DevOps didn't bring anything new to the table.

Is not about doing it right or wrong. Is about the automation that is reused cross-project and the fact that a lot of things that used to require a sysadmin now we automated its job away. If developers implement it correctly or poorly is a different issue. It is also not about hype (or not). I do agree that the name is posterior to the beginning of the practice, but it is the name that we have.

Being a sysadmin was always about automation. DevOps brought nothing to the table about that. Neither tools nor paradigm. And the name is just another one for "administering systems", if we keep what you seem to mean by DevOps.

It's still no progress, hacky scripts written by sysadmins or similarly hacky scripts written by programmers, and systematic way of automating tasks was available to sysadmins and was used by them for a long time. DevOps brought nothing new to the table.

Re: What I Wish I Had Known Before Scaling Uber [video]

#276

Post by user5994461 »

Scarblac wrote: post #242
user5994461 wrote: post #169

Earlier quoted context omitted.

One query on a table is the exact same thing as a HTTP GET call on a service.

Yes, but instead of making it a whole new service, you are probably already using a database and can use that service for this functionality as well. But since asking the question I've realized that if your application already needs a huge amount of servers because it simply gets that much traffic, then putting something like this in its own docker instance is probably the simplest way (it might even use postgres ins…

You're both missing the point here. Both things are conceptually equivalent:

- Select(db, somekey, someparameters) [return some db object]

- http_get_query(http://service.com/somekey/someparameters [return some JSON]

They are external (micro)service:

- they both need the target system to be available.

- they both may fail in weird and unexpected ways.

- they both need to handle failure gracefully.

Their usage have different properties:

- A database call need to have a permanent connection pool to the database, usually requiring db user and password.

- A http call is just call and forget. It's a lot easier to use, in any applications, at any time.

Re: What I Wish I Had Known Before Scaling Uber [video]

#277

Post by asimuvPR »

dozzie wrote: post #275
aurelianito wrote: post #274

Earlier quoted context omitted.

Is not about doing it right or wrong. Is about the automation that is reused cross-project and the fact that a lot of things that used to require a sysadmin now we automated its job away. If developers implement it correctly or poorly is a different issue. It is also not about hype (or not). I do agree that the name is posterior to the beginning of the practice, but it is the name that we have.

Being a sysadmin was always about automation. DevOps brought nothing to the table about that. Neither tools nor paradigm. And the name is just another one for "administering systems", if we keep what you seem to mean by DevOps. It's still no progress, hacky scripts written by sysadmins or similarly hacky scripts written by programmers, and systematic way of automating tasks was available to sysadmins and was used by…

I've always had the imoression that a sysadmin ( a position I deeply respect) was more comprehensive than DevOps. DevOps mostly focuses on automating the infrastructure of software development. A sysadmin does that and more.

Re: What I Wish I Had Known Before Scaling Uber [video]

#278

Post by stephansta »

iamleppert wrote: post #230

It amazes me they have 1,700 services. It would be hilarious satire to actually see the description of each. And the debugging scenarios he listed make it sound like they have very poor engineers working for them. Who on earth lets an application get to prod that iterates through a list of items and makes a request for each thing? When did we loose our heads and think such an architecture is sane? The UNIX philosophy…

>When did we loose our heads and think such an architecture is sane?

It's not sane. That's the entire purpose of the talk.

Re: What I Wish I Had Known Before Scaling Uber [video]

#279

Post by traviscj »

cagataygurturk wrote: post #166
0xmohit wrote: post #148

Earlier quoted context omitted.

And another to monitor whether the logs are being aggregated. And yet another one to ensure that the log-aggregation-monitoring-service is working.

Our grandpa's were running banks with 20 cobol programmers, they were not even calling themselves engineers

And it took 1-3 nights to settle.

Re: What I Wish I Had Known Before Scaling Uber [video]

#280

Post by willejs »

iamleppert wrote: post #270
willejs wrote: post #245

Earlier quoted context omitted.

Saying something like '...they have very poor engineers working for them.' is pretty unfounded, and naive. It's easy to say that having 1700 services is overkill from our point of view, but we don't know the complete architecture, problems being solved and environment that they operate in. One thing to possibly consider is that once you have set up tooling for a platform; logging, request tracing, alerting, orchestra…

This wasn't a derivation from the repo count, it was in response to the debug part of the talk where he was talking about fan out. Some app was iterating through a list of things and for each thing was making a network request, instead of figuring out what it needed up front. It's either complete laziness or incompetence, probably the later. I assume with any company that has that many services their culture is partl…

Or your making assumptions based on limited knowledge.
Post reply on HN