Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You are not Google. You are not Netflix

programmerfriend.com

Post reply on HN
Print view View original

31–40 of 58 posts

Re: You are not Google. You are not Netflix

#31

Post by mgkimsal »

jacques_chester wrote: post #25
mgkimsal wrote: post #22

Earlier quoted context omitted.

postgres. I realize this is heresy to badmouth Postgres here, but it was Postgres (9.0 or 9.1 IIRC). Had a query that selected from table X joining against view X which selected from view A, and view A joins against tables A,B and C, and table B also joins against a view which uses table A and C. This was just bad. But trying to explain to non-tech people how bad it is, when "it used to work", is difficult. It used t…

PostgreSQL has progressively reduced the number of optimisation boundaries that are encountered with views-on-views situations, but you can still wind up telling the database to churn and slosh a whole bunch of data from which you only cherry-pick a tiny portion. I have definitely been guilty of failing to test how my schemata behave with large data sets. Oh then there's ORMs. I've seen ActiveRecord spit out some fra…

> I have definitely been guilty of failing to test how my schemata behave with large data sets.

As have I. It's the dangerous part of the "get it done" approach. And there's no perfect approach - everything is a tradeoff. How much time do you spend dealing with situations that might never happen?

Experience does give you some grounding when making those tradeoff decisions. No, we don't need the architect the application to scale up/down to handle 25k concurrent users in 5 minutes; that's unlikely going to happen. Yes, we do need to spend the extra 2 hours installing and learning a templating system to avoid common XSS pitfalls.

Re: You are not Google. You are not Netflix

#32

Post by geezerjay »

geggam wrote: post #26
empath75 wrote: post #21

I don’t think one needs to plan on being Facebook scale from the beginning, but I do think there is some consideration of scalability that needs to be considered from the start. Maybe you don’t need to think 10 steps ahead but you should at least have plans for 3 or 4 steps ahead of you don’t want to have to rearchitect everything from scratch in a year. It might not be strictly necessary to containerize your app, fo…

Thing is if you want to scale to FB size you _have_ to re architecture / redesign. No one starts an application with the farm model and multiple colos.

You've missed the point. The argument is that eventhough aiming for FB scale is absurd, some considerations should be taken under consideration from the start. We're seeing YAGNI-style arguments being abused to sell the idea that just because it's absurd to think 10 steps ahead it's also absurd to think 2 or 3 steps ahead, which makes no sense at all.

Re: You are not Google. You are not Netflix

#33

Post by geezerjay »

DyslexicAtheist wrote: post #29

Kubernetes, Machine Learning, Microservices, Service-Mesh, are the things that immediately come to (my) mind from such discussions with start-up dev-teams. But somehow all of these are at least a few years old meanwhile. Could it be that the buzzword bingo we experienced in the last years has already peaked or why haven't we seen any new nonsense[1] (at least nothing that I'm aware that is not at least 24 months old)…

Machine learning may have been used and abused as a meaningless buzzword but kubernetes in particular and container orchestration in general is a well established and proven infrastructure technology. If you are not operating a monolith and you need to automatically deploy and scale your service then you have no technically-sound argument to not use this sort of infrastructure.

Re: You are not Google. You are not Netflix

#34

Post by booleandilemma »

mathgeek wrote: post #6

> A developer’s skill is measured in one parameter: how well he writes code. While this could be a valid viewpoint, one thing I’d caution against is that this type of measurement inevitably leads to the commoditization of developers. Rating skill on a single point will lead to “you’re only worth what this metric says you are worth” and eventually to constant competition between developers instead of cooperation. At l…

Actually I’ve found that a lot of development is actually about image projection, or how good you make other people think you are. This is especially true with non-technical folks, who don’t know 1/2 of what developers do anyway.

Our office’s “rockstar” dev has his 3 widescreen monitors tilted vertically and he uses a split keyboard. He’s so quirky and leet.

Re: You are not Google. You are not Netflix

#35

Post by rambojazz »

alfiedotwtf wrote: post #12
OJFord wrote: post #10

Earlier quoted context omitted.

I haven't heard it either, but apparently it's another name for the Toyota system/'lean manufacturing': https://en.m.wikipedia.org/wiki/Just-in-time_manufacturing

Sorry. No, I wasn't referring to JIT Manufacturing... It was merely a tongue-in-cheek to mean just build what's needed right now in front of you, then get it working at scale once there's an actual needs to scale. This is in contrast to starting from day one spending most of your time thinking of scale, then building everything so that it can infinitely scale (along with the complexities that come with it), only to f…

> just build what's needed right now in front of you

What's needed right now, but that doesn't take a complete overhaul when it's time to start growing.

Re: You are not Google. You are not Netflix

#36

Post by sonthonax »

dhuramas wrote: post #19
mgkimsal wrote: post #17

Earlier quoted context omitted.

there's a vast middle ground between 'astronaut architecture' and 'common sense necessities', and knowing where a good cutoff point is is hard without experience. I've met more than a few folks who consider web templating systems to be 'overkill', and have 0 understanding of the risk of xss. Same with sql/db escaping - "I just write the SQL and run it, using all those libraries is just a waste of time", etc. Many of…

Out of curiosity - Which database is this? and is it really taboo to have views joining against multiple other views? If the underlying views were performant- I'd assume the query optimizer would do the right thing(at least 90% of the time). EDIT: I guess it depends - Just did more research and found this [1]. As long as the views don't do unnecessary heavy lifting or joining unnecessary tables, it should be fine. [1…

I think SQL views are a bit of a code smell. They're really only useful in my opinion as a hack, where creating a rather gruesome abstraction within the database is easier than changing the query layer.

I'm currently working on a service written by a bunch of hipster developers who, wrote raw SQL in Python. And they wrote views, lots of really difficult to understand views that are self-joining (on JSON fields no less!). The performance is really unpredictable given the input sizes. It's the only time I'd call performance of a database chaotic because given an input size (beyond certain safe ranges) I have no idea what the performance is.

The problem is from my perspective a key bit of your application logic get hidden, you're then bound to migrations to change it.

Re: You are not Google. You are not Netflix

#37

Post by alfiedotwtf »

tanin wrote: post #28
alfiedotwtf wrote: post #4

I worked on the floor with two teams. One (ours) built our apps and infrastructure the correct way. JIT. Get it working, get it working correct, get it working fast. And given the scale, it did wonders. The other team talked big and gave demos to higher ups and their higher ups and their higher ups. Web scale baby. And with that came the galactic infrastructure and all the buzzword needed in order to run it. About 6…

Could you elaborate more? I'm a bit confused. The first one seems to do the wrong thing because there is no mention of any stakeholder at all. Getting it working first is great though. The second one seems to do the right thing by demoing to stakeholders first. Then, it goes on to make it web scale with buzzwords (which doesn't seem good). The best combination seems to be: 1. demoing to higher up and 2. JIT (I'm not…

In the second case, because sprint velocity was on the up and the walls were covered full of bright colours, it looked like they were doing lots of work. The actually were doing lots of work, but none that the stakeholders cared about.

The second team emphasised the importants of scale and how they were going to solve that problem. The apps to run on top seemed to slip everyone's minds because ticket were being closed hitting KPIs.

Re: You are not Google. You are not Netflix

#38

Post by k1t »

tanin wrote: post #28
alfiedotwtf wrote: post #4

I worked on the floor with two teams. One (ours) built our apps and infrastructure the correct way. JIT. Get it working, get it working correct, get it working fast. And given the scale, it did wonders. The other team talked big and gave demos to higher ups and their higher ups and their higher ups. Web scale baby. And with that came the galactic infrastructure and all the buzzword needed in order to run it. About 6…

Could you elaborate more? I'm a bit confused. The first one seems to do the wrong thing because there is no mention of any stakeholder at all. Getting it working first is great though. The second one seems to do the right thing by demoing to stakeholders first. Then, it goes on to make it web scale with buzzwords (which doesn't seem good). The best combination seems to be: 1. demoing to higher up and 2. JIT (I'm not…

When faced with building a simple web page, Team 1 spends a couple of days producing a single static html page and builds on it as necessary.

Team 2 spends 6 months showing upper management architecture diagrams and flashy mockups. They blow a million dollars on cloud hosting their load balanced web servers. They've got Kafka, ElasticSearch, triple redundant SQL servers, and a 12 person team to run it all. But when asked to present the actual page they announce that nobody on the team knows HTML and they just kind of assumed somebody else would do that part.

Non-technical managers cannot tell the difference between the two teams.

Re: You are not Google. You are not Netflix

#39

Post by spyspy »

Because we’ve railed on “programmers” for so long everyone wants to be a “software engineer” which more often than not leads to equally insane architectures. No you don’t need multi-region availability. No you don’t need to rewrite this in rust. No you don’t need a multi-master db setup. No you don’t need graphql.

Re: You are not Google. You are not Netflix

#40

Post by sanderjd »

I think this article is basically right, but I want to provide a personal perspective on this:

This is why I have found that I prefer working on a widely used product at one of these big companies. I find the most enjoyment in thinking through how to make products work acceptably for huge numbers of users, or how to pick the needles I need out of enormous data sets, or how to structure a codebase such that hundreds of people can contribute to it every day. I have found it very demoralizing in my career when I've been told not to think of user or data or codebase scaling considerations because YAGNI. I've lost a number of debates against just-get-it-done ethos, and mostly I think I was right to lose; I was wrong, YAGNI was right. But the rightness of YAGNI for those products didn't make me enjoy the work any more.

So I think the article is right that most companies don't have these scaling problems, but for me, I want to work for the ones that do. I think this is probably a common sentiment.

Post reply on HN