You are not Google. You are not Netflix • Hacker News Board

Re: You are not Google. You are not Netflix

#21

Post by empath75 » Sat, Jun 22, 2019, 2:30 PM UTC

I don’t think one needs to plan on being Facebook scale from the beginning, but I do think there is some consideration of scalability that needs to be considered from the start. Maybe you don’t need to think 10 steps ahead but you should at least have plans for 3 or 4 steps ahead of you don’t want to have to rearchitect everything from scratch in a year.

It might not be strictly necessary to containerize your app, for example, if you’re just running it on a single ec2 instance behind a load balancer, but it’s not a huge amount of work up front and there are some benefits and it means it’ll make it easier to move to ecs or k8s later if you need to.

Re: You are not Google. You are not Netflix

#22

Post by mgkimsal » Sat, Jun 22, 2019, 2:33 PM UTC

dhuramas wrote: post #19

mgkimsal wrote: post #17

Earlier quoted context omitted.

there's a vast middle ground between 'astronaut architecture' and 'common sense necessities', and knowing where a good cutoff point is is hard without experience. I've met more than a few folks who consider web templating systems to be 'overkill', and have 0 understanding of the risk of xss. Same with sql/db escaping - "I just write the SQL and run it, using all those libraries is just a waste of time", etc. Many of…

Out of curiosity - Which database is this? and is it really taboo to have views joining against multiple other views? If the underlying views were performant- I'd assume the query optimizer would do the right thing(at least 90% of the time). EDIT: I guess it depends - Just did more research and found this [1]. As long as the views don't do unnecessary heavy lifting or joining unnecessary tables, it should be fine. [1…

postgres. I realize this is heresy to badmouth Postgres here, but it was Postgres (9.0 or 9.1 IIRC).

Had a query that selected from table X joining against view X which selected from view A, and view A joins against tables A,B and C, and table B also joins against a view which uses table A and C.

This was just bad. But trying to explain to non-tech people how bad it is, when "it used to work", is difficult. It used to work when you had 50-100 records. No one ever tested was this would be like with 30k records and 50 concurrent users all executing the same nested/circular view mess simultaneously.

But the fact that it was postgres is kind of beside the point. I don't know of any mainstream DB that would handle this well.

The short term fix was to do this large query once at the end of a process and cache the results; the set of queries in question were happening on a 'dashboard' view which everyone hit all the time. It would still cause problems with concurrency, because when 80 people would go through a process and get 'done' (think timed training exercises), the queries would still all be running more or less concurrently, and still cause timeouts, but it wasn't as frequent, because people tended to be staggered a bit more as they finished.

Re: You are not Google. You are not Netflix

#23

Post by alfiedotwtf » Sat, Jun 22, 2019, 2:40 PM UTC

mgkimsal wrote: post #17

alfiedotwtf wrote: post #12

Earlier quoted context omitted.

Sorry. No, I wasn't referring to JIT Manufacturing... It was merely a tongue-in-cheek to mean just build what's needed right now in front of you, then get it working at scale once there's an actual needs to scale. This is in contrast to starting from day one spending most of your time thinking of scale, then building everything so that it can infinitely scale (along with the complexities that come with it), only to f…

there's a vast middle ground between 'astronaut architecture' and 'common sense necessities', and knowing where a good cutoff point is is hard without experience. I've met more than a few folks who consider web templating systems to be 'overkill', and have 0 understanding of the risk of xss. Same with sql/db escaping - "I just write the SQL and run it, using all those libraries is just a waste of time", etc. Many of…

> And they did "get it done", "it" was just a steaming pile of crap when it was done. "Why is this taking weeks to do - the previous guy was so much faster?"

When I mean JGID, I also meant not make it a steaming pile of shit :)

Re: You are not Google. You are not Netflix

#24

Post by nemild » Sat, Jun 22, 2019, 2:41 PM UTC

Two articles on this topic:

You are Not Google (my favorite): https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb

A piece from my Notes to a Young Software Engineer that likens engineering media to a bazaar, not a mirror of the industry:

Beware Engineering Media:

https://www.nemil.com/on-software-engineering/beware-enginee...

(Martin Fowler's YAGNI article is also great)

Re: You are not Google. You are not Netflix

#25

Post by jacques_chester » Sat, Jun 22, 2019, 2:42 PM UTC

mgkimsal wrote: post #22

dhuramas wrote: post #19

Earlier quoted context omitted.

Out of curiosity - Which database is this? and is it really taboo to have views joining against multiple other views? If the underlying views were performant- I'd assume the query optimizer would do the right thing(at least 90% of the time). EDIT: I guess it depends - Just did more research and found this [1]. As long as the views don't do unnecessary heavy lifting or joining unnecessary tables, it should be fine. [1…

postgres. I realize this is heresy to badmouth Postgres here, but it was Postgres (9.0 or 9.1 IIRC). Had a query that selected from table X joining against view X which selected from view A, and view A joins against tables A,B and C, and table B also joins against a view which uses table A and C. This was just bad. But trying to explain to non-tech people how bad it is, when "it used to work", is difficult. It used t…

PostgreSQL has progressively reduced the number of optimisation boundaries that are encountered with views-on-views situations, but you can still wind up telling the database to churn and slosh a whole bunch of data from which you only cherry-pick a tiny portion.

I have definitely been guilty of failing to test how my schemata behave with large data sets.

Oh then there's ORMs. I've seen ActiveRecord spit out some frankly batshit insane queries that would stump a room of Einsteins. But somehow PostgreSQL picked it up, chopped it into a plan and got to work plowing through an incredibly wasteful and repetitious query.

Re: You are not Google. You are not Netflix

#26

Post by geggam » Sat, Jun 22, 2019, 2:59 PM UTC

empath75 wrote: post #21

I don’t think one needs to plan on being Facebook scale from the beginning, but I do think there is some consideration of scalability that needs to be considered from the start. Maybe you don’t need to think 10 steps ahead but you should at least have plans for 3 or 4 steps ahead of you don’t want to have to rearchitect everything from scratch in a year. It might not be strictly necessary to containerize your app, fo…

Thing is if you want to scale to FB size you _have_ to re architecture / redesign.

No one starts an application with the farm model and multiple colos.

Re: You are not Google. You are not Netflix

#27

Post by mgkimsal » Sat, Jun 22, 2019, 3:01 PM UTC

alfiedotwtf wrote: post #23

mgkimsal wrote: post #17

Earlier quoted context omitted.

there's a vast middle ground between 'astronaut architecture' and 'common sense necessities', and knowing where a good cutoff point is is hard without experience. I've met more than a few folks who consider web templating systems to be 'overkill', and have 0 understanding of the risk of xss. Same with sql/db escaping - "I just write the SQL and run it, using all those libraries is just a waste of time", etc. Many of…

> And they did "get it done", "it" was just a steaming pile of crap when it was done. "Why is this taking weeks to do - the previous guy was so much faster?" When I mean JGID, I also meant not make it a steaming pile of shit :)

> When I mean JGID, I also meant not make it a steaming pile of shit :)

the tldr of my post is that people often don't know the difference. We also have grown a culture of people promoting "YAGNI", and its easier for some people to dismiss basic ideas with YAGNI.

Yeah, adding 5 language translations to your project on day 1 - YAGNI.

Storing passwords in plaintext? You need something more than that.

Re: You are not Google. You are not Netflix

#28

Post by tanin » Sat, Jun 22, 2019, 3:03 PM UTC

alfiedotwtf wrote: post #4

I worked on the floor with two teams. One (ours) built our apps and infrastructure the correct way. JIT. Get it working, get it working correct, get it working fast. And given the scale, it did wonders. The other team talked big and gave demos to higher ups and their higher ups and their higher ups. Web scale baby. And with that came the galactic infrastructure and all the buzzword needed in order to run it. About 6…

Could you elaborate more? I'm a bit confused.

The first one seems to do the wrong thing because there is no mention of any stakeholder at all. Getting it working first is great though.

The second one seems to do the right thing by demoing to stakeholders first. Then, it goes on to make it web scale with buzzwords (which doesn't seem good).

The best combination seems to be: 1. demoing to higher up and 2. JIT (I'm not familiar with the word, but I get what you mean).

Re: You are not Google. You are not Netflix

#29

Post by DyslexicAtheist » Sat, Jun 22, 2019, 3:14 PM UTC

Kubernetes, Machine Learning, Microservices, Service-Mesh, are the things that immediately come to (my) mind from such discussions with start-up dev-teams.

But somehow all of these are at least a few years old meanwhile. Could it be that the buzzword bingo we experienced in the last years has already peaked or why haven't we seen any new nonsense[1] (at least nothing that I'm aware that is not at least 24 months old)?

[1] it doesn't have to be nonsense but as the article suggests, not everyone is FAANG.

Re: You are not Google. You are not Netflix

#30

Post by Nursie » Sat, Jun 22, 2019, 3:22 PM UTC

mathgeek wrote: post #6

> A developer’s skill is measured in one parameter: how well he writes code. While this could be a valid viewpoint, one thing I’d caution against is that this type of measurement inevitably leads to the commoditization of developers. Rating skill on a single point will lead to “you’re only worth what this metric says you are worth” and eventually to constant competition between developers instead of cooperation. At l…

It's also wrong - if a developer can't communicate, interact well with team mates, sensibly translate requirements to program structure, discover requirements, pitch technological decisions to management, keep up on OSS tech that could prove useful, document well, review others code, etc etc... they're not at the top of their game.

Being able to write code well is just one of the skills needed in a good developer. It might be the main one, but without the rest it's not really enough.