Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Undebt: How We Refactored 3M Lines of Code

engineeringblog.yelp.com

Post reply on HN
Print view View original

101–110 of 143 posts

Re: Undebt: How We Refactored 3M Lines of Code

#101

Post by huherto »

Kendrick2 wrote: post #14

How do web applications explode out to 3 Million lines of code? Yelp, to me, looks like a typical CRUD app and I would have been surprised if it were more than 100,000 lines of code. The software I develop is pretty large and typically doesn't surpass 40,000 sloc written in-house (i.e. excluding third party libs). Does anyone here maintain such large codebases? Are they truly that big or are people just counting thir…

I have been watching an e-commerce open source product grow over the last four years. The product is getting really big. Part is the amount of features but the other part is that there is a lot of code duplication. Developers are focused on adding features. No one seems to care about removing code. I suppose they do a lot of copy and paste. It just grows and grows.

Re: Undebt: How We Refactored 3M Lines of Code

#102

Post by Cu-chulainn »

bigiain wrote: post #10
wRastel27 wrote: post #5

"...time that could be better spent working on new features and shipping new code" Can we please stop putting forth this idea that features >>> reliable product? The amount of dev time that a company will save from removing technical debt will likely be more than the extra sales the company will get from a new feature. I look forward to the day where the executive team comes to the developers and ask why they are wor…

There is an old Joel On Software blog post I was reading recently that talked about a methodology at Microsoft they called "Zero defects", where fixing known bug _alway_ had priority over working on new features. Here - pont 5 in this post: http://www.joelonsoftware.com/articles/fog0000000043.html

That's exactly what "Lean" is all about. When you spot an issue "pull the Andon cord" and stop the line, deal with the issue then and there. You then potentially conduct Kaizen to address the process to ensure the issue doesn't happen again.

At core all of Agile Software Development is aiming to mirror Lean in the manufacturing world. At the core of which is the concept of minimizing work-in-progress. Because WIP may result in waste if you subsequently find a problem that renders the WIP null and void. Hence the drive for continuous flow to minimize WIP and a "stop the line" mentality to spot and address issues when they arise.

I'm often amazed at how many people "doing" agile through sprints don't understand that the sprint is just an artificial mechanism to minimize WIP.

Not dealing with technical debt when it occurs is just increasing WIP.

Re: Undebt: How We Refactored 3M Lines of Code

#103

Post by huherto »

niftich wrote: post #21
Kendrick2 wrote: post #14

How do web applications explode out to 3 Million lines of code? Yelp, to me, looks like a typical CRUD app and I would have been surprised if it were more than 100,000 lines of code. The software I develop is pretty large and typically doesn't surpass 40,000 sloc written in-house (i.e. excluding third party libs). Does anyone here maintain such large codebases? Are they truly that big or are people just counting thir…

I maintain a line-of-business webapp that could be mistaken for a typical CRUD app, but actually has a lot of business logic enforced in code. That's stuff you don't have to hardcode -- you can pull it out into a 'rules engine' (at the expense of an additional runtime dependency) or push it further down into, say, database stored procedures (I can hear some of you shudder). But for us, the rules rarely change, or cha…

I don't get the rules engine thing. It is still code. I rather have them hardcoded with proper source control and do frequent releases.

May be when you have rules that change every hour back and forth.

Re: Undebt: How We Refactored 3M Lines of Code

#105

Post by vertex-four »

rubber_duck wrote: post #93
vertex-four wrote: post #60

Earlier quoted context omitted.

Even if they changed the standard library from version to version... the result would be that people would stop using the standard library, migration tools or no tools. Nobody really wants to deal with being pinned to a specific minor version and no older/newer - especially libraries.

The correct solution is to version the standard library separately from the language and allow for versioned dependencies, then a new language/VM update doesn't imply a new library that breaks everything, and vice versa. But python grew up in an enviroment where this sort of thing was not practical, and the batteries included is actually a good approach for what python tries to do - scripting. It just doesn't scale w…

Once you have the standard library split apart, you might as well split it up, though, and then you don't really have a standard library any more. You could go the Haskell Platform route... which isn't a wonderful idea.

Re: Undebt: How We Refactored 3M Lines of Code

#106

Post by huherto »

TeeWEE wrote: post #80
crdoconnor wrote: post #67

* Newline at EOF * Double quoted docstring * Remove unused imports. These things are largely cosmetic.

Indeed what i was thinking. A big refactoring is often structural, and changes the program in a bigger way. In Java you can actually easly move classes around by drag-n-drop and they code will refactor. In python this is impossible.

This is a big part of using java for me. I use the refactor tools all the time.

Re: Undebt: How We Refactored 3M Lines of Code

#107

Post by rubber_duck »

vertex-four wrote: post #105
rubber_duck wrote: post #93

Earlier quoted context omitted.

The correct solution is to version the standard library separately from the language and allow for versioned dependencies, then a new language/VM update doesn't imply a new library that breaks everything, and vice versa. But python grew up in an enviroment where this sort of thing was not practical, and the batteries included is actually a good approach for what python tries to do - scripting. It just doesn't scale w…

Once you have the standard library split apart, you might as well split it up, though, and then you don't really have a standard library any more. You could go the Haskell Platform route... which isn't a wonderful idea.

I think if Python was developed from scratch now you would have a very small classic standard library and then stuff like HTTP server/client and JSON parsers would be separate libraries handled by package manager, but since python is a scripting language it would make sense to ship with some packages by default (so not a standard library, but say core packages) - this would let you version the core packages like any other package, but it would still let you run scripts without internet access/need to pull random dependencies for a script run.

Re: Undebt: How We Refactored 3M Lines of Code

#108

Post by sidlls »

skrebbel wrote: post #83
bsaul wrote: post #47

Earlier quoted context omitted.

Your comment opened my eyes. Considering the importance of software development in today's world ( and tomorrow's), the fact that those best practices or code management technics are found mostly in blogs, instead of scientific papers with proper experiments, tells a lot.

I don't follow. What does it tell?

Have you heard of the term/phrase "bro science" with respect to weight lifting? I'd draw an analogy between them.

Clarification: what I mean is there is a lot of ad-hoc stuff in these blogs that may be narrowly true or applicable to a given project or subset of a domain, but generally there is either significant evidence against some of these blogs' contents or the contents themselves have absolutely no formal, rigorous support.

Re: Undebt: How We Refactored 3M Lines of Code

#109

Post by joesb »

rubber_duck wrote: post #93
vertex-four wrote: post #60

Earlier quoted context omitted.

Even if they changed the standard library from version to version... the result would be that people would stop using the standard library, migration tools or no tools. Nobody really wants to deal with being pinned to a specific minor version and no older/newer - especially libraries.

The correct solution is to version the standard library separately from the language and allow for versioned dependencies, then a new language/VM update doesn't imply a new library that breaks everything, and vice versa. But python grew up in an enviroment where this sort of thing was not practical, and the batteries included is actually a good approach for what python tries to do - scripting. It just doesn't scale w…

Then people will complain that you need to specify dependencies just to use leftPad function.

Re: Undebt: How We Refactored 3M Lines of Code

#110

Post by vertex-four »

rubber_duck wrote: post #107
vertex-four wrote: post #105

Earlier quoted context omitted.

Once you have the standard library split apart, you might as well split it up, though, and then you don't really have a standard library any more. You could go the Haskell Platform route... which isn't a wonderful idea.

I think if Python was developed from scratch now you would have a very small classic standard library and then stuff like HTTP server/client and JSON parsers would be separate libraries handled by package manager, but since python is a scripting language it would make sense to ship with some packages by default (so not a standard library, but say core packages) - this would let you version the core packages like any…

That is pretty much the Haskell Platform. Haskell people have issues with it because it contains a whole bunch of really useful packages, but essentially pins them at old versions globally. Upgrading them, then, is a global thing, and if someone depends on an older version... you're stuck.

The solution to this, of course, is sandboxing - never install libraries globally, only on a project-specific basis, and then their dependencies can override global ones. But it's fiddly to get the UX right - in Python, managing that involves two separate tools. And you'd need to create a project directory to get a repl with some library in it, unless you had extensions to the repl to install things temporarily - and then you'd likely have two separate UXes for installing things, whether you're doing it for a REPL or a project.

Post reply on HN