Live data from Hacker News

"I've isolated the bug to a database query"

thedailywtf.com

41–50 of 173 posts

Re: "I've isolated the bug to a database query"

#41
post #5

Earlier quoted context omitted.

* 1. row * id: 1 select_type: SIMPLE table: lots_of_em type: not_good possible_keys: none key: none key_len: n/a ref: NULL rows: googol filtered: 0 Extra: You're screwed, Do not pass go.

This is an old link, but you just reminded me of: http://howfuckedismydatabase.com

That's awesome, especially this - http://browsertoolkit.com/fault-tolerance.png

Re: "I've isolated the bug to a database query"

#42
post #19

Earlier quoted context omitted.

"Why do I find all the really howling bad stuff so close to the databases?" Because the database and queries against were were coded as an afterthought. Because the programmers ran all the db access tests against tables with 4 rows of data when the customers would windup with 100 million rows in production. Because most undergrad education around databases is poor and antiquated. Plus, it is not SQL focused - much ti…

I work with large databases every day. I cannot tell you the amount of people I've interviewed that weren't able to solve a left join question. Let's leave right and full outer joins out of the discussion--a left join is as advanced as I dare get off the bat in an interview. I've once had a guy I worked with ask for help when he was with a new company. I told him to look at the query plan. He replied, "Yeah, but that…

There's no reason ever to use a right join. I consider those to be a code smell.

Re: "I've isolated the bug to a database query"

#43

Earlier quoted context omitted.

The & and | operators weren't good enough for him? That stuff ends up in databases because on your average team the knowledge of databases drops dramatically when you get past selects and maybe left joins. I worked with a few developers who wrote and lived with queries that were taking 30 seconds each to run on their local machines because they couldn't imagine how to fix them and just waited for me to get assigned t…

I tried caching the hell out of Rails, kept adding indexes to my SQL tables, created 4 MySQL replicas, added more RAM to my linodes, tuned many default parameters in my.cnf, but the website was still slow to a crawl. This parameter saved it all "innodb_buffer_pool_size = 768M" ... how am I supposed to know that Rails creates InnoDB by default, i thought MySQL default was always MyISAM.

This is changing because MyISAM is not really a true database (in the ACID compliant sense of the term). I guess I would have assumed they were MyISAM too but they don't perform differently enough at low usage levels that you would notice it right? How big was this was application?

Re: "I've isolated the bug to a database query"

#44

I once worked on a site where the original developer clearly didn't know joins existed, so if he wanted data from two related tables, he'd get all the required results from table one, then loop through them, one by one, querying table two for the corresponding record. Sometimes this went 3 or 4 tables deep, the site would take nearly a minute to load a table of products.

[deleted]

Re: "I've isolated the bug to a database query"

#45

I once worked on a site where the original developer clearly didn't know joins existed, so if he wanted data from two related tables, he'd get all the required results from table one, then loop through them, one by one, querying table two for the corresponding record. Sometimes this went 3 or 4 tables deep, the site would take nearly a minute to load a table of products.

Yep I did this at a summer job in highschool doing PHP/MySQL. One of the more senior developers saw what I was doing (after I had it working btw) and laughed and told me about joins. So I went to lunch and came back and learned about joins and replaced 50 lines with 5.

Re: "I've isolated the bug to a database query"

#46
post #31

When I've seen this article back in my RSS reader, it reminded me of one particular query that was generated in the application I'm maintaining. My irrational fear of sending too many queries to the database (I've outgrown that in the last 6 years) caused a single query to be generated which was 4KB in size. Which of course is much less than the one in the picture, but still very, very bad. Some so we've refactored t…

4KB? That's nothing. Search for 'media.sql' in any recent Adobe installation media. You'll find 3 MB+ SQL files, containing:

  -(BASE64?) encoded InstallerIcon,
  -(BASE64?) encoded EULAs in various languages
  -GUIDs like {01C3BD72-7371-4472-B179-B4DFE6DDD251}
  -and my personal favorite: a 25 KB XML fragment

Re: "I've isolated the bug to a database query"

#47

Earlier quoted context omitted.

The & and | operators weren't good enough for him? That stuff ends up in databases because on your average team the knowledge of databases drops dramatically when you get past selects and maybe left joins. I worked with a few developers who wrote and lived with queries that were taking 30 seconds each to run on their local machines because they couldn't imagine how to fix them and just waited for me to get assigned t…

I tried caching the hell out of Rails, kept adding indexes to my SQL tables, created 4 MySQL replicas, added more RAM to my linodes, tuned many default parameters in my.cnf, but the website was still slow to a crawl. This parameter saved it all "innodb_buffer_pool_size = 768M" ... how am I supposed to know that Rails creates InnoDB by default, i thought MySQL default was always MyISAM.

MySQL's default may be MyISAM, but it's a bad default. It is a good thing that Rails uses InnoDB by default.

Re: "I've isolated the bug to a database query"

#48
post #29

Sorry, I call BS. I might have just been lucky enough to always work at professional companies and startups where this kind of stuff can never happen. But something tells me there is no reasonable way an SQL query can grow to these proportions.

Most of the stuff on Daily WTF comes from corporate America, often from places where the chief focus of the company is not technology/web.

Don't know what to say, other than it's not only possible, it's not all that uncommon. I can't speak to legitimacy of this particular entry, but the absurdity of it doesn't shock me at all.

Spend some time in the marketing or IT divisions of many Fortune 1000 companies, and you'll understand how Daily WTF is therapeutic for those who have to work in those environments.

Re: "I've isolated the bug to a database query"

#49
Really nasty piece of SQL code, definitely not for human-based processing. What do you think about tools that may decipher and visualize such complex queries in a more structured way, like DBClarity (http://www.microgen.com/dbclarity/)? Have you been using something similar recently?

(disclaimer: I work for mcgn)

Re: "I've isolated the bug to a database query"

#50
I'm pleased that most of the commenters recognize that SQL has a need and is it's own language, for good and bad. I really expected to find a troll popping out "NoSQL rulez" type comments, and the level of understanding of how and where SQL can help is very encouraging.
Post reply on HN