Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQL: One of the most valuable skills

craigkerstiens.com

Post reply on HN
Print view View original

271–280 of 390 posts

Re: SQL: One of the most valuable skills

#271

Post by veritas3241 »

justinclift wrote: post #65
max76 wrote: post #31

I spent a year in a role where 50% of my duties was writing sql reports. These reports where usually between 500 and 1000 lines of sql a pop. Sometimes the runtime of the report was measured in hours, so learning efficient sql was important. The company had a lot of people that had been writing sql for awhile, and there were lots of cool code snippets floating around. I learned a lot in that year. I've moved to writi…

Any recommendations of a place for sharing "extremely advanced" SQL skills? Asking from wanting to make use of such a place, and haven't seen anything like it. So, probably need to bootstrap one instead (etc).

Are you asking for examples of advanced SQL skills?

In my experience, if you can grok lateral joins (aka cross apply), recursive CTEs, window functions, and fully understand all the join types, that's a gold star for understanding SQL!

Re: SQL: One of the most valuable skills

#272

Post by veritas3241 »

eleclady wrote: post #126
conception wrote: post #82

Earlier quoted context omitted.

I think what your are doing is fine. SQL is strongest at answering questions not processing data. I think metabase has the right approach: What question do you want to ask your data? If you want to process and transform your data, I think your tools you are using are great for that.

Not trying to play word games, but what is the difference between answering a question and processing data? Aren't they effectively the same? Using another tool for processing data often results in recreating SQL mechanics at application level. E.g. select this data, retrieve it, loop and if this, then set that, etc. SQL does it way better, guaranteed. Of course that's often required for technical reasons (scalabilit…

Agreed that SQL is amazing at processing data! I would argue that a lot of people are trying to both process their data _and_ ask a question of it in the same statement. Separating those out is really important to make analytics more scalable.

We do >95% of our transformations with pure SQL and the queries are primarily in Looker.

Re: SQL: One of the most valuable skills

#273

Post by ehnto »

whynotminot wrote: post #16
sooheon wrote: post #12

Earlier quoted context omitted.

Speaking as not an SQL guru -- isn't this true for every skill? Theoretically anyone can figure anything out with enough time, the value is of specialists having better mental maps for what solutions apply, when you need them.

> isn't this true for every skill? But you're implying then that all skills are equally hard to get a handle on, and that's just not true. Working with sql is fairly simple even for those who aren't experts. But (as the article mentions) it's not always something a dev has done before or consistently so they'll often feel relieved if there's someone who's confident about it around.

Sure, you can get away with some basic SQL knowledge in simple CRUD applications, but any kind of complex reporting and analytics software with a complex data set is going to take a bit more than simple SQL. It's a career specialization, there are people getting paid astronomical amounts of money for their knowledge of SQL alone, and it's their full time job. Some queries are hundreds of lines long, conjure up entirely new databases from the query itself, and trigger complex procedures across the entire application, all within SQL. It can be as complex as any application logic. Simple SQL is pretty easy to learn, but I wouldn't say you "have a handle" on it if all you've got is the basics.

Re: SQL: One of the most valuable skills

#274

Post by veritas3241 »

xpil wrote: post #215

SQL is for data systems what IP protocol is for networks: it is the neck of the hourglass. You can build plenty of various things on it (the top of the hourglass: applications, reporting frameworks and so on) using various underlying technologies (the bottom: storage engines etc.) but you can't remove the neck without breaking the hourglass. This is why SQL language and IP protocol are two most valuable things in com…

You've intrigued me with this comment mainly because I know SQL quite well but IP not at all. Is wikipedia[0] for IP a good reference to dive in a bit? Would you recommend something else to grok why IP is so fundamental?

[0] https://en.wikipedia.org/wiki/Internet_Protocol

Re: SQL: One of the most valuable skills

#275

Post by _b4ut »

I can't remember a single job interview where I was asked SQL questions. But I can say that I conducted dozens of interviews for developers and almost always asked for some simple queries.

The first question would be for the statement retrieving an entire table without any restrictions or conditions and 50% would fail giving that query. If somebody would manage to actually build a query involving a JOIN, WHERE and a GROUP BY then that interviewee is pretty much hired ...

Re: SQL: One of the most valuable skills

#276

Post by radicalbyte »

slap_shot wrote: post #67

SQL is one the most amazing concepts I've ever experienced. It's nearly 5 decades old and there is no sign of a replacement. We've created countless other technologies to store and process data, and we always seem to try to re-create SQL in those technologies (e.g. Hive, Presto, KSQL, etc). I run a early stage company that builds analytics infrastructure for companies. We are betting very heavily on SQL, and Craigs p…

Certain subsets of the developer community are usually excellent with SQL. It's especially evident with people who have been working with C# for > 10 years because Microsoft have always heavily pushed SQL Server with their developers.

There are plenty of use-cases for SQL with developers: especially in batch processes such as invoicing. A well crafted SQL query can execute exponentially faster than iterative code takes a lot less time to implement.

Re: SQL: One of the most valuable skills

#277

Post by radicalbyte »

nostrademons wrote: post #115
slap_shot wrote: post #67

SQL is one the most amazing concepts I've ever experienced. It's nearly 5 decades old and there is no sign of a replacement. We've created countless other technologies to store and process data, and we always seem to try to re-create SQL in those technologies (e.g. Hive, Presto, KSQL, etc). I run a early stage company that builds analytics infrastructure for companies. We are betting very heavily on SQL, and Craigs p…

"we always seem to try to re-create SQL in those languages (e.g. Hive, Presto, KSQL, etc)." This is largely because of the number of non-programmers who know SQL. Add an SQL layer on top of your non-SQL database and you instantly open up a wide variety of reporting & analytics functionality to PMs, data scientists, business analysts, finance people, librarians (seriously! I have a couple librarian-as-in-dead-trees fr…

SQL is amazing IF you understand it. You need to think in sets of things. It's like functional programming paradigms or recursion; once you really truly "get it" you start to feel like a Jedi master.

Unfortunately the vast majority of SQL users aren't that proficient. It's also fairly hard to learn because it's something that you only pick up with experience and specifically longer time experience with a sufficiently complex data model.

I personally would never have gotten good at SQL if I hadn't stayed in my first two jobs for 5 years each working daily with the data-models and the domain.

Re: SQL: One of the most valuable skills

#278

Post by Dowwie »

I highly recommend the SQL and relational data modeling books by Joe Celko, Bill Kirwin:

    - https://www.goodreads.com/author/show/93496.Joe_Celko
    - https://www.goodreads.com/book/show/7959038-sql-antipatterns

Re: SQL: One of the most valuable skills

#279

Post by jve »

lmm wrote: post #257
ako wrote: post #254

Earlier quoted context omitted.

What tooling do you need? They're just source code files that you can edit with a code editor or an IDE (DataGrip). In addition you need a file to deploy your sql to a database, this can be a sql or shell script, or a gradle file or even make.

> What tooling do you need? Jump-to-definition when something calls something else. Unit tests. Atomic build/install. > this can be a sql or shell script, or a gradle file or even make. Precisely the problem. There's too many different ways to do it, and no consistency.

Atomic build/install - well, you get transactions OOTB. Unit tests - In Microsoft world, 1st class citizen: https://docs.microsoft.com/en-us/sql/ssdt/verifying-database...

Debugging leaves more to wish, but still available in some limited form.

Re: SQL: One of the most valuable skills

#280

Post by andy_boot »

justinclift wrote: post #65
max76 wrote: post #31

I spent a year in a role where 50% of my duties was writing sql reports. These reports where usually between 500 and 1000 lines of sql a pop. Sometimes the runtime of the report was measured in hours, so learning efficient sql was important. The company had a lot of people that had been writing sql for awhile, and there were lots of cool code snippets floating around. I learned a lot in that year. I've moved to writi…

Any recommendations of a place for sharing "extremely advanced" SQL skills? Asking from wanting to make use of such a place, and haven't seen anything like it. So, probably need to bootstrap one instead (etc).

I made this: https://www.windowfunctions.com
Post reply on HN