Live data from Hacker News

SQL: One of the most valuable skills

craigkerstiens.com

31–40 of 390 posts

Re: SQL: One of the most valuable skills

#31
I spent a year in a role where 50% of my duties was writing sql reports. These reports where usually between 500 and 1000 lines of sql a pop. Sometimes the runtime of the report was measured in hours, so learning efficient sql was important. The company had a lot of people that had been writing sql for awhile, and there were lots of cool code snippets floating around. I learned a lot in that year.

I've moved to writing backend code. I'm surprised most of my peers cannot write anything more complicated than a join. Most people are perfectly happy to let the orm do all the work, and never care to dig into the data directly. Every once in a while my sql skills save the day and several people in other departments contact me directly when they need excel files of data in our database we don't have UIs to pull yet.

Re: SQL: One of the most valuable skills

#32

SQL is nice on a surface level and helpful in practice. Having a working intuition for relational databases is valuable on a deep level. I mean having a sense of how to organize the tables, what sizes are large and small, when to add what kind of index and what the size and speed limits are likely to be for a given data structure. That's extremely valuable. BTW, we're preparing to move a postgres database that's a fe…

Ok, we had a MySQL DB around 500GB, and not on Heroku, but we used the AWS Database Migration Service: https://aws.amazon.com/dms/

We had just a few minutes of downtime.

I did have few issues around permissions, but an AWS support person walked me through the process and we got it up and running. The support was great, and we just have standard business support.

EDIT: DMS might be a no go, as jpatokal points out. While Heroku allows external connections, it doesn't allow super user access which, AFAIK, DMS requires.

https://devcenter.heroku.com/articles/heroku-postgresql#conn...

Re: SQL: One of the most valuable skills

#33
post #12

Earlier quoted context omitted.

Speaking as not an SQL guru -- isn't this true for every skill? Theoretically anyone can figure anything out with enough time, the value is of specialists having better mental maps for what solutions apply, when you need them.

> isn't this true for every skill? But you're implying then that all skills are equally hard to get a handle on, and that's just not true. Working with sql is fairly simple even for those who aren't experts. But (as the article mentions) it's not always something a dev has done before or consistently so they'll often feel relieved if there's someone who's confident about it around.

>Working with sql is fairly simple even for those who aren't experts.

Its not though. There are windows, subqueries, joins , various functions, proprietary extensions, partitions, aggregations ect ect. Even if you get all the down, figuring out a performant version of the query is another hurdle, the whole 'simple' abstraction falls apart once you start tuning your query.

Re: SQL: One of the most valuable skills

#34

> Because so few actually know SQL well you can seem more elite than you actually are. Thank you Craig, I'm convinced. Anyone know where best to begin learning SQL?

https://www.sqlteaching.com/ https://blog.codinghorror.com/a-visual-explanation-of-sql-jo... https://pgexercises.com/

More SQL learning awesomeness, thanks zappo2938. It strikes me that the exercises identify the correct result very quickly. They must check against a required result, meaning it is possible for learners to enter a variety of different solutions. Any tips on SQL best practices?

Re: SQL: One of the most valuable skills

#35

> Because so few actually know SQL well you can seem more elite than you actually are. Thank you Craig, I'm convinced. Anyone know where best to begin learning SQL?

The most fun I've had learning SQL. The project is a little dead by still a great project. https://schemaverse.com/tutorial/tutorial.php

If you're on MSSQL server this guy is great. https://blog.sqlauthority.com/. I also love the mug shots he puts everywhere.

Explore some weird things in SQL https://wiki.postgresql.org/wiki/Mandelbrot_set. It's fun to learn how much you can do.

Try to do things you'd normally do in code. Many algorithms can be implimented in SQL. DON'T DO THIS IN PRODUCTION but it's a good way to have fun with SQL and learn a few things.

Find an open source project in an area you're interested in that uses a lot of SQL.

Re: SQL: One of the most valuable skills

#36
post #27

I recently had a candidate that had very good Scala skills and very poor SQL skills. Despite Scala and spark being powerful, you can over complicate a solution in those languages that could otherwise be easily solved in SQL. That’s interesting to me because I see SQL as a fundamental tool that you build on top of and some people these days seems to skip it.

Do you know how good is the SparkSQL optimization in e.g. Databricks? Conversely I have also seen SQL used in overcomplicated ways to replace a more general programming language.

When scripting in SQL I have seen that in certain SQL flavors there is no good way to split a large file into k smaller ones for a fixed k without creating a new column 1-, selecting k new tables modulo the number in the new column, then writing the k files. The way this is implemented is usually naive so the performance is piss poor due to writing and subsequently reading so much data.

If this candidate was very good at Scala, assuming they were a Spark programmer, they could probably do lots of dataframe operations that were as good or better than SQL commands regarding performance no?

Re: SQL: One of the most valuable skills

#37
post #22

The more queries I write on Elasticsearch, the more I value SQL.

Me too. I spent years complaining about databases and all the structure and rules and inflexibility. It was initially great to go to NoSQL backends where suddenly you're free. Now I really miss them :(

Re: SQL: One of the most valuable skills

#38

SQL is nice on a surface level and helpful in practice. Having a working intuition for relational databases is valuable on a deep level. I mean having a sense of how to organize the tables, what sizes are large and small, when to add what kind of index and what the size and speed limits are likely to be for a given data structure. That's extremely valuable. BTW, we're preparing to move a postgres database that's a fe…

That does sound in our wheelhouse. If you drop me an email happy to setup a time to talk.

Re: SQL: One of the most valuable skills

#39
> Learning SQL once will allow you to re-use it heavily across your career span without having to re-learn.

Like all good abstractions, SQL is the practical expression of a mathematical theory. In the case of SQL, you use Zermelo-Fränckel (ZF) set theory to reason about data sets.

While it is easy to come up with merely conjectural implementations for haphazardly doing things -- arbitrary trial and error, really -- it is notoriously hard to develop a systematic axiomatization such as ZF. Mathematics do not change and are hard to defeat. That is why math-based knowledge and skills are able to span an entire career without having to re-learn.

SQL got kicked off in all earnest when David Child's March 1968 paper, "Description of A Set-Theoretic Data Structure", explained that programmers can query data using set-theoretic expressions instead of navigating through fixed structures. In August 1968, Childs published "Feasibility of a Set-Theoretic Data Structure. A General Structure Based on a Reconstituted Set-Theoretic Definition for Relations".

Data independence, by relying on set-theoretical constructs, was explicitly called out as one of the major objectives of the relational model by Ted Codd in 1970 in his famous paper "A Relational Model of Data for Large Shared Data Banks" (Communications of the ACM 13, No. 6, June 1970).

Using Codd's work, Donald Chamberlin and Raymond Boyce at IBM, developed in the early 1970s what would later turn out to become SQL.

Re: SQL: One of the most valuable skills

#40

> Because so few actually know SQL well you can seem more elite than you actually are. Thank you Craig, I'm convinced. Anyone know where best to begin learning SQL?

I wrote this interactive SQL tutorial (you can write SQL in the page, and it tells you if you're right or not) last year! Always looking for feedback:

https://chartio.com/learn/sql/

Post reply on HN