Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQL: One of the most valuable skills

craigkerstiens.com

Post reply on HN
Print view View original

251–260 of 390 posts

Re: SQL: One of the most valuable skills

#251

Post by bonesss »

brazzledazzle wrote: post #216
anarazel wrote: post #203

Earlier quoted context omitted.

It's near trivial to exhaust resources when you have SQL access. That's not information disclosure, but it's darn painful if the set of users includes a sizable enough group of bad faith actors.

Some people will even do it in good faith. They’ll tank performance and for some reason it will never cross their mind that they could be causing the issue. Happens with any shared system without rate limiting.

A near canonical example of "Tragedy of the Commons".

That which is no one's responsibility will inevitably trend towards crap, IME.

Re: SQL: One of the most valuable skills

#252

Post by bonesss »

roystonvassey wrote: post #211

Can't agree more. SQL is the one skill that I continue to use across a 15-year span while many others have come and gone - SAS, SPSS, Clementine and name-your-latest-plug-and-play-analytics platform. It is a skill with the highest bang for buck and is a great way to introduce non-programmers to a basic querying language. It is intuitive, easy-to-read and unbelievably versatile.

As an app developer & architect I'm continually surprised by how often I bump into other techs on projects where BIG EFFORT is being put into all kinds of crazy ORM shenanigans, or subsystem development, or integration because people didn't understand how the underlying database platform could be managed through SQL or how the underlying data could be manipulated through SQL.

Developing an data-driven app without understanding how your data-management solution works is like using just one rollerskate. Sure, you can get places, but...

Re: SQL: One of the most valuable skills

#253

Post by haskman »

ako wrote: post #239
haskman wrote: post #234

Earlier quoted context omitted.

Preach! I just started building something like this in Purescript which might interest you - https://github.com/ajnsit/purescript-consequence . It allows writing things like - type PersonRel = Relation (name :: String, age :: Int) type EmployeeRel = Relation (employeeName :: String, managerName :: String) -- Get all managers older than 60 oldManagers :: PersonRel oldManagers = employeeTable # renameField (SProxy :: S…

Resembles R dplyr pipelines using magrittr, although i find the dplyr version easier to read.

I am not familiar with R dplyr.

The code snippet I posted doesn't adequately demonstrate it but the DSL is strongly typed with full type inference at every step. So for example, it knows that the type of the table after the "join" is has a field "age" of type "int", and that filtering on it is okay. Trying to filter on a non-existent field will cause a compile (not runtime) error.

Second, the "steps" are actually plain functions. `#` is simply the reverse function application operator defined as `a # f = f a`. Which means you can abstract operations. For example, you can define a generic function that can take any relation with an age field and filters for age > 60. Thanks to strong types, using it incorrectly (i.e. on a relation without an age field of type integer) would cause an compile error.

Also, the language itself is a deeply embedded DSL. So "ordinary" code also compiles down to SQL. You can do things like loops etc. (for example, create a function that filters over a list of age limits rather than just one) and it will try to generate the best SQL possible.

of course the SQL generation is currently not implemented, so all this is theoretical, but I'm getting there!

Re: SQL: One of the most valuable skills

#254

Post by ako »

lmm wrote: post #244
ako wrote: post #237

Earlier quoted context omitted.

There's no reason you can't store your sql/tsql/plsql in version control. We were doing this 20+ years ago, all code was in csv (we upgraded from rcs to csv), and we had a productized distributed scheduling system that would deploy all the sql scripts every night on a number of oracle databases running from aix, to solaris, to vms, to hpux, to irix, and later linux and windows NT. Similar like you would now use jenki…

There's no reason you can't - but there's limited tooling support and limited worker mindshare for that kind of approach.

What tooling do you need? They're just source code files that you can edit with a code editor or an IDE (DataGrip). In addition you need a file to deploy your sql to a database, this can be a sql or shell script, or a gradle file or even make.

Re: SQL: One of the most valuable skills

#255

Post by sueders101 »

dhbradshaw wrote: post #17

SQL is nice on a surface level and helpful in practice. Having a working intuition for relational databases is valuable on a deep level. I mean having a sense of how to organize the tables, what sizes are large and small, when to add what kind of index and what the size and speed limits are likely to be for a given data structure. That's extremely valuable. BTW, we're preparing to move a postgres database that's a fe…

If you're going Postgres to Postgres I enthusiastically recommend Bucardo.[1] I'm not very technical, but even I managed to get an ~1TB DB successfully replicating using it. In my case it was going from AWS RDS to AWS Aurora. Previously others had tried AWS DMS, but we kept having data integrity issues. And DMS cost us significantly more than Bucardo whose only cost was a medium size EC2 instance and data in/out.

[1] https://bucardo.org/Bucardo/

Re: SQL: One of the most valuable skills

#256

Post by leethargo »

nostrademons wrote: post #196
3pt14159 wrote: post #186

Earlier quoted context omitted.

I understand your perspective, but I look at it a different way. SQL is troublesome to some programming types because it seems alien to ask what you want instead of telling the computer what to do and I find most programmers, especially ASD-types (who I think have an edge for some situations, like writing certain code in a huge org like Google) find this an unfamiliar and strange way of thinking. You're right about s…

Most programmers who are at all familiar with functional programming, DSLs, configuration languages, or optimizing compilers are very well versed with asking the computer for what you want rather than telling it what to do . At least when I was there, this was a very large percentage of Googlers. My issue with SQL is that a programming language should allow you to compose and name building blocks, and then recombine…

There is also the approach of https://querycombinators.org which is inspired by functional programming but results in a declarative language. Queries are composed of functions that can be named and tested separately.

Re: SQL: One of the most valuable skills

#257

Post by lmm »

ako wrote: post #254
lmm wrote: post #244

Earlier quoted context omitted.

There's no reason you can't - but there's limited tooling support and limited worker mindshare for that kind of approach.

What tooling do you need? They're just source code files that you can edit with a code editor or an IDE (DataGrip). In addition you need a file to deploy your sql to a database, this can be a sql or shell script, or a gradle file or even make.

> What tooling do you need?

Jump-to-definition when something calls something else. Unit tests. Atomic build/install.

> this can be a sql or shell script, or a gradle file or even make.

Precisely the problem. There's too many different ways to do it, and no consistency.

Re: SQL: One of the most valuable skills

#259

Post by johnmarinelli »

I'm a frontend web developer, but for a year I worked on a custom ETL system and wrote lots & lots of SQL. I still find myself using knowledge I learned back then and applying it today - for example, being able to review BI's queries is immensely useful. I'm also much less intimidated when I look at backend code, since a lot of it is interfacing with a database in some way.

I think SQL is one of those essential "secondary" skills for developers.

Re: SQL: One of the most valuable skills

#260

Post by clanrebornsx »

slap_shot wrote: post #67

SQL is one the most amazing concepts I've ever experienced. It's nearly 5 decades old and there is no sign of a replacement. We've created countless other technologies to store and process data, and we always seem to try to re-create SQL in those technologies (e.g. Hive, Presto, KSQL, etc). I run a early stage company that builds analytics infrastructure for companies. We are betting very heavily on SQL, and Craigs p…

I disagree you can't use SQL for Google search.

No engineer has ever made this possible.

We've CommonCrawl data but you can't run SQL queries on that data this makes SQL useless.

When smart people have figured out how to do that, come back claiming SQL is important.

Post reply on HN