Live data from Hacker News

What SQL Analysts Need to Know About Python (2016)

segment.com

101–110 of 115 posts

Re: What SQL Analysts Need to Know About Python (2016)

#101
post #12

I've found that SQL and data frames are pretty exceptional together in combination. I'm a big fan of a Python module called PandaSQL https://pypi.org/project/pandasql . What makes this work so well (for me, at least) is that I can combine pandas operations that transform data frames with SQL operations that transform data frames in the same pipeline, and if I really need to, I can just break it all apart with python…

PandaSQL was last updated 3 years ago. Do you know any other alternatives?

Although a little old, it works out fairly well

Re: What SQL Analysts Need to Know About Python (2016)

#102
post #22
post #14

Python + SQL seems like a good match for many analysis problems, but Excel + SQL is not bad either. I like the ability to combine complex SQL views or SQL functions with Excel pivot functionality, querying the database directly from Excel

Even after learning python, pandas, and SQL I still find myself firing up Excel first when exploring a new problem. The interface is incredibly simple for pasting, manipulating, and visualizing data. The warts really only start appearing when you try to build business applications on top of it and shortcomings with data integrity (amongst other issues) begin creeping in. But as a sketchpad for handling a problem? Exc…

I also avoid pandas for simple stuff. But rather than using Excel I use dplyr and other R tidyverse packages. I think pandas is a little bit more powerful but I find R much easier to use (easier than Excel when you get used to it).

Re: What SQL Analysts Need to Know About Python (2016)

#103

Earlier quoted context omitted.

I’ve seen devs just run select * from table then filter it and sort it in their own code. Then they complain “the database is slow” when it’s spending all its time shipping gigabytes of data they don’t need to them!

From what I encountered, this is generally the case when someone is in the "analysis/reports" mode. Rather than get summary statistics on each column, find number of nulls, etc by writing a sql query, they instead get the data into the Python/R instance, and use general purpose functions, utilities, etc. "Programmers are expensive" statement probably applies here as well. I'm not trying to be defensive here, just say…

From what I encountered, this is generally the case when someone is in the "analysis/reports" mode

I understand this use case, but this is in actual application code!

Re: What SQL Analysts Need to Know About Python (2016)

#104

Earlier quoted context omitted.

Ibis: https://docs.ibis-project.org/ https://docs.ibis-project.org/notebooks/tutorial/2-Basics-Ag...

Dang, Ibis doesn't support Redshift or SQL Server. I'm also having trouble understanding what it really is - it's an entire framework for big data it seems and not just a translator. What I'd really like is just that, something that turns pandas dataframe operation into ANSI SQL. So input pandas2sql('tablename["col"]') -> "select col from tablename". Something really simple to use.

Pandas has from_sql and to_sql methods that are compatible with SQLAlchemy if you insist on using an ORM, that gets you most of the way there...

Re: What SQL Analysts Need to Know About Python (2016)

#105
post #17

Earlier quoted context omitted.

SQL Server doesn't make it easy to calculate median values. There are a number of slightly convoluted ways to do it but it's far from straightforward and usually I end up calculating it in C# if I can.

This legitimately sounds like a great reason not to use SQL Server, to be honest.

It's a big flaw to be sure. I wouldn't use it outside of a dotnet context.

Re: What SQL Analysts Need to Know About Python (2016)

#106
post #52
post #50

Earlier quoted context omitted.

Im surprised ppl dont use ORM libs for this instead..

ORMs are just as capable as any developer of generating bad SQL queries that crush databases and plug network connections.

Yes, but in the example used, all orm libs I know would fetch only a single column or two (often for a single row). While "select *" would fetch everything.

Re: What SQL Analysts Need to Know About Python (2016)

#107
post #79

I had a problem where there was some fixed width data in a sql database - basically someone put mainframe data in a database. There were 3 fields that had multiple entries in them, fixed width delimited. I had to split the fields by width and re-combine them, then also recombine them with another set of data with weird delimiters and rules. It took a day and half (not full time) to figure it out in python. I can't ev…

Turns out a good database is really good at data munging. A solution for postgresql might look something like this:

    CREATE VIEW my_table_improved AS
    SELECT SUBSTRING(the_column, 0, 8) as col1,
           SUBSTRING(the_column, 8, 8) as col2,
           SUBSTRING(the_column, 16, 8) as col3
    FROM my_table
After which you can query my_table_improved as a normal table. col1, col2, and col3 contain the data split out from the_column. In practice you'd probably also want to do some type conversion, e.g. if col1 is supposed to be an integer you can simply update the view to select `CAST(SUBSTRING(the_column, 0, 8) as INTEGER) as col1` instead. In production use you might find this to be slow at which point you will want to create indexes for your new columns. Something like this (adjusted to which queries you're running of course) should work:

    CREATE INDEX ON my_table ((SUBSTRING(the_column, 0, 8)));
Of course this is a lot of work if you're unfamiliar with SQL, and above examples aren't quite complete yet for your use case, but it should get you an idea of how SQL is the exact right tool for the job here. which is somewhat the point of many commenters here: get yourself familiar with SQL and save yourself a metric tonne of work in the future.

An alternate approach might be to write a query that migrates the fixed-width format to a format where each entry is in their own column. The ease of this mostly depends on if applications depend on that column being in that format.

side note: above sql code is untested but should be roughly correct.

Re: What SQL Analysts Need to Know About Python (2016)

#108
post #100
post #22

Earlier quoted context omitted.

Even after learning python, pandas, and SQL I still find myself firing up Excel first when exploring a new problem. The interface is incredibly simple for pasting, manipulating, and visualizing data. The warts really only start appearing when you try to build business applications on top of it and shortcomings with data integrity (amongst other issues) begin creeping in. But as a sketchpad for handling a problem? Exc…

> But as a sketchpad for handling a problem? Excel is pretty hard to beat. I'd recommend trying J. Download here - https://code.jsoftware.com/wiki/System/Installation#Installa... . There is, of course, the famous barrier to entry, but the result, unsurprisingly, is also respectable.

Thanks for the recommendation! I've honestly never heard of J and I'll take a look at it.

Re: What SQL Analysts Need to Know About Python (2016)

#109
post #12

I've found that SQL and data frames are pretty exceptional together in combination. I'm a big fan of a Python module called PandaSQL https://pypi.org/project/pandasql . What makes this work so well (for me, at least) is that I can combine pandas operations that transform data frames with SQL operations that transform data frames in the same pipeline, and if I really need to, I can just break it all apart with python…

PandaSQL was last updated 3 years ago. Do you know any other alternatives?

No, though it has been a while, and your question may prompt me to take a look around the landscape. Not that I have had a problem with pandasql, it does work nicely.

This is one of those technologies that I was hoping would make its way into the framework, kind of how connection pooling for databases was once an external module but is now often built into the various web frameworks. DataFrames with common columns are such a natural match to relational tables that it seems that a way to call SQL would be (ok, in my opinion, should be) part of pandas (I think that this is the case for R data frames).

I also just really enjoy writing SQL. To some extent, this is a personal preference - some people just mentally line up with certain ways of thinking. There are things that are clearly better to do in python, things that are clearly better to do in SQL, and then a grey area. I personally lean much farther to the SQL side of that grey area, but there's certainly nothing incorrect about going the other direction.

Re: What SQL Analysts Need to Know About Python (2016)

#110
post #107
post #79

I had a problem where there was some fixed width data in a sql database - basically someone put mainframe data in a database. There were 3 fields that had multiple entries in them, fixed width delimited. I had to split the fields by width and re-combine them, then also recombine them with another set of data with weird delimiters and rules. It took a day and half (not full time) to figure it out in python. I can't ev…

Turns out a good database is really good at data munging. A solution for postgresql might look something like this: CREATE VIEW my_table_improved AS SELECT SUBSTRING(the_column, 0, 8) as col1, SUBSTRING(the_column, 8, 8) as col2, SUBSTRING(the_column, 16, 8) as col3 FROM my_table After which you can query my_table_improved as a normal table. col1, col2, and col3 contain the data split out from the_column. In practice…

If it was just one column, that would be easy enough. Or if one object was on one row, it would be easy enough. In my case, up to 6 rows could be required to represent one object, and I had to slice 3 columns with an arbitrary number of slices.

Oh, and there were two types of sub-record per object, and they had to be processed in database order.

Post reply on HN