I've found that SQL and data frames are pretty exceptional together in combination. I'm a big fan of a Python module called PandaSQL https://pypi.org/project/pandasql . What makes this work so well (for me, at least) is that I can combine pandas operations that transform data frames with SQL operations that transform data frames in the same pipeline, and if I really need to, I can just break it all apart with python…
PandaSQL was last updated 3 years ago. Do you know any other alternatives?
What SQL Analysts Need to Know About Python (2016)
101–110 of 115 posts
Re: What SQL Analysts Need to Know About Python (2016)
#102Python + SQL seems like a good match for many analysis problems, but Excel + SQL is not bad either. I like the ability to combine complex SQL views or SQL functions with Excel pivot functionality, querying the database directly from Excel
Even after learning python, pandas, and SQL I still find myself firing up Excel first when exploring a new problem. The interface is incredibly simple for pasting, manipulating, and visualizing data. The warts really only start appearing when you try to build business applications on top of it and shortcomings with data integrity (amongst other issues) begin creeping in. But as a sketchpad for handling a problem? Exc…
Re: What SQL Analysts Need to Know About Python (2016)
#103Earlier quoted context omitted.
I’ve seen devs just run select * from table then filter it and sort it in their own code. Then they complain “the database is slow” when it’s spending all its time shipping gigabytes of data they don’t need to them!
From what I encountered, this is generally the case when someone is in the "analysis/reports" mode. Rather than get summary statistics on each column, find number of nulls, etc by writing a sql query, they instead get the data into the Python/R instance, and use general purpose functions, utilities, etc. "Programmers are expensive" statement probably applies here as well. I'm not trying to be defensive here, just say…
I understand this use case, but this is in actual application code!
Re: What SQL Analysts Need to Know About Python (2016)
#104Earlier quoted context omitted.
Ibis: https://docs.ibis-project.org/ https://docs.ibis-project.org/notebooks/tutorial/2-Basics-Ag...
Dang, Ibis doesn't support Redshift or SQL Server. I'm also having trouble understanding what it really is - it's an entire framework for big data it seems and not just a translator. What I'd really like is just that, something that turns pandas dataframe operation into ANSI SQL. So input pandas2sql('tablename["col"]') -> "select col from tablename". Something really simple to use.
Re: What SQL Analysts Need to Know About Python (2016)
#105Earlier quoted context omitted.
SQL Server doesn't make it easy to calculate median values. There are a number of slightly convoluted ways to do it but it's far from straightforward and usually I end up calculating it in C# if I can.
This legitimately sounds like a great reason not to use SQL Server, to be honest.
Re: What SQL Analysts Need to Know About Python (2016)
#106Earlier quoted context omitted.
Im surprised ppl dont use ORM libs for this instead..
ORMs are just as capable as any developer of generating bad SQL queries that crush databases and plug network connections.
Re: What SQL Analysts Need to Know About Python (2016)
#107I had a problem where there was some fixed width data in a sql database - basically someone put mainframe data in a database. There were 3 fields that had multiple entries in them, fixed width delimited. I had to split the fields by width and re-combine them, then also recombine them with another set of data with weird delimiters and rules. It took a day and half (not full time) to figure it out in python. I can't ev…
CREATE VIEW my_table_improved AS
SELECT SUBSTRING(the_column, 0, 8) as col1,
SUBSTRING(the_column, 8, 8) as col2,
SUBSTRING(the_column, 16, 8) as col3
FROM my_table
After which you can query my_table_improved as a normal table. col1, col2, and col3 contain the data split out from the_column. In practice you'd probably also want to do some type conversion, e.g. if col1 is supposed to be an integer you can simply update the view to select `CAST(SUBSTRING(the_column, 0, 8) as INTEGER) as col1` instead. In production use you might find this to be slow at which point you will want to create indexes for your new columns. Something like this (adjusted to which queries you're running of course) should work: CREATE INDEX ON my_table ((SUBSTRING(the_column, 0, 8)));
Of course this is a lot of work if you're unfamiliar with SQL, and above examples aren't quite complete yet for your use case, but it should get you an idea of how SQL is the exact right tool for the job here. which is somewhat the point of many commenters here: get yourself familiar with SQL and save yourself a metric tonne of work in the future.An alternate approach might be to write a query that migrates the fixed-width format to a format where each entry is in their own column. The ease of this mostly depends on if applications depend on that column being in that format.
side note: above sql code is untested but should be roughly correct.
Re: What SQL Analysts Need to Know About Python (2016)
#108Earlier quoted context omitted.
Even after learning python, pandas, and SQL I still find myself firing up Excel first when exploring a new problem. The interface is incredibly simple for pasting, manipulating, and visualizing data. The warts really only start appearing when you try to build business applications on top of it and shortcomings with data integrity (amongst other issues) begin creeping in. But as a sketchpad for handling a problem? Exc…
> But as a sketchpad for handling a problem? Excel is pretty hard to beat. I'd recommend trying J. Download here - https://code.jsoftware.com/wiki/System/Installation#Installa... . There is, of course, the famous barrier to entry, but the result, unsurprisingly, is also respectable.
Re: What SQL Analysts Need to Know About Python (2016)
#109I've found that SQL and data frames are pretty exceptional together in combination. I'm a big fan of a Python module called PandaSQL https://pypi.org/project/pandasql . What makes this work so well (for me, at least) is that I can combine pandas operations that transform data frames with SQL operations that transform data frames in the same pipeline, and if I really need to, I can just break it all apart with python…
PandaSQL was last updated 3 years ago. Do you know any other alternatives?
This is one of those technologies that I was hoping would make its way into the framework, kind of how connection pooling for databases was once an external module but is now often built into the various web frameworks. DataFrames with common columns are such a natural match to relational tables that it seems that a way to call SQL would be (ok, in my opinion, should be) part of pandas (I think that this is the case for R data frames).
I also just really enjoy writing SQL. To some extent, this is a personal preference - some people just mentally line up with certain ways of thinking. There are things that are clearly better to do in python, things that are clearly better to do in SQL, and then a grey area. I personally lean much farther to the SQL side of that grey area, but there's certainly nothing incorrect about going the other direction.
Re: What SQL Analysts Need to Know About Python (2016)
#110I had a problem where there was some fixed width data in a sql database - basically someone put mainframe data in a database. There were 3 fields that had multiple entries in them, fixed width delimited. I had to split the fields by width and re-combine them, then also recombine them with another set of data with weird delimiters and rules. It took a day and half (not full time) to figure it out in python. I can't ev…
Turns out a good database is really good at data munging. A solution for postgresql might look something like this: CREATE VIEW my_table_improved AS SELECT SUBSTRING(the_column, 0, 8) as col1, SUBSTRING(the_column, 8, 8) as col2, SUBSTRING(the_column, 16, 8) as col3 FROM my_table After which you can query my_table_improved as a normal table. col1, col2, and col3 contain the data split out from the_column. In practice…
Oh, and there were two types of sub-record per object, and they had to be processed in database order.