Live data from Hacker News

What SQL Analysts Need to Know About Python (2016)

segment.com

71–80 of 115 posts

Re: What SQL Analysts Need to Know About Python (2016)

#71

Earlier quoted context omitted.

Or worse, they use cursors. Cursors should never be used in SQL. Ever. That's my philosophy.

Could you elaborate further?

This link explains it quite well: https://stackoverflow.com/questions/58141/why-is-it-consider...

Re: What SQL Analysts Need to Know About Python (2016)

#72

Earlier quoted context omitted.

Or worse, they use cursors. Cursors should never be used in SQL. Ever. That's my philosophy.

What’s wrong with cursors aren’t they just iterators?

This link explains it well

https://stackoverflow.com/questions/58141/why-is-it-consider...

Re: What SQL Analysts Need to Know About Python (2016)

#73
post #9
post #8

uMatrix prevents it from loading. Against my better judgement, I turned off uMatrix and loaded the page. Then had a look at uMatrix. Wow, I think we have a winner for a site with the most scripts, most 3rd party domains, and the sheer number of XHRs. Sorry for this being off topic.

Do you have a sane setup you can share with uMatrix? Last time I looked at it, I was a bit overwhelmed and shelved it for later.

So, I don't know about setting rules beforehand that work very well universally, but what I personally think is the better method is to just slowly build up your ruleset as you use it every day. I don't think lots of people know how to do this so here it is:

1) Upon using a website you frequent, enable one root at a time until it functions properly. (or you can do the more granular per grid block enable) 2) Click umatrix, then click at the top where it says "uMatrix $ver" which will take you to the dashboard. 3) In the dashboard click on "my rules" 4) on the right click the "commit" button.

What this does is commit your changes to the permanent ruleset and websites you frequent will start to "just work". A word of warning, if you blindly commit after browsing for a while when you may have done some temporary allows on random websites, those will be commited too. So I suggest either reviewing the commits first (always a good idea) and deselecting the ones you don't want, or having a new session for each of your frequented websites that you then commit from.

I have been planning to write up a tutorial on things like this for family and friends, maybe I'll post it to show hn.

Re: What SQL Analysts Need to Know About Python (2016)

#74
post #70

Tangential question: I'm curious how many people (here on HN, or in general) learned SQL before they learned more traditional programming (e.g. Python, Java, C)? I learned traditional programming (through college) and only stumbled upon SQL years later (someone left a "How to use Microsft Access" book around at work). I absolutely love SQL, and am in the middle of writing a book on how to use just SQLite for large, c…

I'm a data engineer and I learned SQL before I learned python. I started out as a marketing specialist (mostly working in Google AdWords), then I graduated up to BI Analyst, and eventually to data engineer. I've picked up 100% of my python in my current job.

Re: What SQL Analysts Need to Know About Python (2016)

#75
post #6

I'm worried about the inverse - what Python data analysts should know about SQL. Because I've met tons of analysts who wouldn't be able to even run a basic select. I've seen tons of (often non-reproducible) code written in place of a simple SQL query. I really wish bootcamps and other learning platforms focused on SQL a bit more. (I am a Python data analyst who properly learned SQL only after several years in the ind…

Or worse, they use cursors. Cursors should never be used in SQL. Ever. That's my philosophy.

I wouldn’t be so orthodox, there may be some special cases if a database is used in non standard ways to do strange stuff. However, if cursors are used to retrieve or update data, to do actual database stuff, cursors always look like a capitulation and retreat into procedural territory. If you use cursors you’re not giving the database a chance to shine and make it very sad.

Re: What SQL Analysts Need to Know About Python (2016)

#76
post #9

Earlier quoted context omitted.

Do you have a sane setup you can share with uMatrix? Last time I looked at it, I was a bit overwhelmed and shelved it for later.

So, I don't know about setting rules beforehand that work very well universally, but what I personally think is the better method is to just slowly build up your ruleset as you use it every day. I don't think lots of people know how to do this so here it is: 1) Upon using a website you frequent, enable one root at a time until it functions properly. (or you can do the more granular per grid block enable) 2) Click uma…

When I visit a site that doesn't work. Usually I carefully whitelist things that I am willing to whitelist until it does work. If I can't get it working, then in most cases, I don't consider the content of that site to be valuable enough to whitelist things that seem sketchy.

In the case of this site, uMatrix just flat out blocked the entire site. it's not that the side page didn't load, or that nothing happened because of, say, missing javascript. I got a huge warning page in FireFox that uMatrix was flat out blocking it. Probably because the primary site is already blacklisted, maybe?

Re: What SQL Analysts Need to Know About Python (2016)

#77

Earlier quoted context omitted.

I’ve seen devs just run select * from table then filter it and sort it in their own code. Then they complain “the database is slow” when it’s spending all its time shipping gigabytes of data they don’t need to them!

Is anyone working on a translator for pandas dataframe syntax to SQL?

There's also this: https://pypi.org/project/pandasql/

Re: What SQL Analysts Need to Know About Python (2016)

#79
I had a problem where there was some fixed width data in a sql database - basically someone put mainframe data in a database.

There were 3 fields that had multiple entries in them, fixed width delimited. I had to split the fields by width and re-combine them, then also recombine them with another set of data with weird delimiters and rules.

It took a day and half (not full time) to figure it out in python. I can't even imagine tackling the problem in a non-repl language.

Re: What SQL Analysts Need to Know About Python (2016)

#80

If you are working a lot with databases, and would like to automate some of the workflow in python, I'd highly recommend Pony ORM, which really feels like LINQ for python, and feels very close to the original SQL.

Pony is the LAST product I’d ever recommend to anyone, ever. It’s probably the worst open source software I’ve ever used.

That being said, thanks to Pony I was called in to fix a shitshow and it likely earned me an additional 25% in consulting fees... so maybe I do like it in that regard.

But in all seriousness, stay far away.

Post reply on HN