111–120 of 178 posts
Earlier quoted context omitted.
Would you know why sql server generally isn't used?
Probably because: * SQL Server is proprietary * SQL Server licenses are expensive * SQL Server runs only on Windows (or least used to?)
Customer: "How much does an Oracle database license cost?" Oracle Rep: "Well, how much do you have?"
Earlier quoted context omitted.
The really fun aspect of this for me, personally, is that I've been doing computers for ages (25 yrs or so?) now and 3TiB still intuitively feels like a massive amount of data even though I think I have something like 6TiB free space on my home server disks... which, in total, didn't even cost as much as a months' grocery shopping. Sometimes it really takes effort to rid yourself of these old intuitions that don't re…
I can remember going through this with a 10MB file about 15 year ago. It felt like a lot after growing up with floppy disks. But even a modest CPU could iterate over it quickly, I just didn't realise. I just assumed I would need to process it in a database!
On a somewhat related note: The original Another World[1] would probably fit into the caches that your CPU has as a matter of course these days.
Earlier quoted context omitted.
This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…
thanks for the SQLite tip, I've been meaning to add it to my tool set. question: is SQLite incrementally helpful when I'm already comfortable with a local pgsql db to handle the use case you suggested? would SQLite be redundant for me in this case? question: between postgres and unix tools (sed, awk) is there reason to use SQLite?
Reasons to prefer sqlite: it's easier to embed in an app, you want sort, join, split-apply-combine, scale, transactions, compression, etc.
Reasons to prefer pgsql: sqlite's perf tools suck compared to pgsql (last time I got stuck anyway) and I'm sure there are lots of sql-isms that sqlite doesn't handle if that's your jam. EDIT: forgot everything-is-a-string in sqlite, just wanted to add that it has bit me before.
The notion that a data scientist's only job is to "write a statistical model" and then it's someone else's problem to run it in a distributed environment only exacerbates the problem and lowers DS code quality.
Full disclosure: my company Pachyderm [2] is trying to solve exactly the problem Jeff is talking about in the post. We've built a data processing platform on top of the container ecosystem. Basically, the data scientist has complete control over the runtime environment for their analysis since everything is bundled into a container. It scales to work for actual "big" data, but it also great for small teams that don't have massive infrastructure resources.
[1] http://venturebeat.com/2015/04/01/the-geek-shall-inherit-the... [2] github.com/pachyderm/pachyderm
ETL (Extract, Transform and Load) is a process in data warehousing responsible for pulling data out of the source systems and placing it into a data warehouse. http://datawarehouse4u.info/ETL-process.html
Earlier quoted context omitted.
Would you know why sql server generally isn't used?
Probably because: * SQL Server is proprietary * SQL Server licenses are expensive * SQL Server runs only on Windows (or least used to?)
To your own points:
SQL Server is as proprietary as Oracle SQL Server is cheaper than Oracle SQL Server is being ported to Linux in 2017 :)
... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…
This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…
Earlier quoted context omitted.
Probably because: * SQL Server is proprietary * SQL Server licenses are expensive * SQL Server runs only on Windows (or least used to?)
I chuckled a bit because the GP mentioned Oracle. I'm sure you've heard this one: Customer: "How much does an Oracle database license cost?" Oracle Rep: "Well, how much do you have?"
Nobody enjoys writing and maintaining data pipelines or ETL. It’s the industry’s ultimate hot potato. It really shouldn’t come as a surprise then that ETL engineering roles are the archetypal breeding ground of mediocrity. There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. This is like... your opinion. Some people…