Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ways Data Projects Fail

martingoodson.com

Post reply on HN
Print view View original

71–79 of 79 posts

Re: Ways Data Projects Fail

#71

Post by Declanomous »

srean wrote: post #54
Declanomous wrote: post #12

>Your Data Scientists are about to quit. This is me. I work for a non-profit that is stuck in the stone age--not for lack of money, mind you, but because the IT Director is an incompetent megalomaniac who views "security" as a reasonable justification to refuse any and all requests, and treats everyone like an enemy. I haven't been allowed to use Python or R. In fact, the only programming language I have access to is…

I am with you on everything but the complaint about > my computer is a Core 2 Duo from 2007 or so with 4 gb of ram That's a super freaking powerful machine, you have to be efficient in your programs and good on your algorithms. My machine was a Pentium III 800MB RAM for the longest time. There is a lot you can do on that. Use algorithms that need to load data in chunks, exploit memory mapping and generate native code…

I can make my programs work, but it's not a good allocation of resources. Two hours of my time costs as much as that computer. The computer that would allow me to do my job with the most efficacy is equivalent in cost to about 10 hours of my time.

The question is whether the better computer save more than 8 hours over the course of a year? The answer is yes. The amount of time it takes to load stuff in from the disk slows down programs immensely. If my computer had 16 gb of memory I'd be able to store and manipulate all my data in memory.

That's not even counting how much time is wasted optimizing code that I run rarely. That time is much better spent doing other things.

Re: Ways Data Projects Fail

#72

Post by Declanomous »

jupiter90000 wrote: post #46
Declanomous wrote: post #12

>Your Data Scientists are about to quit. This is me. I work for a non-profit that is stuck in the stone age--not for lack of money, mind you, but because the IT Director is an incompetent megalomaniac who views "security" as a reasonable justification to refuse any and all requests, and treats everyone like an enemy. I haven't been allowed to use Python or R. In fact, the only programming language I have access to is…

That sounds horrible. Ever thought of writing a VBA for data science book? ;-)

I've thought about putting all my code on github, just for kicks. My code is a nightmare though, because I don't have version control, and Office VBA doesn't allow inheritance. As such, the amount of code that has been copy-pasted is bonkers.

That being said, the amount of VBA I've seen that doesn't work with Option Explicit is a little staggering, so maybe I should be so self-conscious.

Re: Ways Data Projects Fail

#73

Post by vram22 »

gerhardi wrote: post #69
vram22 wrote: post #68

Earlier quoted context omitted.

MDM is Master Data Management? (from a quick google). Hadn't heard of that specific term before. I'm interested in data projects.

Yes correct. In a way it's a combination of philosophy, agreed practices and techical solutions. I think this one is a good introduction: The What, Why, and How of Master Data Management https://msdn.microsoft.com/en-us/library/bb190163.aspx

Just checked it out - looks good. Thanks.

Re: Ways Data Projects Fail

#74

Post by marcosdumay »

mswen wrote: post #20

In spite of assurances from business process owners that the underlying data sources are clean ... they almost certainly are not. Multiple legacy systems with no consistent cross reference to unambiguously identify the same customer. Assured that systems have been gone through and all the names made consistent. Consistency for a human is not consistency for a computer. "Commers Ltd" is not the same as "Commers Ltd."…

> And, isn't it lovely when a salesperson decides to add a location to a customer name. Now we have "Commers Ltd Dallas" as a unique customer.

Good luck modeling data to not need this. I have caught myself actively telling people to do this more than once, and I'm really thinking about replicating some data so those changes are less disrupting.

That "Commers Ltd Dallas" probably has differing billing and delivery addresses, points of contact, invoice formats, customers representatives and preferred sellers, product selection, and probably everything else you have on your DB.

Re: Ways Data Projects Fail

#75

Post by zgramana »

mcrad wrote: post #35
throw_away_777 wrote: post #25

Earlier quoted context omitted.

I've always disliked the term data cleaning for the reasons you mention - it doesn't tell me anything about what is meant by "cleaning".

Besides, cleaning implies some entry level gig, something for a QA hack not someone experienced in complex systems. Marketing types love this kinda twist as a way to maintain control (and ensuring failure) of the project. It's like a scapegoat. Just got out of meeting where marketing claims that data "hygiene" is gonna be a priority in 2017.

I have spent a lot of time talking with customers/prospects about this topic, but I use "data remediation" which I feel brings more accurate and precise connotations, to wit:

* Implies that the data is deficient/falls short of expectations.

* Implies that the shortcoming currently makes it ineligible to graduate to the next level.

* Implies that with hard work and additional time likely it can be made sufficient though still not ideal.

* Implies that someone failed to help the data to meet expectations.

* Implies that you need special outside expertise, namely someone with the knowledge needed to assess the shortfall, possibly help you clarify your standards, design steps that when followed should result in "good enough" data, and who is able to articulate the remaining weakness(es) which need to be accounted when assessing future suitably of that dataset for a given purpose.

* Implies that your data will be stuck in school all summer while their friends are out having so much fun.

Re: Ways Data Projects Fail

#76

Post by zo1 »

marcosdumay wrote: post #74
mswen wrote: post #20

In spite of assurances from business process owners that the underlying data sources are clean ... they almost certainly are not. Multiple legacy systems with no consistent cross reference to unambiguously identify the same customer. Assured that systems have been gone through and all the names made consistent. Consistency for a human is not consistency for a computer. "Commers Ltd" is not the same as "Commers Ltd."…

> And, isn't it lovely when a salesperson decides to add a location to a customer name. Now we have "Commers Ltd Dallas" as a unique customer. Good luck modeling data to not need this. I have caught myself actively telling people to do this more than once, and I'm really thinking about replicating some data so those changes are less disrupting. That "Commers Ltd Dallas" probably has differing billing and delivery add…

It's really the only option if your input/managing software can not model the complexity you require "after the fact". E.g. Company has multiple offices. Delivery address is per office, not per company, etc.

The real problem is when there is chaotic, and organic mixing, matching and re-purposing. I've seen it many times with "non-technical" individuals. They don't know what their software can do. E.g. Redmine. So the support individuals just log everything under the same "IssueType. They then "categorize" it using Category custom field, instead of the standard category which has enumerations. And then they then use that Category field to drive reports/process. Instead of using a different IssueType or Tracker, which is what it was designed for, and has tools that help you leverage/manage the complexity of different standardized processes.

Then, they decide to to add "Sub-categories" into the category field, instead of using a project-hierarchy or something. Then they want to do billing reports from the time logged per X and of types A,B,C, and at that point it's a giant mess and I stop caring. If they want to not use the software as intended, then do "fixing" by filtering and fiddling with Redmine CSV exports in Excel afterwards, that's their problem. Oh, and they ask that everyone has permissions to everything, allowing all users to change the status of each IssueType as they please, without any process.

I just feel sorry for the poor individual that get's a raw extract of that data and has to use it for something.

Re: Ways Data Projects Fail

#77

Post by maxxxxx »

Declanomous wrote: post #70
maxxxxx wrote: post #40

Earlier quoted context omitted.

To be honest your IT director sounds like an idiot but the real problem are YOUR managers. They should have protected you and told the IT director to f...k off a long time ago.

I've had the same thought. I actually met with my boss and HR about this, and they said there was nothing they could do. The issue is the company structure. I work for a 501c3 affiliated to another organization. The parent company provides all of our admin functions like HR, IT, etc. We share a board, and the director of each department reports directly to the board. The IT director is really well liked by the board…

In the end the only way to fix this is probably to leave the company if you can. If senior leadership doesn't see a problem there isn't much you can do.

Re: Ways Data Projects Fail

#78

Post by mswen »

marcosdumay wrote: post #74
mswen wrote: post #20

In spite of assurances from business process owners that the underlying data sources are clean ... they almost certainly are not. Multiple legacy systems with no consistent cross reference to unambiguously identify the same customer. Assured that systems have been gone through and all the names made consistent. Consistency for a human is not consistency for a computer. "Commers Ltd" is not the same as "Commers Ltd."…

> And, isn't it lovely when a salesperson decides to add a location to a customer name. Now we have "Commers Ltd Dallas" as a unique customer. Good luck modeling data to not need this. I have caught myself actively telling people to do this more than once, and I'm really thinking about replicating some data so those changes are less disrupting. That "Commers Ltd Dallas" probably has differing billing and delivery add…

That is fine but don't later ask why the system has 10 different customers for Commers Ltd

I ended up building another table and logic just to do roll-ups and account for name variations. But I told the client that they really need to invest in a serious cross-referencing middleware that tracks identities across all these systems and uses ID numbers to coordinate all the legacy systems.

But yes trying to develop models or even simple aggregations that rely on this kind of data can be quite frustrating.

Re: Ways Data Projects Fail

#79

Post by ernestbro »

"Somebody heard: Data is the new Oil: No it isn't. Data is not a commodity, it needs to be transformed into a product before it's valuable"

Err...Oil needs to be transformed and refined before it can be called a product (like gasoline, plastics). So the analogy is good and even supports #1!

Post reply on HN