Live data from Hacker News

Ask HN: How did you automate your day job?

news.ycombinator.com

41–50 of 51 posts

Re: Ask HN: How did you automate your day job?

#41

I am a trader. A million different factors drive commodity prices, and their significance ebbs and flows over time in unpredictable ways. So typically you'll have a spreadsheet that monitors one of these occasional factors. You run it 98 days in a row and it's irrelevant. This lulls you into complacency on the 99th day, where you don't look at it, and on the 100th day suddenly it becomes relevant again and you didn't…

> My feeling now is that Python and similar tools are the equivalent of my Excel 20+ years ago, and Excel is the 'desk calculator'.

I'll second this sentiment, having built a crude automated trading system using Python and the Interactive Brokers API, for use in trading commodities, futures, and derivatives.

Re: Ask HN: How did you automate your day job?

#42
A major part of my job is to create reports. These are mind numbingly created in Powerpoint / Word with graphs from Excel. I've been told that automating any of the reports I do is basically impossible. I'm expected to do overtime as needed to get reports done on time. This happens most weeks.

Unfortunately scripting and devops aren't a thing where I am even if they could be applied. I'll need to find another corp. The next position I'm going for in is DevOps. But I haven't got enough of the silly certs / experience yet.

If only I had time off to study and wasn't creating reports all day. Then I'd be able to build my project portfolio and do the exam preparation. I've been here 18 months doing nothing but reports and a little system administration. My plan is to be here for no more than 36 months as this level of manually creating reports is too mind numbing.

Also me:

I generate a very important report for presentation each Monday afternoon and it takes around 4-6 hours of data gsthering and preparation. For the first four weeks I did it manually then realised I could automate some parts and then gradually up to all of it. Now it takes 30 minutes of script runtime with no manual intervention.

There's a patchwork of many other 1-2 hour reports I do across other days. They are now all fully automated by me. They take around 5 minutes of script runtime but no interaction.

I have another big report about to be assigned for Fridays. Its expected to take around 5-7 hours of manual collation. My estimation is that I will be able to run this in around 20 minutes.

In all I generate multiple word documents and powerpoint documents using excel spreadsheets with most of the heavy lifting being done using python, powershell, R and an ecosystem of little elixir critters.

As soon as I reach 24 to 36 months I'll have everything lined up and I'm out of here. I've done enough networking with various people to know that my portfolio of experience is relevant and useful. I've gotten my interview skills up. Plenty of my contacts agree.

Re: Ask HN: How did you automate your day job?

#43

I am a trader. A million different factors drive commodity prices, and their significance ebbs and flows over time in unpredictable ways. So typically you'll have a spreadsheet that monitors one of these occasional factors. You run it 98 days in a row and it's irrelevant. This lulls you into complacency on the 99th day, where you don't look at it, and on the 100th day suddenly it becomes relevant again and you didn't…

Would you be open to sharing more? What sort of things are you doing in Python that are new and unique, distinct from capabilities you had in spreadsheets?

What things are new/unique? Hmmm, I'll bet someone could implement almost all of what I do inside excel sheets one way or another, but it would just be super-cumbersome to do.

One big example is regression. All Excel really does is simple linear regression. Add more variables you get a better r2'd and that's it. Comparatively, with scikit it's trivial to avoid all horrid overfitting problems that come with simple regressions in excel. I use of lasso-cv model, and better practices like cross-validation, etc.

Another big improvement is I replaced Excel -> Tableau visualizations pipeline with using Altair-Viz. Altair makes almost-Tableau-like charts pretty easily and cuts out lots of intermediary steps -- jump right from dataframe to a chart.

It's much easier to add sanity/safety assertion checks in python than in Excel. Everyone knows examples of excel spreadsheets that are subtly poisoned.

I took someone else's technical charting ideas and reimplemented them in python, which then just alerts me for key events (changes in trend, potential exciting issues, etc). Now I don't have to look through the data myself, which is a chore, and which is susceptible to my mood. It just spits out its objective 'truth'.

One thing about (at least commodities trading) that outsiders might not realize is -- none of the sexy FAANG big data stuff is particularly useful or appropriate. The data is way too sparse and the quality varies immensely and is normally quite lagged. As a concrete example perhaps you think the price of oil = f(oil demand in china, oil production in saudi arabia). Great. At best you get monthly data, probably lagged, and probably manipulated by the government. So that's 24 pieces of data per year....not great candidate for machine learning ideas. Perhaps very high-frequency, ultra short term automated trading can use tick data for things like this, but not for general prop trading that holds risk for days or weeks.

I also expect outsiders imagine oil price = f(factor_a, factor_b, ...factor_z) and it's just the job to figure out that ka, kb, ... kz are. But markets aren't physical phenomena like a spring where repeated measurements of length and weight can eventually can estimate what the K-factor is. If markets are springs, then their K-factor changes all the time. Factors that sometimes matter for one month don't matter for the next six or twenty four. That's why it's important to monitor the many spaces that can drive a market -- underlying fundamentals, 'flows' from the financial/hedging markets, and myriad non-specialist investors who behave based totally different drivers than oil specialists. As much as oil trading could be trying to accurately count barrels of oil, it's also important to understand the 'meta' of who is buying/selling and why.

To be clear, my trading is not automated. At end of the day I'm making a bet myself, not a machine deciding. It's more like I have machine exoskeleton helping me to see a little bit more clearly and maybe a little less emotionally, which hopefully leads to slightly smarter bets.

Re: Ask HN: How did you automate your day job?

#44

I am a trader. A million different factors drive commodity prices, and their significance ebbs and flows over time in unpredictable ways. So typically you'll have a spreadsheet that monitors one of these occasional factors. You run it 98 days in a row and it's irrelevant. This lulls you into complacency on the 99th day, where you don't look at it, and on the 100th day suddenly it becomes relevant again and you didn't…

you trading your own money though? awesome answer btw.

Heh, no. I wouldn't want to trade my own money when there are people that will give me money to trade instead.

Re: Ask HN: How did you automate your day job?

#45
post #31

Earlier quoted context omitted.

Yep. I got a job offer. It was nowhere near competitive with other opportunities, though, so the talks didn't go far.

This checks out. Banks have no problem spending millions on poor solutions but zero interest in rewarding optimisations.

That's because you are a threat to the manager if you implement something like this without approval.

Re: Ask HN: How did you automate your day job?

#46

I actually got my first dev job by doing exactly that. While I was searching for a new career and vaguely interested in something to do with IT, I was doing short-term temp work as an office admin. One was in a startup - not tech-related - where I spent most of my day searching for customers' names and addresses in a spreadsheet, then copy-pasting them into a bunch of Word templates and printing them out. I already k…

You should thank that IT Manager today. Most of them would just ask you to get approval to use a database on your own PC at this point.

Re: Ask HN: How did you automate your day job?

#47
post #2

I was a high school teacher in a school that worked primarily with underserved students, who often had transcripts with many academic gaps. So we had to spend more time than usual examining transcripts to make sure students were taking the right classes to graduate. It was a small school, with three teachers and a few support staff. We literally spent hundreds of hours each year poring over text-based transcripts wit…

We need more teachers like this! Optimizing to spend time on getting their students to their goals instead of optimizing to go home early.

Thank you!

Re: Ask HN: How did you automate your day job?

#48
Last year in school I also worked for a car dealership one day a week. One of their employees transfered a task to me. She showed me a spreadsheet in Lotus 1-2-3, in which they had a list of cars with the day the received the car and they day the sold the car. She was counting the number of days in between and entered that number into the next cell. Took here 2hrs each week. During the explanation by her I already showed here how to do this with a simple formule. She was ecstatic.

Second thing I automated was automating their invoices. There was AS400 system that generated a CSV file which they printed. Then they opened a template Word-document and manualy copied the data from the print into the document, printed that and continued with the next. With some simple marco's I saved them about a days work every week.

Re: Ask HN: How did you automate your day job?

#49

Earlier quoted context omitted.

Yes, but let me elaborate. I quickly learned the way to get promoted is to be replaceable. If you’re relied upon too much or if you’re the guy who always fixes things - it’s actually too risky for a business to promote you. Throughout my career I always made sure I was giving people below me the opportunity to step up, which allowed me to step up too. I quickly appointed team leaders as soon as a team I managed had m…

This seems to me like a real failure of late stage capitalism. Similar to how factories in the soviet union were overstaffed with middle managers who barely lifted a finger. I might be wrong but this just seems almost like a ponzi scheme of some sort - keep moving up the middle management chain and getting somebody else to replace you. No offense to darrenwestall, I'm hating the game not the player.

This quote might be relevant: “I divide my officers into four groups. There are clever, diligent, stupid, and lazy officers. Usually two characteristics are combined. Some are clever and diligent — their place is the General Staff. The next lot are stupid and lazy — they make up 90 percent of every army and are suited to routine duties. Anyone who is both clever and lazy is qualified for the highest leadership duties, because he possesses the intellectual clarity and the composure necessary for difficult decisions. One must beware of anyone who is stupid and diligent — he must not be entrusted with any responsibility because he will always cause only mischief.” — General Kurt von Hammerstein-Equord

https://medium.com/@jcmrgo/i-divide-my-officers-into-four-gr...

Re: Ask HN: How did you automate your day job?

#50

When I worked as a lawyer, one of my tasks as a junior was to check the court lists every day against the spreadsheet of clients we had who were involved in litigation. If their name appeared then I had to send an email to the relevant lawyer containing the information. This took about 30 - 60 mins each day and was monotonous. Just copying and pasting the client name into the browser search tool. I automated the proc…

There are several businesses that grew out of people that worked in law firms or corporate legal departments and spotted the numerous deficiencies and opportunity for automation. I know this is true for most businesses but having worked in that area for a few years made me think they're specially lacking process automation considering how bug the Legal industry is in the US.
Post reply on HN