Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What tasks do you automate?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 349 posts

Re: Ask HN: What tasks do you automate?

#41

Post by naturalgradient »

ibizaman wrote: post #31
naturalgradient wrote: post #4

I take enormous pleasure in automating every part of my research pipelines (comp sci). As in, I like to get my experiment setup (usually distributed and many different components interacting with each other) to a point where one command resets all components, starts them in screen processes on all of the machines with the appropriate timing and setup commands, runs the experiment(s), moves the results between machine…

How much time does a full reset take?

A few minutes - killing processes on one machine, restarting a database on another machine/cluster, reloading a schema, importing data (for some experiments), deploying a new service, loading tensorflow models, warming up benchmark clients. I found that the key of good research pipelines is having a really consistent way of passing around configurations between components.

My components arent strictly microservices (a mixture of open source components and handwritten tools) and they interact in all sorts o fprotocols with each other (importing jsons, csv, GRPC, HTTP), but I essentially treat the configuration flags as their API, so there are no implicit configurations that I could forget about. The rest is just naming things well, e.g. descriptive names for experiment result files etc.

Initially I thought everyone is doing that, but from talking to PhDs in other domains I noticed that there is a strong bias towards people working in any kind of complex distributed setting having these pipelines.

My friends who devise ML models and just test them on datasets on their laptop never had a real need to get a pipeline in place because they never felt the pain points of setting up large distributed experiments.

Re: Ask HN: What tasks do you automate?

#42

Post by patd »

Most of my side projects have been about automating the little things that end up taking me a lot of time.

At my first job, part of my work (next to junior dev) was to deploy EARs on Websphere. I automated it so that people just had to drop it on a shared folder and I'd just take a look if it failed to install automatically.

I wrote a command-line tool to search and download subtitles https://github.com/patrickdessalle/periscope

I made a browser plugin to compare the price of the European Amazon and a few other websites (it grew to more countries and websites) http://www.shoptimate.com

And now I'm working on a tool that regularly checks if some of my content is getting adblocked because it's something I periodically do by hand http://www.blockedby.com

In the end, automating things can take more time than actually doing it. But if it's used by others and saves them time as well, it's gratifying.

Re: Ask HN: What tasks do you automate?

#43

Post by dotancohen »

naturalgradient wrote: post #4

I take enormous pleasure in automating every part of my research pipelines (comp sci). As in, I like to get my experiment setup (usually distributed and many different components interacting with each other) to a point where one command resets all components, starts them in screen processes on all of the machines with the appropriate timing and setup commands, runs the experiment(s), moves the results between machine…

Another nice thing about setups like yours is reproducibility. So long as you've got your setup in git and you've stored the flags/lines/functions, you can instantly redo the same experiment.

Re: Ask HN: What tasks do you automate?

#44

Post by kensoh »

tokenizerrr wrote: post #19
jessedhillon wrote: post #16

I have a script that downloads bank and credit card transaction data, then applies rules to create a journal in GNU Ledger format.

Do you use an API? Or just scraping? I've done the same, but stopped using it. Afraid my bank, if they detect it, might not like it.

I did the same lol (via scraping), run every midnight to multiple bank accounts, then sync to google spreadsheet. I was writing web automation scripts the hard way for a while, until I made a tool to simplify the process of writing web automation scripts - https://github.com/tebelorg/TagUI (frankly I am surprised why my banks never banned or send me a letter)

Re: Ask HN: What tasks do you automate?

#45

Post by naturalgradient »

dotancohen wrote: post #43
naturalgradient wrote: post #4

I take enormous pleasure in automating every part of my research pipelines (comp sci). As in, I like to get my experiment setup (usually distributed and many different components interacting with each other) to a point where one command resets all components, starts them in screen processes on all of the machines with the appropriate timing and setup commands, runs the experiment(s), moves the results between machine…

Another nice thing about setups like yours is reproducibility. So long as you've got your setup in git and you've stored the flags/lines/functions, you can instantly redo the same experiment.

Yes, this comes especially handy when reviewers ask for additional experiments.

Re: Ask HN: What tasks do you automate?

#46

Post by tchaffee »

The_Notorious wrote: post #27

Find yourself a configuration management server such as Puppet, Chef, CFEngine etc, and learn to automate system deployment and management with it. I use Puppet CE as my main automation tool. Use Python/Shell for tasks that are not well suited for a configuration management server. Usually, this is when procedural code makes more sense than the declarative style of Puppet manifests. Interactive "wizards" (i.e. add do…

I evaluated Puppet, Chef, and Ansible and Ansible was by far the easiest for me to learn. My favorite thing I use it for is my personal laptop setup. I can go from clean Ubuntu install to having all the software I use along with system settings all perfect within an hour, without me being involved.

Re: Ask HN: What tasks do you automate?

#47

Post by rcarmo »

- Data pipelines (as seen elsewhere here)

- Anything related to infra (I do Azure, so I write Azure templates to deploy everything, even PaaS/FaaS stuff)

- Linux provisioning (cloud-init, Ansible, and a Makefile to tailor/deploy my dotfiles on new systems)

- Mail filing (I have the usual sets of rules, plus a few extra to bundle together related e-mails on a topic and re-file as needed)

- Posting links to my blog (with screenshots) using Workflow on iOS

- Sending SMS from my Watch to the local public transport info number to get up-to-the minute bus schedules for some pre-defined locations (also using Workflow)

- Deploying my apps on Linux (I wrote a mini Heroku-like PaaS for that - https://github.com/rcarmo/piku)

- Searching for papers/PDFs on specific topics (built a Python wrapper for arxiv/Google/others that goes and fetches the top 5 matches across them and files them on Dropbox)

- Converting conference videos to podcasts (typically youtube-dl and a Python loop with ffmpeg, plus a private RSS feed for Overcast)

Every day/week I add something new.

(edit: line breaks)

Re: Ask HN: What tasks do you automate?

#50

Post by natch »

Many things. Trivial one, recently wrote a script to electronically sign six documents from my divorce and related tax paperwork using ImageMagick. Just to avoid having to do it with Gimp or Preview or some other GUI tool, and then re-do it when there are revisions. Yes there are online tools but I'm working with people who don't use those, nor do I want to upload these documents anywhere I don't have to.

Often I'll spend as much time writing an automated solution as it would take to do the task manually, even if I'm only going to run the automated solution once. The work is way more fulfilling, and I can fix mistakes easier, and can learn and develop new techniques.

Post reply on HN