Ask HN: What tasks do you automate? • Hacker News Board

Re: Ask HN: What tasks do you automate?

#71

Post by dozzie » Sun, Jul 16, 2017, 6:03 PM UTC

tchaffee wrote: post #46

The_Notorious wrote: post #27

Find yourself a configuration management server such as Puppet, Chef, CFEngine etc, and learn to automate system deployment and management with it. I use Puppet CE as my main automation tool. Use Python/Shell for tasks that are not well suited for a configuration management server. Usually, this is when procedural code makes more sense than the declarative style of Puppet manifests. Interactive "wizards" (i.e. add do…

I evaluated Puppet, Chef, and Ansible and Ansible was by far the easiest for me to learn. My favorite thing I use it for is my personal laptop setup. I can go from clean Ubuntu install to having all the software I use along with system settings all perfect within an hour, without me being involved.

It's like choosing programming language by selecting the one that's easiest for you to learn. You haven't learned anything beside new punctuation and keywords.

Ansible is basically a glorified shell script executed semi-interactively through SSH. It doesn't exhibit a new paradigm in server management by any stretch.

CFEngine, on the other hand, shows a whole another way to approach managing systems, and thus it's quite hard to learn (akin to learning Prolog or Coq after writing in Java). Puppet stems from that, but then slaps on top of that many weird and possibly unnecessary concepts, without much of an apparent plan or strategy, so it ended up a mess. Chef is a Ruby framework and some tooling around it, so no wonder it's more difficult to deploy than shell scripts through SSH.

Re: Ask HN: What tasks do you automate?

#72

Post by zwischenzug » Sun, Jul 16, 2017, 6:13 PM UTC

tchaffee wrote: post #46

The_Notorious wrote: post #27

Find yourself a configuration management server such as Puppet, Chef, CFEngine etc, and learn to automate system deployment and management with it. I use Puppet CE as my main automation tool. Use Python/Shell for tasks that are not well suited for a configuration management server. Usually, this is when procedural code makes more sense than the declarative style of Puppet manifests. Interactive "wizards" (i.e. add do…

I evaluated Puppet, Chef, and Ansible and Ansible was by far the easiest for me to learn. My favorite thing I use it for is my personal laptop setup. I can go from clean Ubuntu install to having all the software I use along with system settings all perfect within an hour, without me being involved.

I wrote a simpler ansible :)

https://learnxinyminutes.com/docs/shutit/

use it all the time, eg:

https://zwischenzugs.wordpress.com/2017/03/04/a-complete-ope...

Re: Ask HN: What tasks do you automate?

#73

Post by Huhty » Sun, Jul 16, 2017, 6:16 PM UTC

MY team and I run a reddit/HN-like community platform called Snapzu and we automate most (90%) of our social media channels.

We have 15 main categories, each with their own Twitter, Medium, WP, Blogger, etc. Here's an example of our science Twitter account: http://twitter.com/@Snapzu_Science

Re: Ask HN: What tasks do you automate?

#74

Post by aabaker99 » Sun, Jul 16, 2017, 6:19 PM UTC

dotancohen wrote: post #43

naturalgradient wrote: post #4

I take enormous pleasure in automating every part of my research pipelines (comp sci). As in, I like to get my experiment setup (usually distributed and many different components interacting with each other) to a point where one command resets all components, starts them in screen processes on all of the machines with the appropriate timing and setup commands, runs the experiment(s), moves the results between machine…

Another nice thing about setups like yours is reproducibility. So long as you've got your setup in git and you've stored the flags/lines/functions, you can instantly redo the same experiment.

I agree and I have been working to do this with some of my pipelines as well. One challenge I have been facing is that my compute environment may be quite different than others'. This is mainly the case with respect to distributed computing that seems to be an essential part of the pipelines: I often wish to experiment with multiple hyperparameter settings which creates a lot of processes to run.

Do you or the parent or others take steps to abstract away the distributed computing steps so that others may run the pipelines in their distributed computing environments? More specifically, I use Condor (http://research.cs.wisc.edu/htcondor/) but other batch systems like PBS are also popular. Ideally my pipeline would support both systems (and many others).

Re: Ask HN: What tasks do you automate?

#75

Post by dmak » Sun, Jul 16, 2017, 6:21 PM UTC

SirLJ wrote: post #49

Stock market trading systems, so I don't have to watch screens, also backups and also constantly improving monitoring for smooth operations

How can I get started with this? Did you create machine learning algorithms too?

Re: Ask HN: What tasks do you automate?

#76

Post by dmak » Sun, Jul 16, 2017, 6:24 PM UTC

kensoh wrote: post #66

SirLJ wrote: post #49

Stock market trading systems, so I don't have to watch screens, also backups and also constantly improving monitoring for smooth operations

I used to write trading systems for Amibroker. This one signaled the 2009 march bull market in equities and oil. But it is a swing/longterm trading system. I was a daytrader so it is not really useful for that. https://github.com/tebelorg/Tump/blob/master/trading_z.js

Reading into this, and this seems pretty dense. Any suggestions for beginners who are Software Engineers but need more explanation in this domain?

Re: Ask HN: What tasks do you automate?

#77

Post by edwinksl » Sun, Jul 16, 2017, 6:29 PM UTC

aabaker99 wrote: post #74

dotancohen wrote: post #43

Earlier quoted context omitted.

Another nice thing about setups like yours is reproducibility. So long as you've got your setup in git and you've stored the flags/lines/functions, you can instantly redo the same experiment.

I agree and I have been working to do this with some of my pipelines as well. One challenge I have been facing is that my compute environment may be quite different than others'. This is mainly the case with respect to distributed computing that seems to be an essential part of the pipelines: I often wish to experiment with multiple hyperparameter settings which creates a lot of processes to run. Do you or the parent…

Maybe containers like Docker are useful for your use case?

Re: Ask HN: What tasks do you automate?

#78

Post by dmak » Sun, Jul 16, 2017, 6:29 PM UTC

tokenizerrr wrote: post #19

jessedhillon wrote: post #16

I have a script that downloads bank and credit card transaction data, then applies rules to create a journal in GNU Ledger format.

Do you use an API? Or just scraping? I've done the same, but stopped using it. Afraid my bank, if they detect it, might not like it.

Whole businesses like Yodelee and Plaid do this.

Re: Ask HN: What tasks do you automate?

#79

Post by dmak » Sun, Jul 16, 2017, 6:29 PM UTC

kensoh wrote: post #44

tokenizerrr wrote: post #19

Earlier quoted context omitted.

Do you use an API? Or just scraping? I've done the same, but stopped using it. Afraid my bank, if they detect it, might not like it.

I did the same lol (via scraping), run every midnight to multiple bank accounts, then sync to google spreadsheet. I was writing web automation scripts the hard way for a while, until I made a tool to simplify the process of writing web automation scripts - https://github.com/tebelorg/TagUI (frankly I am surprised why my banks never banned or send me a letter)

I clicked it expecting a basic script, but this is pretty comprehensive. Nice work!

Re: Ask HN: What tasks do you automate?

#80

Post by nurettin » Sun, Jul 16, 2017, 6:31 PM UTC

In my city, there are many stadiums which cause traffic congestion during rush hours. I made a scraping bot which tells me if there's going to be traffic on my designated routes the next day. Going to try making it an app and see if it's any useful to others.