I need to upload invoices every month from all ~20 SaaS products I subscribe to an accounting software. Most of the invoices can be just redirected from email to another SaaS that will let me download a zip file containing all invoices from a date range. Other software requires me to login to the product, navigate to a page and download a PDF or print an HTML page. I have browser-automated all of these laborious ones as well so everything will be in that zip file. Saves me 30 min monthly and especially saves me from the boring work.
Ask HN: What tasks do you automate?
101–110 of 349 posts
Re: Ask HN: What tasks do you automate?
#102Earlier quoted context omitted.
Do you use an API? Or just scraping? I've done the same, but stopped using it. Afraid my bank, if they detect it, might not like it.
Your bank would object to you accessing your own transaction data?
Do a careful reading of your banks terms of service and you will likely find a clause either barring you from doing this completely, or voiding fraud protection.
Re: Ask HN: What tasks do you automate?
#103Earlier quoted context omitted.
Do you use an API? Or just scraping? I've done the same, but stopped using it. Afraid my bank, if they detect it, might not like it.
I want to do this too. Details would be handy. Even if its extracting data from PDF statements or saved HTML pages.
I used CasperJS to download a specific CSV file, providing the chrome user agent. Then I used a simple NodeJS script to parse this.
Re: Ask HN: What tasks do you automate?
#104Earlier quoted context omitted.
Another nice thing about setups like yours is reproducibility. So long as you've got your setup in git and you've stored the flags/lines/functions, you can instantly redo the same experiment.
I agree and I have been working to do this with some of my pipelines as well. One challenge I have been facing is that my compute environment may be quite different than others'. This is mainly the case with respect to distributed computing that seems to be an essential part of the pipelines: I often wish to experiment with multiple hyperparameter settings which creates a lot of processes to run. Do you or the parent…
I ended up writing a simple distributed build system (https://github.com/azag0/caf, undocumented) that abstracts this away. (I use it at a couple of clusters with different batch systems). You define tasks as directories with some input files and a command, which get hashed and define a unique identity. The build system then helps with distributing these tasks to other machines, execution, and collection of processed tasks.
Ultimately though, I rely on some common environment (such as a particular binary existing in PATH) that lives outside the build system and would need to be recreated by whoever who would like to reproduce the calculations. I never looked into abstracting that away with something like Docker.
(I plan to document the build system once it's at least roughly stable and I finish my Phd.)
Re: Ask HN: What tasks do you automate?
#105Earlier quoted context omitted.
I evaluated Puppet, Chef, and Ansible and Ansible was by far the easiest for me to learn. My favorite thing I use it for is my personal laptop setup. I can go from clean Ubuntu install to having all the software I use along with system settings all perfect within an hour, without me being involved.
It's like choosing programming language by selecting the one that's easiest for you to learn. You haven't learned anything beside new punctuation and keywords. Ansible is basically a glorified shell script executed semi-interactively through SSH. It doesn't exhibit a new paradigm in server management by any stretch. CFEngine, on the other hand, shows a whole another way to approach managing systems, and thus it's qui…
> It's like choosing programming language by selecting the one that's easiest for you to learn.
It is. Is there anything wrong with that if the most important feature to you based on your requirements is that it be the quickest to learn?
I needed something that was very quick to learn and I had most of my setup automated in Ansible before I had even finished trying to learn the other options. Ansible won hands down on the "easy to learn" factor. So far it meets all my needs, and thanks to Ansible Galaxy, I've had to do very little config myself. 95% of the time someone has already written what I need.
> thus it's quite hard to learn
Then it's definitely the wrong product based on my requirements at the time.
Re: Ask HN: What tasks do you automate?
#106- Downloading a song of youtube, adding meta data via beets and moving to my music lib - Adding tasks to my todolist client from every app I use(including my bookmarking service when I bookmark with specific tags) - Changing terminal colours based on time of the day(lower brightness in the evenings and hence dark colours, too much sunlight in the mornings and hence solarized themes) - Automatically message people who…
I wonder if charging phone charging at 95% actually gives better battery lifetime. It leads to your battery cycling between 95% and whatever state of charge you start charging at again. If you charge up to 100% your phone will stop charging, but use power from the charger bypassing the battery therefore keeping the battery in better shape.
Charging to 80% and keeping it there by just letting the charger supply the operating current would probably be best if phones supported it.
Re: Ask HN: What tasks do you automate?
#107i wrote little sync script to my server. it is save my mysql backups to google drive.
Re: Ask HN: What tasks do you automate?
#108I'm aggregating flash sales and sending post requests to azure ml using huginn. It's a work in progress, but huginn seems to be working well. Also considering giving nifi a go, but the setup seems a bit over my head. https://github.com/huginn/huginn http://nifi.apache.org
What are you doing with the data in Azure ML ? What is your experience with Azure ML ? How does it compare to the ML offerings from AWS ?
Re: Ask HN: What tasks do you automate?
#109I have a script that downloads bank and credit card transaction data, then applies rules to create a journal in GNU Ledger format.
Do you use an API? Or just scraping? I've done the same, but stopped using it. Afraid my bank, if they detect it, might not like it.
I use Selenium to drive a firefox browser under the assumption that banks don't care enough to detect it: https://github.com/tmerr/bank_wrangler . I am also slightly worried about banks not liking it, but if they do care I guess I will receive a scary cease and desist in the mail and I will cease and desist.
Re: Ask HN: What tasks do you automate?
#110Earlier quoted context omitted.
Maybe containers like Docker are useful for your use case?
Docker can distribute the software needed to run the job well, which is definitely part of the issue and something I should use more. However, I also have in mind a pipeline of scripts where one script may be a prerequisite to the other. Condor has some nice abstractions for this by organizing the scripts/jobs as a directed acyclic graph according to their dependencies. I was thinking other batch systems might suppor…
Apache Airflow defines and runs DAGs in a sane way, IMO. Takes some configuration, but worth it for more complicated projects.