Being able to incrementally load compressed files. Sometimes I have to load large tables, do some processing and save the result. The easiest way to do that has been to load the rows of csv files by blocks (it's really easy to load parts of csv files). But I'd love a way to do the same row wise loading with compressed file formats such as parquet. (Bonus points if that's possible even if I'm not the one who built the original file)
Ask HN: What are some tools you wish you had while doing your day to day work?
11–20 of 106 posts
Re: Ask HN: What are some tools you wish you had while doing your day to day work?
#12I wish I had a tool or app that would successfully get me to reduce both phone usage and use of certain apps or websites. Something that could adapt to different people would be good.
Re: Ask HN: What are some tools you wish you had while doing your day to day work?
#13Python
On my first sysadmin job fresh out of school I got a locked down laptop where nothing could be installed. So no Python, no Perl, no nothing, except Batch files. My morning job consisted of going through hundreds of report emails from all systems to identify issues (full disks, high nr of errors, etc), every morning, every day. This could easily be scripted and save me a lot of time, but no such luck for me. Until I found an application in the "allowed software catalog" which came installed with a Python interpreter. Python 1.x mind you, but nonetheless: Python. Within a few days I automated my morning tasks and got some free time to play with the servers in the lab to beef up my networking skills :).
Re: Ask HN: What are some tools you wish you had while doing your day to day work?
#14Earlier quoted context omitted.
I wish I had. but I don't because I haven't learned it yet
Why not start today? https://automatetheboringstuff.com/#toc https://learnxinyminutes.com/docs/python/
Yes, definitely recommend "Automate the Boring Stuff" for anybody who have a serious job and .... well wants to automate (some of) it.
If you are on windows, also checkout AutoHotKey. Makes some types of tasks a freaking breeze. +1 for great non-technical docs
Re: Ask HN: What are some tools you wish you had while doing your day to day work?
#15> And it was most frustrating when they had like 100 columns or so. Having to change column 63 at line 58 is not something you do easily in Notepad++. Import to Excel was my way to go, but since the file was not a standard CSV file (e.g. also had sections like an .INI file) it was time consuming: first just select a subsection from the file, save to a different file, import to Excel, modify, paste back, etc.
Suggestion: have you considered learning to use Vim? I imagine most Vim users are reading your example wondering what the big deal is.
Not to take away from the spirit of your post, though. You have valid questions.
Re: Ask HN: What are some tools you wish you had while doing your day to day work?
#16An equivalent to excel's record macro functionality, but at the os level, based on what's on the screen using ocr. Sikuli got close, but something with a little more AI behind it to get a sort of "fuzzy" macro recording.
Windows at least has AutoHotKey[1] -- it's kind of awful, but also kind of great. It has been a path into programming for more than one person I know and has saved others countless hours of repetition.
Re: Ask HN: What are some tools you wish you had while doing your day to day work?
#17Time-web-browser. That would allow me to look at future webpages and see for example what HNers in 2150 are into. If the sky is the limit of course...
Re: Ask HN: What are some tools you wish you had while doing your day to day work?
#18I wish I had tools that were just a little smarter at working with code. There are some things that I find I have to manually track down in a large codebase, but the things I do could be automated. I'm thinking of operations like the following:
* Tell me where in package (or module) X it calls functions that can eventually reach function Y in package Z (that is, there is a possible stack trace starting in X that leads to Y.
* Recursively expand the tree of callers of this function.
* Show me where the value passed to this function are created.
* List what packages have functions that can get called when I call function X.
* Given two functions A and B, what functions can result in calls to both? (This is sort-of the greatest common denominator of possible stacks leading to A and B)
(It seems like this must surely exist, but I have yet to find it)
I also want better tools for tracing what a system does when running. For example:
* Say I have a test over some deterministic code, and I make a small change to that code. I'd like to be able to run that test before and after, and get a diff of where the computation was the same vs. where it was different.
* When debugging, I'd like to be able to ask questions about the history, like what set this field on this model to the current value? (With debuggers I've used, you have to know these questions in advance and set up things to watch - you can't ask those questions after the fact)
* List all packages that had a line of code run while running this test.
edit: formatting
Re: Ask HN: What are some tools you wish you had while doing your day to day work?
#19> And it was most frustrating when they had like 100 columns or so. Having to change column 63 at line 58 is not something you do easily in Notepad++. Import to Excel was my way to go, but since the file was not a standard CSV file (e.g. also had sections like an .INI file) it was time consuming: first just select a subsection from the file, save to a different file, import to Excel, modify, paste back, etc. Suggesti…
As a longtime vim user, I don't see how vim makes going to the Nth logical column (for several values of N in different rows) in a large "CSV-like" file much easier.
Navigating in general is easier, sure, but... do you have a trick to share perhaps?
Re: Ask HN: What are some tools you wish you had while doing your day to day work?
#20Being able to incrementally load compressed files. Sometimes I have to load large tables, do some processing and save the result. The easiest way to do that has been to load the rows of csv files by blocks (it's really easy to load parts of csv files). But I'd love a way to do the same row wise loading with compressed file formats such as parquet. (Bonus points if that's possible even if I'm not the one who built the…
Not applicable to parquet, but I have used zless/zgrep often on tar.gz-compressed logs.
(I'm not sure what they're doing is actually that efficient though...)