Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lab Notebooks (2020)

sambleckley.com

Post reply on HN
Print view View original

61–70 of 70 posts

Re: Lab Notebooks (2020)

#61

Post by rramadass »

shcheklein wrote: post #18

It reminds me an approach to data science and ML. It's way more common and essential for ML teams to log their attempts (experiments). There is a whole set of tools for this - experiments trackers. Primarily to even being able to compare and pick the best direction, but also to ensure reproducibility (in some areas it might be required). ML/DS always seemed to me closer to science in its nature vs software engineerin…

> ML/DS always seemed to me closer to science in its nature vs software engineering. Can be because of its nature as well - in a lot of cases it's a process of incremental improvements

Right, i have felt this way too (even though i am just a noob at ML/DS). For me it is the use of Statistics/Probability/Mathematics in driving understanding/intuition about the problem.

Re: Lab Notebooks (2020)

#62

Post by rramadass »

devsda wrote: post #24

Does anybody have a reference/example of a lab notebook for software engg ? If I'm forced to log my thoughts/actions it comes out as either too terse(everthing feels obvious) or it's too verbose and it's often difficult to find the right balance.

Don't over think it; just trust your intuition and do what seems natural. I also have the same problem but find that when it is too terse it acts like a "bookmark" into my "Memory Library" where i can recall the details. The process of "writing it down" is an aid to understanding/remembering/sorting/connecting ideas/concepts in one's head and it doesn't matter what system you follow.

Re: Lab Notebooks (2020)

#63

Post by jstanley »

plg wrote: post #53
jstanley wrote: post #40

I've started writing a markdown file every day, with a separate heading for each project, and a tool that automatically creates an index page that says what topics (based on headings) were present each day. Apart from the format of the headings, it is deliberately informal and unstructured. I've found it very useful.

It would be great for you to share the code

It's in https://github.com/jes/notes/tree/master/mkjesnotes but it's not particularly user-friendly for anyone but me. Nor is it supposed to be, it is very much a "home-cooked app".

Re: Lab Notebooks (2020)

#64

Post by malfist »

Aerbil313 wrote: post #51
ac50hz wrote: post #47

Earlier quoted context omitted.

Record keeping, for better or worse, is essential. Memories fade. Ink fades but rarely corrupts.

I grew up with the stories of real people (students and scholars of the past times) being able to memorize things by just hearing or reading them once. Seems a forgotten thing in the modern world.

Because it's a myth. Photographic memory isn't real. Check out the wikipage on eidetic memory. People that claimed it in the past were not telling the full truth. There are only a few examples of savants being able to recall specific things after one showing, but it's terribly narrow. Like being able to draw a horse you saw, but not human faces.

https://en.wikipedia.org/wiki/Eidetic_memory

Re: Lab Notebooks (2020)

#65

Post by analog31 »

doctorpangloss wrote: post #29
analog31 wrote: post #5

The article mentions, but doesn't elaborate, on the purpose of the notebook as a legal document. This was much more important when notebooks could be used to show priority in patent disputes, but only if well kept. My parents were both research scientists. Where my dad worked, the company patent lawyer read and critiqued your notebooks, and maintained the discipline. Many of those rules became unnecessary when the US…

To your specific point, the legal issues are so idiosyncratic. You can’t write an article about them. You can’t tease out the difference between “this is a best practice everyone should follow” and “the patent lawyer is selling the holistic social experience of compliance,” which is the non-trial attorney’s primary product. It’s the same energy as DocuSign: when your biggest competitor is a pen, you know all the bull…

I just looked up Benchling. My first impression is that it's enterprise software. You're not expected to just buy a license and start using it, and being the first user at your site won't gain you much over what you're using right now.

Where I think software like that gains its use is when you have multiple people working on similar research or processes, and trying to manage that kind of research and the data that it generates. At the extreme, I see job openings like this:

Wanted, quality control chemist for third shift...

On the other hand, I work in a "small lab" setting. I'm often the only person working on a project. No two of my projects resemble one another, though there are themes that progress through my work, and tools that I re-use and share. My choice of a notebook system, if you can grace it with such a description, is almost solely for my own benefit.

Re: Lab Notebooks (2020)

#66

Post by nolamark »

maCDzP wrote: post #33
nolamark wrote: post #12

Earlier quoted context omitted.

Here is a first draft. I'll expand if I have more time and energy. Rule #1. Every thing gets a date (And more an aspiration than reality a time.) I am truely an oddball and have my own date format 2023_12Dec_26Tue_1312. Rational, when used a a corresponding file name, it sorts naturally assuming you used 0 prefix on one digit numbers. Day of week, just another memory clue to try to recall specific thought experiment.…

Thanks for the write up. I am seriously considering the NeoSmartpen since it’s cheaper in the beginning and I really like the feeling of ink flow. I looked at their website and if you buy the idea pads you get 5 for 20 USD, seems reasonable.

Mentioning for completeness, they do also have printable ncode pdf files of various flavors for download on their site. Others have had success with them, I have not.

Livescribe has a similar product line. No direct experience with it, so can't contrast and compare.

Re: Lab Notebooks (2020)

#67

Post by crispyambulance »

PNewling wrote: post #43
crispyambulance wrote: post #36

I have seen meticulous, beautiful lab notebooks before. Paper ones. With numbered oversize greenish pages, faintly quadrille ruled, filled with exceptional INK penmanship, paragraph after paragraph of cogent technical observations, theoretical musings, responses to papers, thought-experiments and real experiments, taped graphs and images. Everything dated and signed. I've seen stacks of them spanning YEARS of work. I…

>Perhaps these are becoming a relic of a lost era? Paper is too much of an "air-gap" when everything is done on computer. Moreover, the standard 2-week Agile "sprint" leaves no time for careful thinking and writing. One can't put epiphanies gained from writing up notes into "the deliverable" for a freaking Jira Ticket So, I get what you're getting at here (I think), especially as someone who came from research labs a…

Yeah, it depends on the team.

FWIW, some do structure a "data review" state prior to closure of a ticket where the deliverable is some type of report (often with a presentation). Obviously, this can't be done for every ticket but it's appropriate for periodic "capstone" tickets.

Re: Lab Notebooks (2020)

#68

Post by eternauta3k »

crispyambulance wrote: post #37
j0k3r_85 wrote: post #30

Earlier quoted context omitted.

Have you tried quarto markdown? It’s similar to rmarkdown but language agnostic and allows me to think in code in a way that can easily be added to notebooks. To work with them in notebook fashion I can work through one file a day and save the rendered pages in a folder.

Quarto, I think, is definitely a step in the right direction. It allows for publication quality rendering into pdf, and various html flavors including Confluence. You can write fluently in it, mix-in actual running code, and you've got options for sophisticated graphics, tables and mathematics. I do hope the Quarto project makes it and survives. It's a cut above the other notebook solutions like jupyter.

What's the advantage of Quarto over Jupyter? I looked at Quarto and rejected it because I couldn't find a way for a single cell to programatically generate interleaved markdown and plots (I can do this with Jupyter).

Re: Lab Notebooks (2020)

#69

Post by crispyambulance »

eternauta3k wrote: post #68
crispyambulance wrote: post #37

Earlier quoted context omitted.

Quarto, I think, is definitely a step in the right direction. It allows for publication quality rendering into pdf, and various html flavors including Confluence. You can write fluently in it, mix-in actual running code, and you've got options for sophisticated graphics, tables and mathematics. I do hope the Quarto project makes it and survives. It's a cut above the other notebook solutions like jupyter.

What's the advantage of Quarto over Jupyter? I looked at Quarto and rejected it because I couldn't find a way for a single cell to programatically generate interleaved markdown and plots (I can do this with Jupyter).

Not sure what you mean as there is no notion of "cell" in Quarto.

Instead, everything is markdown and you can interleave codeblocks (or inline code statements) anywhere you like to generate plots/tables, etc.

I haven't had a need to generate markdown programmatically, though I imagine there's probably at least a "hard way" to do that. FWIW, quarto is still in its early stages. The quarto devs on github are very nice and responsive to questions if you ever look into again.

Re: Lab Notebooks (2020)

#70

Post by analog31 »

j0k3r_85 wrote: post #30
analog31 wrote: post #5

The article mentions, but doesn't elaborate, on the purpose of the notebook as a legal document. This was much more important when notebooks could be used to show priority in patent disputes, but only if well kept. My parents were both research scientists. Where my dad worked, the company patent lawyer read and critiqued your notebooks, and maintained the discipline. Many of those rules became unnecessary when the US…

Have you tried quarto markdown? It’s similar to rmarkdown but language agnostic and allows me to think in code in a way that can easily be added to notebooks. To work with them in notebook fashion I can work through one file a day and save the rendered pages in a folder.

I finally gave it a look, and it seems attractive. A feature, in my mind, is that it doesn't replace Jupyter, so I can still use the more basic tool when I'm up to my elbows in lab stuff. I actually use Jupyter to automate experiments, not just theory and data analysis.

But... the first thing I'll do is try quarto for my passive web page, which I presently generate using nbconvert.

Post reply on HN