Hey, great to see some interest and thanks for the ideas. I am going to keep the conversation going here instead of a private email thread, in case anyone else shows interest in this. I am also thinking about refining the design and asking for more participants on r/emacs + r/spacemacs or even language-specific subs.
I remember checking the thread several hours after posting my comment and assumed that it was so offtopic people didn't engage, hence the delayed reply.
> Are you sure using just one language for the code is a good idea?
The more languages the better, I was only assuming that I would be preparing the codebase myself, hence wanted to maximise ROI by choosing a popular language that I can write.
Adding more languages would be great, however, we would need to keep the code + tests + failures as consistent as possible i.e. no language-specific test failures due to multi-threading unsafety of C++, which Python's GIL prevents. Also I am not sure how useful cross-language comparisons are.
> how much time would participants have to prepare?
Prepare what? Ideally nothing in their language + emacs setup. The whole point is to learn about people's natural emacs workflow and what tricks they use when in the state of flow. I think each participant should only do it once in their "first-class" language, otherwise participants won't enter flow. That way, we control for no previous familiarity with the problem and get the best emacs knowledge and maybe even some elisp snippets others can follow.
To borrow a sports metaphor - I am sure that Tiger Woods can play a decent game of tennis, I would rather get tips from him to improve my golf swing.
> Starts a fresh emacs instance
That was a suggestion born from my workflow that minimises the risk of other buffers getting in the way of the test and preventing users' from leaking personal data during the screen recording.
> commands and functions would be quite enough
Yes and no. Some of Mike Zamansky's youtube videos have an on-screen feed of keypresses, which is interesting to watch for evil-vimlike navigation practices. I cannot promise I will be good at analysing a stream of keypresses and extracting meaningful analyses though, so commands and functions is what I can promise to analyse.
> I'd prefer a short chat with a participant afterward, maybe coupled viewing the replay at 10x the speed.
A chat is a great idea, however, if we manage to get a decent number of participants (ideally 40+ for each language for meaningful stats), my availability for a sync chat will become the bottleneck of the experiment.
I am thinking about replacing it with an async Q&A that the participant can add after completiion.