"How do you deal with pages that use Javascript to fetch page content after the initial markup is loaded?"
Provide an example page and I will demonstrate how I would solve the problem.
Not every user visits the same websites and web pages, so without giving specific examples, discussions about how to deal with these pages never go anywhere on HN.
To be honest, out of all the websites I have visited over entire lifetime using the www, the number where I have had to make any extra effort because of Javascript in order to retrieve some text/html, image or video is very small proportion. Not one that is large enough to justify using a JavaScript-enabled browser as default. For me, these are exceptional cases, not the norm.
The extra effort is usually a one-off script, not something I need to save.
Occasionally it is something I save for future use. One example of a saved script would be for non-commercial YouTube channels. Goal was a 2-column CSV of all videos from a channel in the form of title, url. Goal was not "perfection", just quick solution.
yy025 and yy032 are custom utilties for generating HTTP and decoding HTML, respectively.
Using a short script called "ytc" the process would be something like the following. openssl s_client is used as an example of a TLS client. "XYZ" is the name of the channel.
echo https://www.youtube.com/channel/XYZ/videos|ytc|sed wXYZ
Connection=keep-alive yy025 1.html
ytc title XYZ.1
ytc url XYZ.2
paste -d, XYZ.[12] > XYZ.csv
Here is the "ytc" script
case $1 in
"")exec 2>/dev/null;
export Connection=close;
yy025|openssl s_client -connect www.youtube.com:443 -servername whatever -ign_eof |sed 's/%25/%/g'|yy032 > 1.tmp;
while true;do
x=$(sed 's/%25/%/g;s/\\//g' 1.tmp|yy032|grep -o "[^\"]*browse_ajax[^\"\\]*" |sed 's/u0026amp;/\&/g;s/&direct_render=1//;s,^,https://www.youtube.com,')
echo > 1.tmp;
test ${#x} -gt 100||break;
echo "$x";
echo "$x"|yy025|openssl s_client -connect www.youtube.com:443 -ign_eof > 1.tmp;
done;rm 1.tmp
;;-h|-?|-help|--help)echo usage: echo https://www.youtube.com/user/XYZ/videos \|$0;echo "usage: $0 {title|url}