Not everyone is affected this way and I believe the root cause that caffeine exacerbates is the particular kind of stress you feel when you want to get something done already, a typical state of mind to be in running a startup. Stress and psychological factors feed directly into bad posture and this is why taking regular breaks only upped my stress levels as now I needed to work even faster.
Lastly on the level of equipment fixes, ensure your elbows are supported by arm rests.