I know they are trying to promote their own Front-End Framework (while trying to pretend that it isn't one), but with regards to the question on the title, my experience is: you can, but please, don't. It actually isn't THAT hard, but what will happen is that you'll end up reinventing the wheel badly, and you'll be chasing bugs and fixing platform inconsistencies for months. It won't be optimized and it won't work in…
It's VERY rare, almost unheard-of, to find any front-ends these days that are just HTML, CSS, and Javascript. That wasn't the case 10 years ago. IMO we don't know what a modern app would be like without a framework because it's just not done anymore. Even this post here uses Turbolinks, a framework.
It's also rare these days (in my experience anyway) that clients will ask for things that are simply not possible with HTML, CSS, and JS. Website design has started to get very same-y, and both HTML and CSS have gone through huge upgrades. Depending on the requirements you might be able to pull off a modern-looking design with just the basics.
Browser compatibility remains a problem, but most JS tools called "frameworks" don't exist to solve that one, but mostly to help structure large amounts of JS code. These tools simply did not exist at all back then, because people just didn't write that much front-end code. On SPAs, a lot of the code is only there to reinvent things the browser and the back-end could be doing.
If I was starting a new project I would probably go very light on the tooling.