They can make a fair guess at which websites you're accessing. A lot of websites that are related to your particular interests aren't behind shared IP addresses these days.
Consider this:
* Build a list of domains you're interested in. There's fun community build blocking lists that can help you, if you need it.
* Periodically resolve every domain. Odds are if you're an ISP your servers already have the records cached, but it isn't too hard to resolve them all.
* Dynamically adjust your routing to specifically re-route those IP addresses to your special infrastructure (or maybe null route it if you want to block access)
* Use your special infrastructure to build up whatever profile you want about the source IP address, which of course being an ISP you'll be able to link directly to a user. Your end user won't even know.
You'll know when they access sites, be able to build up patterns of websites they access, in which order, and spot variations in that pattern.
There's a lot you can infer from metadata without decrypting the traffic. Everything from which domains you access, ports used for the communication, what order, and what sort of size the payloads are. You could identify that someone has an email account they only access after visiting, say, the Ashley Madison website. By tracking the size of communication on that is being sent to figure out if someone is sending dick picks or some such, or if it's likely just plain text.