I feel like I understand your answer from within, because it's similar to how I looked at commenting in years past. I've written about that before:
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... (the earliest of those posts was one month after I became public as HN mod).
The problem is that you're only referring to what's going on inside yourself—that is, your ideas about debate, provocation, liveliness, and so on. If you want to be a valuable contributor instead of damaging the container, you need also to take into account what's going on in others—not just one or two others but many, in the case of a large forum like HN. More than that, you need to take into account the medium: what a large, weakly cohesive internet forum is capable of and what it is not. If you don't do that, you'll end up hurting the commons—which is fragile—even while being sure of the rightness and interestingness of your own intentions.
Imagine someone who's into boxing showing up at a dance, say, or a concert or a lecture—who, while milling around talking, is in the habit of punching other people now and then. Nothing serious; just a light jab to the torso or the side of head every once in a while. When asked not to do that, imagine that they reply: "Actually, I disagree with your approach. I think sparring is very valuable for developing alertness and reflexes. It focuses the mind and is a good starting point for interacting directly and truthfully. The fault lies with your rules, which care only about politeness and propriety and assume that people are soft and can't take a punch. These aren't even real punches, just taps, and they are a good device for getting people to reveal what they are really like behind their facade. I believe that we should not abstain from getting to know others as they really are, and that is why my interacting style includes some degree of pugilism, to elevate feelings for more lively and less stylised interaction."
The thing is, they're not wrong. That is, nothing they've said there is wrong—but it is wrong for this context, and that is enough to be disastrously wrong, not only for them and the people they're provoking but for the whole community. In a context with a different implicit contract—like a sparring ring, or a group of roughhousing friends—it would work fine.
When we ask people not to post flamebait a.k.a. provoke others on HN, we're not necessarily telling them that what they said was wrong, or what they did was wrong. We're just saying it's wrong here. That's why I say "here" so much in moderation comments (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que..., https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...).
That word here macroexpands in two dimensions. Along one axis it means: "given the nature of a large, anonymous internet forum"—i.e. the medium we're all communicating through. Along a second axis it means: "given the specific type of site we're trying to have". We're trying to optimize this place for one thing, namely curiosity (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...). The HN guidelines are a distillation of what we've learned about how we can all perform this optimization together. Since it's in all our interests to have a site that gratifies curiosity, it's in all our interests to follow them. You don't have to follow them for moral reasons; raw self-interest is fine, if that's what gets you there.
The problem with provocation and flamebait is easy to derive from first principles: you can't provoke or flame others into curiosity. All you will achieve is to agitate them, and then they will defend themselves in a hostile and predictable way. That is the opposite of curiosity, which is an open and relaxed state. It is how we get flamewars, and (again) the problem with those is not that they are intrinsically wrong somehow, it's that they are not interesting, and thus are wrong here, given how we're trying to optimize HN.
Some of you will say "But wait! I can be provoked into curiosity. As a matter of fact, I like it when people do that. I don't take it personally, and it makes me think. Actually, that's just the sort of conversation I think we should have on HN." Yes, some people, by virtue of being neuroatypical or having done a lot of self-work or who knows why, sometimes respond to provocation and flamebait by getting more curious. But you know what? It doesn't matter, because statistically the overwhelming majority of participants on a large, open internet forum are not functioning that way—not at all—and it is their responses which determine the threads.
In other words, it's the medium again. You need to understand the medium in order to know what sort of messages to send. If your messages are firebombs, you are going to set this place on fire, even if one or two people do happen to understand the game you're playing and are up for playing it too—just as when you throw punches at a party you're going to start a brawl, even if one or two people enjoy the sparring and respond playfully.
In other words, the argument "that's the sort of conversation I think we should have on HN" is wrong, not because you're wrong to think that or because such conversation is wrong in itself, but because it can't work here and there soon won't be any HN left if people do it (https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...).
Instead, you should follow the site guidelines and play the game they describe (even if you'd rather be playing, say, rugby: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...), because it is the only game we can play here—note that word "here"—given the medium and mandate of the site. Switching to some other game you like better isn't an alternative; the alternative is the destruction of the community (https://news.ycombinator.com/item?id=10411333), which isn't in any of our interests.
There are other places to play more rough-and-tumble games. You'd need a smaller, more cohesive forum (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...). Rugby teams who beat each other up on the pitch and then go out drinking together can do that because they have a shared identity and pre-existing relationships. Random groups can't do that, and large random groups absolutely can't.
Quite a few HN users, including some of the most prominent ones (and some of the best writers too), started off with a pugnacious commenting style and learned over the years to modulate that in the interest of curiosity, both in themselves and others. That's the learning curve we all have to go through here, and are still going through.