Inside The Decline of Stack Exchange

That being said, I realize you do have to be careful about subtle bugs.

My canonical example is “write a Python script that returns all of the AWS IAM roles that contain a given list of policies”.

The code it usually generates works correctly as long as you don’t have more than 50 roles in your account. It won’t add pagination support unless you spot the bug and tell it.

The second question I have is what type of questions do you find are better suited for Stack Overflow than ChatGPT? Are they questions based on post 2021 knowledge?

Is that correct? I would think that its ‘interpolation’ and ‘extrapolation’ capabilities can easily make it produce incorrect statements.

As a simple example, train it on “π is a number”, “1 is a number”, “3/7 is a number”, “1.3457 is a number”, etc.

and

“1 is a rational number”, “3/7 is a rational number”, “1.3457 is a rational number”, etc.

and I give you a good chance that it will ‘think’ “π is a rational number”.

Here's how I've seen it played out before:

---

johnnydoe2002 posts a question titled "How do I get my code to work?" with two sentences explaining their problem and an attached screenshot of their code. Within two minutes it gets downvoted thrice, flagged seven times as vague, and modman♦ closes the question, but not before posting the comment reply "Welcome to SO! Thank you for your interest, blah blah, your question has been closed because it needs clarification, blah blah, [link to new user portal]"

OK maybe that was a bit hyperbolic and most new users aren't that bad, so let's say johnnydoe2002 is undeterred by this and actually formats and titles their question in a reasonable way. Within four minutes it has one short answer from repseeker and a long-form answer that is more of a code review from overeagerverbosity. The question gets no upvotes despite having two answers. Within seven minutes poweruser98 closes the question as duplicate because it received enough flags.

If I were johnnydoe2002, I would have a mix of feelings. On the one hand, someone (two people, in fact!) answered my question; on the other hand, it was summarily closed and nobody found it interesting. I can clearly tell that I am not a part of the "in-group."

But I can also commiserate with modman♦ and poweruser98. johnny's question only adds noise and they clearly didn't read the contribution guidelines. The mods and power users are willing to be nice, but they don't have the time to hold anyone's hand: in the past ten minutes they've had to deal with all sorts of questions from larryloe2001, minnymoe2003, and randyroe2000. I wouldn't be surprised if for every useful question on SO there are two that have to be closed.

---

I honestly think that changes in phrasing and formatting could help the new user experience a lot.

Like if instead of closing questions as duplicates, the mod/user in question would post an answer that says "I think this other thread answers your question" and the page is invisibly redirected everywhere else to the other thread. Essentially "shadow closing" so that it appears to the new user as if they just received immediate feedback in the form of a (helpful) answer, while not being told that they are bad because they dared post a duplicate.

Or instead of closing a question as "needs clarification," sending it back to the edit modal with some adjoining commentary or links that explains the clarification it needs.

I think that for a community with standards these exacting, most users will not be able to just pop in - which is difficult because we usually refer to SO in passing. But I also think that those exacting standards can be upheld without stomping over new users with downvotes and close votes.