Indeed, as you say, the development of both LRBni and of AVX by 2 separate Intel teams stretched over many years.
Most of the development of LRBni was between 2005 and 2009, when it became publicly known. The first product with LRBni was Knights Ferry, which was introduced in 2010, being made with the older 45-nm process. Knights Ferry was used only in development systems, due to insufficient performance.
Sandy Bridge, using the newer 32-nm process, was launched in 2011. I do not know when the development of Sandy Bridge had started, but in any case the first few years of development must have overlapped with the last few years of the development of LRBni.
I suppose that there was little, if any, communication between the 2 Intel teams.
AVX was developed as an instruction set extension in the same way as the majority of the instruction set extensions had been developed by Intel since the days of Intel 8008 (1972) and until the present x86 ISA.
Intel has only very seldom introduced new instructions that had been designed having a global view of the instruction set and making a thorough analysis of which instructions should exist in order to reach either the best performance or the least programming effort.
In most cases the new instructions have been chosen so that they would need only minimal hardware changes from the previous CPU generation for their implementation, while still providing a measurable improvement in some benchmark. The most innovative additions to the Intel ISA had usually been included in the instruction sets of other CPUs many years before, but Intel has delayed to also add them as much as possible.
This strategy of Intel is indeed the best for ensuring the largest profits from making CPUs, as long as there is no strong competition.
Moreover, now the quality of the ISA matters much less for performance than earlier, because the very complex CPUs from today can perform a lot of transformations on the instruction stream, like splitting / reordering / fusion, which can remove performance bottlenecks due to poor instruction encoding.
Most programmers use only high-level languages, so only the compiler writers and those that have to write extremely optimized programs have to deal with various ugly parts of the Intel-AMD ISA.
So AVX for Sandy Bridge has been designed in the typical Intel way, having as target to be a minimal improvement over SSE.
On the other hand LRBni was designed from the ground, to be the best instruction set that they knew how to implement for performing its tasks.
So it was normal that the end results were different.
For the Intel customers, it would have been much better if Intel did not have 2 divergent developments for their future SIMD ISA, but they would have established a single, coherent, roadmap for SIMD ISA development during the next generations of Intel CPUs.
In an ideal company such a roadmap should have been established after discussions with a wide participation, from all the relevant Intel teams.
For cost reasons, it is obvious that it would not have been good for Sandy Bridge to implement the full LRBni ISA. Nevertheless, it would have been very easy to implement a LRBni subset better than AVX.
Sandy Bridge should still have implemented only 256-bit operations, and the implementation of some operations, e.g. gather and scatter, could have been delayed for a later CPU generation.
However other LRBni features, should have been present since the beginning, e.g. the mask registers, because they influence the instruction encoding formats.
The mask registers would have required very little additional hardware resources (the actual hardware registers can reuse the 8087 registers), but they would have simplified AVX programming a lot, by removing the complicated code needed to handle correctly different data sizes and alignments.
The current CPUs with AVX-512 support would have been simpler, by not having to decode 2 completely distinct binary instruction formats, for AVX and for AVX-512, which is a fact that made difficult the implementation of AVX-512 in the small cores of Alder Lake.