Building a Virtual Machine Inside ChatGPT

> Write a press release for Apple's AR glasses, including it's name, two standout features, and price.

"Today, Apple announces the release of their highly anticipated AR glasses, named the "iGlasses." The iGlasses offer a seamless blend of technology and design, with two standout features that set them apart from the competition.

First, the iGlasses offer a revolutionary new display system that projects a high-resolution image directly onto the user's retina. This allows for a clear and immersive augmented reality experience, without the need for a bulky headset or cumbersome display.

Second, the iGlasses feature advanced AI technology that allows for real-time object recognition and interaction. Users can simply look at an object and the iGlasses will provide relevant information and capabilities, such as the ability to make a purchase or access additional content.

Priced at $499, the iGlasses are available for pre-order today and will be shipped to customers in early 2022. With their sleek design and cutting-edge technology, the iGlasses are sure to be a must-have for tech enthusiasts and early adopters."

Sounds good, sign me up!

I don’t know why but this response style is funny to me. Kind of like a moody teen who doesn’t want to talk to you.

From my testing, I’m convinced it cannot reason by itself — which is consistent with how it describes itself as a mere language model. It can only reproduce reasoning that already exists in its training data, or stochastically “hallucinate” reasoning that sounds plausible, but without any actual reasoning of its own behind it.

Why the heck are they trying to prevent creating original content?

>On a side note it feels like each command takes longer to process than the previous - almost like it is re-doing everything for each command (and that is how it keeps state).

That's because it's probably redoing everything. But that's probably to keep the implementation simple. They are probably just appending the new input and re-running the whole network.

The typical data dependency structure in a transformer architecture is the following :

outputt0 outputt1 outputt2 outputt3 | outputt4

featL4t0 featL4t1 featL4t2 featL4t3 | featL4t4

featL3t0 featL3t1 featL3t2 featL3t3 | featL3t4

featL2t0 featL2t1 featL2t2 featL2t3 | featL2t4

featL1t0 featL1t1 featL1t2 featL1t3 | featL1t4

input_t0 input_t1 input_t2 input_t3 | input_t4

The features at layer Li at time tj only depends on the features of the layer L(i-1) at times tIf you append some new input at the next time t4 and recompute everything from scratch it doesn't change any feature values for time To compute the features and output at time t4 you need all the values of the previous times for all layers.

The alternative to recomputing would be preserving the previously generated features, and incrementally building the last chunk by stitching it to the previous features. If you have your AI assistant running locally that something you can do, but when you are serving plenty of different sessions, you will quickly run out of memory.

With simple transformers, the time horizon of the transformer used to be limited because the attention of the transformer was scaling quadratically (in compute), but they are probably using an attention that scale in O(n*log(n)) something like the Reformer, which allows them to handle very long sequence for cheap, and probably explain the boost in performance compared to previous GPTs.