An all-transistor physical-learning transformer can be trained to perform in-context learning
The paper shows how a physical circuit, modeled using MOSFETs, can be trained to act as a transformer for in-context learning tasks. It achieves this by explicitly restoring bidirectional information flow in a double circuit, which allows training with the necessary cross-entropy loss. This enables the system to learn associative recall tasks that are hard for standard neural networks.