Encoder-decoder models
Published:
Encoder decoder is a deep neural network architecture that consists of 2 components:
- Encoder: input -> encoder memory (real-valued vector),
- Decoder: encoder memory -> output. Input is variable length, encoder memory is fixed length.
9.6. Encoder-Decoder Architecture — Dive into Deep Learning 0.17.5 documentation
The encoder and decoder portions can be swapped out. Sometimes the encoder is initialized with pretrained weights, for example in CodeBERT.
They are often used for Seq2seq tasks such as Neural Machine Translation (NMT).