In a naive encoder-decoder model, one RNN unit reads a sentence, and the other one outputs a sentence, as in machine translation.
But what can be done to improve this model’s performance? Here, we’ll explore a modification to this encoder-decoder mechanism, commonly known as an attention model.
In machine translation, we’re feeding our input into the encoder (green part) of the network, with the output coming from the decoder (purple part) of the network, as depicted above.