diff --git a/chapters/en/chapter1/6.mdx b/chapters/en/chapter1/6.mdx index 08e306e43..4c12e5bc9 100644 --- a/chapters/en/chapter1/6.mdx +++ b/chapters/en/chapter1/6.mdx @@ -87,7 +87,7 @@ You can experiment with decoder-based LLMs directly in your browser via model re -Encoder-decoder models (also called *sequence-to-sequence models*) use both parts of the Transformer architecture. At each stage, the attention layers of the encoder can access all the words in the initial sentence, whereas the attention layers of the decoder can only access the words positioned before a given word in the input. +Encoder-decoder models (also called sequence-to-sequence models) use both parts of the Transformer architecture. The encoder's attention layers can access all the words in the initial sentence. In the decoder, self-attention layers can only access words positioned before a given word in the target sequence being generated, whereas cross-attention layers have full access to the entire source sequence encoded by the encoder. The pretraining of these models can take different forms, but it often involves reconstructing a sentence for which the input has been somehow corrupted (for instance by masking random words). The pretraining of the T5 model consists of replacing random spans of text (that can contain several words) with a single mask special token, and the task is then to predict the text that this mask token replaces.