Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion chapters/en/chapter1/6.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ You can experiment with decoder-based LLMs directly in your browser via model re

<Youtube id="0_4KEb08xrE" />

Encoder-decoder models (also called *sequence-to-sequence models*) use both parts of the Transformer architecture. At each stage, the attention layers of the encoder can access all the words in the initial sentence, whereas the attention layers of the decoder can only access the words positioned before a given word in the input.
Encoder-decoder models (also called sequence-to-sequence models) use both parts of the Transformer architecture. The encoder's attention layers can access all the words in the initial sentence. In the decoder, self-attention layers can only access words positioned before a given word in the target sequence being generated, whereas cross-attention layers have full access to the entire source sequence encoded by the encoder.

The pretraining of these models can take different forms, but it often involves reconstructing a sentence for which the input has been somehow corrupted (for instance by masking random words). The pretraining of the T5 model consists of replacing random spans of text (that can contain several words) with a single mask special token, and the task is then to predict the text that this mask token replaces.

Expand Down
Loading