Catches from text to text transfer transformers:
- As an example, consider the case of English to German translation: If we have a training
datapoint with input sentence “That is good.” and target “Das ist gut.”, we would simply train the model on next-step prediction over the concatenated input sequence “translate English to German: That is good. target: Das ist gut.” If we wanted to obtain the model’s prediction for this example, the model would be fed the prefix “translate English to German: That is good. target:” and would be asked to generate the remainder of the sequence autoregressively.
- In order to conquer the disadvantage of causal transformers, the prefix LM model tries to
avoid this by making the model to see the sequence non causally during the prefix . e.g in translating from english to german , it sees whole english sequence but causality is maintained on generated translation.
- Lesser the parameters lesser is the training loss: this is explained as the model remembering the
training examples rather than really learning to do the task.
- Repeating pre training data is not always harmful but after a certain limit it is. In case of example of
C4 database with 2^35 tokens, when using dataset with 2^29 tokens, we can repeat this by 2^6 times means 64. Till that much the performance doesn’t deteriorate.
-
artificial “limit”
-
Temperature specific mixing
-
We find that fine-tuning after multi-task
pre-training results in comparable performance to our baseline.
- Interestingly, the performance of “leave-one-out” training was only slightly
worse, suggesting that a model that was trained on a variety of tasks can still adapt to new tasks (i.e. multi-task pre-training might not result in a dramatic task interference)
- Instead of pre-training 4 separate models, a cheaper alternative is to take
a single pre-trained model and produce 4 separate fine-tuned versions
- Separately, we note that ensembling N separate models
has a similar cost to using a model that has an N× higher computational cost.
-
Language agnostic models