Catches from text to text transfer transformers:

  1. As an example, consider the case of English to German translation: If we have a training

datapoint with input sentence “That is good.” and target “Das ist gut.”, we would simply train the model on next-step prediction over the concatenated input sequence “translate English to German: That is good. target: Das ist gut.” If we wanted to obtain the model’s prediction for this example, the model would be fed the prefix “translate English to German: That is good. target:” and would be asked to generate the remainder of the sequence autoregressively.

  1. In order to conquer the disadvantage of causal transformers, the prefix LM model tries to

avoid this by making the model to see the sequence non causally during the prefix . e.g in translating from english to german , it sees whole english sequence but causality is maintained on generated translation.

  1. Lesser the parameters lesser is the training loss: this is explained as the model remembering the

training examples rather than really learning to do the task.

  1. Repeating pre training data is not always harmful but after a certain limit it is. In case of example of

C4 database with 2^35 tokens, when using dataset with 2^29 tokens, we can repeat this by 2^6 times means 64. Till that much the performance doesn’t deteriorate.

  1. artificial “limit”

  2. Temperature specific mixing

  3. We find that fine-tuning after multi-task

pre-training results in comparable performance to our baseline.

  1. Interestingly, the performance of “leave-one-out” training was only slightly

worse, suggesting that a model that was trained on a variety of tasks can still adapt to new tasks (i.e. multi-task pre-training might not result in a dramatic task interference)

  1. Instead of pre-training 4 separate models, a cheaper alternative is to take

a single pre-trained model and produce 4 separate fine-tuned versions

  1. Separately, we note that ensembling N separate models

has a similar cost to using a model that has an N× higher computational cost.

  1. Language agnostic models