Menu Close

Google researchers developed and benchmarked techniques which enabled them to train a language model containing more than a trillion parameters

Parameters are the key to machine learning algorithms. They’re the part of the model that’s learned from historical training data. Generally speaking, in the language domain, the correlation between the number of parameters and sophistication has held up remarkably well. For example, OpenAI’s GPT-3 — one of the largest language models ever trained, at 175 billion parameters — can make primitive analogies, generate recipes, and even complete basic code.

In what might be one of the most comprehensive tests of this correlation to date, Google researchers developed and benchmarked techniques they claim enabled them to train a language model containing more than a trillion parameters. They say their 1.6-trillion-parameter model, which appears to be the largest of its size to date, achieved an up to 4 times speedup over the previously largest Google-developed language model (T5-XXL).

As the researchers note in a paper detailing their work, large-scale training is an effective path toward powerful models. Simple architectures, backed by large datasets and parameter counts, surpass far more complicated algorithms. But effective, large-scale training is extremely computationally intensive. That’s why the researchers pursued what they call the Switch Transformer, a “sparsely activated” technique that uses only a subset of a model’s weights, or the parameters that transform input data within the model.

READ more: Google trained a trillion-parameter AI language model


Leave a Reply

Your email address will not be published. Required fields are marked *

(UN General Assembly, 1948) The Universal Declaration of Human Rights: 1. All human beings are free and equal 2. No discrimination 3. Right to life 4. No slavery 5. No torture and inhuman treatment 6. Same right to use law 7. Equal before the law 8. Right to be treated fair by court 9. No unfair detainment 10. Right to trial 11. Innocent until proved guilty 12. Right to privacy 13. Freedom to movement and residence 14. Right to asylum 15. Right to nationality 16. Rights to marry and have family 17. Right to own things 18. Freedom of thought and religion 19. Freedom of opinion and expression 20. Right to assemble 21. Right to democracy 22. Right to social security 23. Right to work 24. Right to rest and holiday 25. Right of social service 26. Right to education 27. Right of cultural and art 28. Freedom around the world 29. Subject to law 30. Human rights can’t be taken away