The number of parameters is important in LLMs, although more parameters don’t necessarily translate to a better-performing model.

PaLM 540B is in the same league as some of the largest LLMs available regarding the number of parameters: OpenAI’s GPT-3 with 175 billion, DeepMind’s Gopher and Chinchilla with 280 billion and 70 billion, Google’s own GLaM and LaMDA with 1.2 trillion and 137 billion and Microsoft – Nvidia’s Megatron–Turing NLG with 530 billion.

The first thing to consider when discussing LLMs, like any other AI model, is the efficiency of the training process.

PaLM uses a standard Transformer model architecture, with some customizations. Transformer is the architecture used by all LLMs and although PaLM deviates from it in some ways, what is arguably more important is the focus of the training dataset used.

The dataset used to train PaLM is a mixture of filtered multilingual web pages (27%), English books (13%), multilingual Wikipedia articles (4%), English news articles (1%), GitHub source code (5%) and multilingual social media conversations (50%). This dataset is based on those used to train LaMDA and GLaM.

Nearly 78% of all sources are English, with German and French sources at 3.5% and 3.2% and all other sources trailing far behind.

PaLM 540B surpassed few-shot performance of prior LLMs on 28 of 29 tasks.

PaLM outperforms the prior top score of 55% achieved by fine-tuning GPT-3 with a training set of 7,500 problems and combining it with an external calculator and verifier. This new score also approaches the 60% average of problems solved by 9- to 12-year-olds — the target audience for the question set.