"Google’s new large language model (LLM) called PaLM (Pathways Language Model) is the first outcome of Pathways, Google’s new AI architecture, which aims to handle many tasks at once, learn new tasks quickly and reflect a better understanding of the world."

"The number of parameters is important in LLMs, although more parameters don’t necessarily translate to a better-performing model."

"PaLM 540B is in the same league as some of the largest LLMs available regarding the number of parameters:"

"OpenAI’s GPT-3 with 175 billion, DeepMind’s Gopher and Chinchilla with 280 billion and 70 billion, Google’s own GLaM and LaMDA with 1.2 trillion and 137 billion and Microsoft – Nvidia’s Megatron–Turing NLG with 530 billion."

"The first thing to consider when discussing LLMs, like any other AI model, is the efficiency of the training process."

"“Given a certain quantity of compute, how large of a model should I train in order to get the best possible performance?”"

"In 2022, DeepMind published a paper, “Training Compute-Optimal Large Language Models,” in which analysts claim that training LLMs has been done with a deeply suboptimal use of compute."

"At the hardware level, PaLM 540B was trained over two TPU v4 Pods connected over a data center network (DCN) using a combination of model and data parallelism."

"PaLM uses a standard Transformer model architecture, with some customizations."

"First, it’s worth asking whether the selection of sources reflects Google’s goals. Social media conversations are by far the most prevalent source and while web pages have been selected taking their assigned quality scores into account, that doesn’t seem to be the case for social media conversations."

"However, Google notes, this may have disproportionately excluded casual language, code-switching (or behavioral adjustments in actions or speech) or dialectal diversity, and may limit PaLM’s capability to model the nondominant dialects across the English-speaking regions globally."

"Google notes that the language capabilities of PaLM are likely constrained by the limitations of language present in the training data and evaluation benchmarks."

"Google’s vision for Pathways is to “enable a single AI system to generalize across thousands or millions of tasks, to understand different types of data and to do so with remarkable efficiency.”"

"To summarize, it seems that PaLM can do more with less — i.e., achieve comparable or better performance to existing state-of-the-art LLMs, while needing fewer resources and less customization than they do."