How Does PaLM 2 Compare With GPT-4 and Gemini?

TL;DR

PaLM 2 is broadly competitive with GPT-4, although GPT-4 scores higher on a direct appendix comparison and PaLM 2 appears stronger in translation, multilingual tasks, and some mathematics tests. Google made PaLM 2 smaller, cheaper, and faster than PaLM, while withholding architecture, parameter, compute, and training-source details. Gemini was already in training with planning capabilities, raising broader safety concerns.

Transcript

less than 24 hours ago Google released the Palm 2 technical report I have read all 92 Pages watch the Palm 2 presentation read the release notes and have already tested the model in a dozen ways but before getting into it all my four main takeaways are these first Palm 2 is competitive with gpt4 and while it is probably less smart overall it's bett... Read More

Key Insights

• PaLM 2 is broadly competitive with GPT-4, but the evidence does not establish that it is smarter overall. A direct appendix comparison placed Flan-PaLM 2 at 81.2 versus GPT-4 at 86.4, while individual tests showed each model answering questions the other missed.
• PaLM 2 is estimated to contain between 100 and 200 billion parameters based on the report's scaling information and estimated training compute. Google withheld the actual model size and architecture, stating only that PaLM 2 is significantly smaller than the 540-billion-parameter PaLM model.
• PaLM 2 is designed around a multilingual and diverse pretraining mixture covering hundreds of languages, programming, mathematics, documents, books, and conversational data. Google did not identify the specific sources of that material, leaving important questions about the training corpus unanswered.
• PaLM 2 is particularly capable at translation and linguistic tasks, with the technical report claiming performance above Google Translate for certain languages. Unlike GPT-4's reported English advantage, PaLM 2 often performed comparably or better in other languages on the benchmarks presented.
• Benchmark comparisons are sensitive to prompting methods because Google used chain-of-thought prompting and self-consistency for some PaLM 2 results. Self-consistency selects the most consistent answer from multiple outputs, so comparisons may be unfair when GPT-4 is evaluated without an equivalent strategy.
• PaLM 2 is smaller, cheaper, and faster to serve than PaLM, according to Google. Bard's PaLM 2-powered responses were observed at roughly ten times the inference speed of GPT-4 for the same prompt, although factors beyond parameter count can affect inference speed.
• The PaLM 2 family is offered in four sizes named Gecko, Otter, Bison, and Unicorn. Gecko is lightweight enough to operate on mobile devices, support interactive applications, and work offline, illustrating Google's plan to deploy language models across very different computing environments.
• Gemini is already in training and is being given planning abilities, while Sam Altman reportedly said GPT-5 was not in training and would not be for a long time. The analysis argues that Google's emphasis on capability development is not matched by discussion of broader AI impacts.

Questions & Answers

Q: How does PaLM 2 compare with GPT-4?

PaLM 2 is competitive with GPT-4 but appears somewhat less capable overall. In a direct appendix comparison, the original PaLM 2 scored 78.3, instruction-tuned Flan-PaLM 2 scored 81.2, and GPT-4 scored 86.4. However, PaLM 2 showed advantages in translation, multilingual work, and selected mathematics problems, including cases where it answered advanced questions that GPT-4 missed.

Q: Why is PaLM 2 strong at translation and multilingual tasks?

PaLM 2 was pretrained on a diverse mixture extending across hundreds of languages and domains, giving non-English material greater importance in its development. Its benchmark performance in English was not dramatically better than its performance in many other languages, unlike the reported pattern for GPT-4. Google also claimed that PaLM 2 exceeded Google Translate for certain languages and passed mastery exams across several languages.

Q: How many parameters does PaLM 2 have?

Google did not disclose PaLM 2's exact parameter count, model architecture, or training compute. The technical report said it was significantly or dramatically smaller than the largest PaLM model, which contained 540 billion parameters. By extending the report's optimal parameter scaling information to the estimated training compute, the analysis derives a possible range of 100–200 billion parameters, but this remains an educated estimate rather than a confirmed figure.

Q: Why are some PaLM 2 and GPT-4 benchmarks difficult to compare?

Some favorable PaLM 2 results used chain-of-thought prompting and self-consistency, while the GPT-4 results may not have used the same methods. Self-consistency generates multiple outputs and chooses the answer that appears most consistently, which can improve benchmark scores. Consequently, a PaLM 2 victory under advanced prompting does not necessarily demonstrate superior underlying performance unless GPT-4 receives an equivalent evaluation procedure.

Q: What are the PaLM 2 model sizes and their purposes?

Google named four PaLM 2 sizes: Gecko, Otter, Bison, and Unicorn. The transcript provides the most detail about Gecko, describing it as lightweight enough to run on mobile devices, respond quickly in interactive applications, and work offline. The range indicates that PaLM 2 was designed for deployment across different computing environments rather than being limited to one large, centrally served model.

Q: How capable is PaLM 2 at coding?

The available comparisons do not establish that PaLM 2 is better than GPT-4 at coding. PaLM 2's specialized coding model reportedly achieved 37.6 percent pass-at-one accuracy, while the cited Sparks of AGI paper gave GPT-4 an 82 percent zero-shot pass-at-one result. That comparison is uncertain because the paper acknowledged that GPT-4 might have seen or memorized some or all of the HumanEval material.

Q: What does PaLM 2 reveal about emergent abilities?

Google argued that PaLM 2 demonstrates emergent abilities in multi-step arithmetic, temporal sequences, and hierarchical reasoning. Early tests described in the analysis produced inconsistent behavior: PaLM 2 solved some advanced college-level mathematics questions that GPT-4 missed, yet failed a basic question about prime numbers. These uneven results complicate simple claims about intelligence and contrast with papers arguing that emergent abilities may be a measurement mirage.

Q: What safety concerns were raised about PaLM 2 and Gemini?

The PaLM 2 technical report devoted about 20 pages to bias, toxicity, and misgendering but did not dedicate a page to broader AI impacts, according to the analysis. Google also announced that Gemini was already in training and would receive planning abilities. The concern is that rapid capability development, including planning, was presented without a comparably extensive discussion of wider societal risks or safeguards.

Summary & Key Takeaways

• PaLM 2 uses a multilingual and diverse pretraining mixture spanning hundreds of languages, programming, mathematics, books, documents, and conversational data. The report withholds exact sources, architecture, parameter count, and compute, although scaling information suggests an estimated optimal range of 100–200 billion parameters and a much smaller model than the 540-billion-parameter PaLM.

• Benchmark evidence presents a mixed comparison with GPT-4. Flan-PaLM 2 scored 81.2 compared with GPT-4 at 86.4 on one direct appendix comparison, yet PaLM 2 exceeded GPT-4 on some mathematics tests. Several favorable results used chain-of-thought prompting and self-consistency, making comparisons against GPT-4 difficult when prompting methods differed.

• PaLM 2 demonstrates particular strength in multilingual tasks, sometimes exceeding Google Translate and performing similarly or better outside English. Its model family ranges from the mobile-oriented Gecko to Otter, Bison, and Unicorn. Google also announced Gemini in training with planning capabilities, while the report devoted extensive attention to bias and toxicity but little to broader societal impacts.