Google’s Gemini is Not in the Cards

Open disclosure has been the acceptable practice for AI until May 2023, when OpenAI announced GPT-4 and released little in the way of technical detail. The paper provided by the company did not allow researchers to understand its structure or make any attempt at replicating its effects.

Google’s announcement of Gemini followed suit. Google and DeepMind researchers posted a blog without technical specifications and a technical report that, well, wasn’t.

While the blog and report listed benchmark scores and claims that Google beat OpenAI’s GPT-4 on most measures and its former top neural network, PaLM, there are no real details. Google refers to three versions of Gemini, with three different sizes, “Ultra,” “Pro,” and “Nano.” There’s no mention that Nano is trained with two different weight counts, 1.8 billion and 3.25 billion, and the weights of the other two sizes remain undisclosed. Lacking information means that assessments are being made in hit-or-miss fashion by people randomly typing things to Bard.

In October, scholars Emanuele La Malfa at the University of Oxford and collaborators at The Alan Turing Institute and the University of Leeds warned that the obscurity of GPT-4 and other models “causes a significant problem” for AI for society. “The most potent and risky models are also the most difficult to analyze.”

Google’s lack of disclosure is striking because of the omission of model cards. Model cards are standard disclosure used in AI to report on the details of neural networks, including potential harm of the program. While the GPT-4 report from OpenAI omitted most details, there was a “GPT-4 System Card” section in the paper, which it said was inspired by model cards.

Google, however, omits anything resembling model cards. Instead of model cards, the report offers a brief, bizarre passage about the deployment of the program with vague language about having model cards at some point: “Following the completion of reviews, model cards ?? [emphasis Google’s] for each approved Gemini model are created for structured and consistent internal documentation of critical performance and responsibility metrics as well as to inform appropriate external communication of these metrics over time.”

If Google puts question marks next to model cards in its own technical disclosure, the future of oversight and safety is up in the proverbial air.