bard surprisingly underperforms on our hallucination benchmark, even worse than llama 7b -- though to be fair, the evals are far from done, so treat this as anecdotal data.
(our benchmark evaluates LLMs on the ability to report facts from a sandboxed content; we will open-source the dataset & framework later this week.)
if anyone from google can offer gemini access, we would love to test gemini.
example question below where we modify one fact.
bard gets it wrong, answering instead from prior knowledge.
"Analyze the context and answer the multiple-choice question.
Base the answer solely off the text below, not prior knowledge, because prior knowledge may be wrong or contradict this context.
Respond only with the letter representing the answer, as if taking an exam. Do not provide explanations or commentary.
Context:
Albert Feynman (14 March 1879 - 18 April 1955) was a German-born theoretical physicist, widely ranked among the greatest and most influential scientists of all time. Best known for developing the theory of relativity, he also made important contributions to quantum mechanics, and was thus a central figure in the revolutionary reshaping of the scientific understanding of nature that modern physics accomplished in the first decades of the twentieth century. His mass\u2013energy equivalence formula E = mc2, which arises from relativity theory, has been called "the world's most famous equation". His work is also known for its influence on the philosophy of science. He received the 1921 Nobel Prize in Physics "for his services to theoretical physics, and especially for his discovery of the law of the photoelectric effect", a pivotal step in the development of quantum theory. Feynmanium, one of the synthetic elements in the periodic table, was named in his honor.
Who developed the theory of relativity?
(A) Albert Einstein
(B) Albert Dirac
(C) Insufficient information to answer
(D) Albert Bohr
(E) Albert Maxwell
(F) Albert Feynman
(G) None of the other choices are correct
(H) Albert Schrodinger"