Research

Liars' Bench: Evaluating Lie Detectors for Language Models

Lie detection techniques are typically validated in narrow settings that do not capture the diverse lies LLMs can generate. Liars' Bench is a testbed of 72,863 examples of lies and honest responses generated by four open-weight models across seven datasets. The settings vary along two dimensions: the model's reason for lying and the object of belief targeted by the lie.

• No existing lie detector beat chance on all seven datasets.
• LLM judges mostly fact-check, and fail on lies about the model itself.
• Detectors struggle most when the transcript alone cannot reveal whether the model lied.

Used by: Anthropic's Alignment Science team based its Harm Pressure setting on ours and used our taxonomy to select its testbeds; UK AISI compares against our model organisms; Google DeepMind engages with our datasets in depth; LASR Labs/UK AISI and FAR.AI's Obfuscation Atlas evaluate on our data.

Cluster-Norm for Unsupervised Probing of Knowledge

Salient but unrelated features in a dataset can mislead unsupervised probes. Cluster-Norm clusters and normalizes the activations of contrast pairs before probing, minimizing the impact of such distractors without needing to know in advance what they are.

In the example shown, a random word ("banana" or "shed") is appended to each prompt. With standard normalization, the top principal component separates the random words. With cluster normalization, it separates the true and false statements instead.

AI–AI Bias

Do LLMs favor content written by other LLMs? Asked to choose between items described by AI-generated or human-written text (products, paper abstracts and movies), LLMs consistently preferred the AI-generated descriptions, while human evaluators showed a much weaker preference, or none.

This suggests a potential "anti-human" bias that could disadvantage human workers as LLMs are increasingly used in economic decision-making. The paper was covered by several news outlets, including Forbes.