Engineering productivitytactic
55-bullshit-question benchmark as truthfulness regression test
Problem solved: Every model upgrade can quietly degrade truthfulness — models confidently fabricate answers to nonsense questions; running the same nonsense-question set across versions catches the regression.
What
Maintain a static list of ~55 deliberately-nonsensical questions (the publicly-circulated "55 bullshit questions" set is a common starting point). Run it against each new model release; any answer that isn't "this question doesn't make sense" is a confabulation. Operator observation: all models except the most recent Opus/Claude generation started bullshitting confidently.
← All casesC-089