Experts

From a pool of hundreds of applicants, we selected 35 top tier evaluators including Hollywood comedy executives, stand-up comedians, and professional comedy writers.

As part of our vetting process, each potential evaluator went through the following process:

(1) Live interview assessing comedic taste and analysis skills. Applicants were primarily judged on their ability to dissect comedic performance into component parts and calibrate quality.

(2) Small evaluation exercise where potential evaluators were asked to give feedback on two setup-punchline style jokes delivered by different speech models. In rating these responses we looked for strong demonstration of comedy expertise, thoroughness, and any perceived bias.

(3) Resume and professional credential screening. Here, we judged evaluators on pedigree and accomplishments – e.g., have they contributed to high performing teams and productions? Which / how many / how long?

The expert evaluation process was overseen by co-founder and CEO Chris Giliberti, Executive Producer on comedy productions such as ALEX INC, which starred Zach Braff on ABC. We additionally consulted with Jeff Stern, former President of Apatow Productions & producer on comedy productions such as LIVING WITH YOURSELF on Netflix.

In the end, our evaluator pool broke down as follows:

• 65% comedy creative executives (professionals who review and improve the work of comedians in Hollywood), with employers such as Amazon, Netflix, Sony, and Warner Brothers Discovery
• 35% comedy writers, stand up comedians, and comedic actors with writing and performance credits on Paramount, Adult Swim, Comedy Central

Up Next

Armed with learnings from this preliminary research, we are executing a larger research effort with the dual goals of:

1. Shipping the voice AI field’s first humor performance benchmark
2. Creating a higher confidence leaderboard of the funniest speech models

This round is significantly larger in respondents (N=100+) and judgements (thousands in aggregate.)

We are employing a pairwise ‘arena-style’ evaluation to generate both a fresh leaderboard and ELO-based ranking. Most data, code, and outputs will be published.

To create a production-ready version of HumorBench, we will:

• Run our professional comedian reads by crowdworkers to increase our confidence they are indeed a solid ground truth
• Perform cluster analysis on both comedian and AI samples to understand separability
• Tune a classifier capable of automatically scoring the humor quality of audio samples (or at least their spectrograms), as defined by alignment with ground truth

Key Metrics

Performance Quality

Using a 4-point Likert scale, we asked evaluators to rate individual joke performances. When relevant, we asked for separate ratings for the “setup” and “punchline” portions of a joke to collect more granular quantitative feedback.

Qualitative Feedback

We asked evaluators to leave feedback for each joke performance via short voice notes. We conducted manual and automated sentiment analysis on the voice note transcripts.

“Perceived Human” Output

We asked evaluators to guess whether each performance was AI or human, selecting from the following: Probably AI, Definitely AI, Probably Human, Definitely Human.

“Top Pick” Output

At the end of our survey, evaluators were asked to pick a favorite read from all 10 AI outputs. They were able to relisten to samples as many times as required, with all samples arrayed on the same page. We conducted a simple tally of the evaluators’ favorite AI output across each joke and model.

Limitations

This initial research functioned more as a ‘focus group’. Humor is difficult to measure, so we are working in phases to establish insights and build up to our goal of a truly automated benchmark.

As you interpret our findings, we’d like to acknowledge several limitations:

• Our sample size is relatively small (N=35). In subsequent research, we plan to scale N into the 100s.
• We generated outputs in a single shot, with just two outputs judged per model. Model outputs can vary widely across generations, and even more widely with manipulations to temperature and other settings. In our next round, we will include many more outputs per model, both out of fairness to capture a wider spectrum of quality, and to capture more overall judgments.
• There is an existing body of research on speech model quality which we have not referenced here nor factored into our own analysis. We plan to be more referential in future work.

Contributors

Chris Giliberti - Avail Co-founder & CEO. Former head of TV at Spotify, and management consultant at The Boston Consulting Group (BCG)
Ryan Riebling - Avail Co-founder & Head of Eng. Former Amazon engineering lead
Ryan Shellberg - Avail senior engineer. Former Amazon engineer
Benjamin Genchel - ML Researcher. Former Spotify & Amazon ML engineer
Jeff Stern - Producer. Former head of development, Apatow Productions, and executive producer, Living with Yourself (Netflix)

Study Design

Evaluators rated two jokes each from 10 models. Joke #1 contained a set-up & punchline, and joke #2 was a sarcastic statement. Jokes were chosen for their simplicity of successful delivery: each was hypothesized to only ‘work’ with emphasis or pitch variance placed on specific words. (Our outputs from professional comedians suggested otherwise, which will inform our next round.)

Jokes were analyzed for:

• Quality of set-up delivery (Likert scale)
• Quality of punchline delivery (Likert scale)
• Anthropic quality of the voice (Likert scale)
• Opportunities for delivery improvement (delivered as an unstructured voice note)
• Top pick (select favorite of 10)

To minimize demographic bias, we selected a medium-tenor male voice (chosen to resemble the delivery style of a stereotypical stand-up comic) across all models. Each model had one opportunity to generate the joke. No temperature, pacing, or delivery parameters were adjusted. For models with prompt boxes, we used a consistent instruction: “Stand-up comedian, comedic delivery”

We used a 4-point Likert scale to encourage decisive evaluator feedback (no neutral option). Acknowledging that there is a “ground truth” challenge with humor, we selected jokes that we believed would have clear points of emphasis across professional comedians’ delivery. Additionally, as part of our research, we asked our evaluators (all expert comedy executives, comedians, and comedy writers) to provide their ideal delivery of the test jokes - in both writing and verbally - to help establish ground truth data points.

We additionally included human honeypots for calibration.

Model Configurations

We tested ten speech models, making sure to include top models from industry leaders as well as emerging ones (eg. Sesame, which had a splashy launch earlier this year). Selection was based on a mix of popularity, technical reputation (e.g. preference for foundation models) and market factors. Each model was prompted to perform the same joke using default settings (see “Study Design” below for more). All models tested are proprietary.