The author reports that a GenD detector, reporting 91.2% average cross-dataset AUROC on 14 academic benchmarks, dropped to just over 60% on Incode's identity verification data.
Why deepfake detection benchmarks fall apart on production data ↗
What happened
- The author reports that Incode's labeling experiment found its current model had a lower error rate than the average of five expert labelers.
- The author reports that Incode's few-shot adaptation cut its error rate on new generators roughly 10x.
- The author states that Incode is building an agent-based system that scans for new generative models and tests its detector against their output.
Why it matters to Signicat
- Vendor-reported internal metrics (AUROC drop, 10x error reduction) are uncorroborated; Signicat should not treat them as established. Relevant to document_idv_biometrics and fraud_deepfake_injection positioning.
- Content frames deepfake detection as one part of agentic fraud and whole-capture-process defense, a narrative Signicat can compete on with its own fraud and orchestration offering.
Evidence from the source (5 quotes)
- “Our current model had a lower error rate than the average labeler.” source ↗
- “That cut our error rate on new generators roughly 10x, without hurting accuracy on the ones we already knew.” source ↗
- “We are building an agent-based system that scans the internet for new generative models, evaluates whether they are capable of producing convincing face impersonations, figures out how to access them, and tests our detector against their output automatically.” source ↗
- “That cut our error rate on new generators roughly 10x, without hurting accuracy on the ones we already knew.” source ↗
- “deepfake detection becomes one piece of a much larger fraud problem, not the whole of it.” source ↗
Fraud deepfake injectionDocument IDV biometricsAgentic AI identityThought leadershipGlobal
4 of 4 facts verified against the source · confidence medium · claude-haiku-5-5