ChainThink report, on August 11, AI Agent infrastructure company Composio integrated DeepSeek V4 Flash into 8 Harnesses and conducted testing using the same batch of 30 Agent tasks, covering real-world applications such as Gmail, GitHub, Slack, Calendar, and Notion.

The results showed that Pi Agent achieved 20/30, with the highest success rate;

Oh My Pi reached 17/30, while Claude Code, Codex, and Deep Agents each scored 16/30, Hermes Agent achieved 15/30, Prime Agent reached 15/24, and OpenCode scored 14/30.

Simply switching the Harness boosted DeepSeek V4 Flash’s successful task count from 14 to 20.

In terms of cost and speed, Pi Agent incurred approximately $0.028 per successful task, whereas Claude Code cost $0.195—nearly 7 times higher;

Claude Code had a median execution time of 122.7 seconds, the fastest among all tested agents, while Oh My Pi recorded a median time of 272.4 seconds, the slowest.

Composio also noted that Pi’s test configuration deviated from uniform conditions: it used "high" rather than "max" inference intensity, and 24 out of the 30 tasks were routed through DeepSeek’s official API instead of OpenRouter, meaning the performance gap may not be entirely attributable to the Harness alone.

Previously, in Kimi K3’s Harness benchmarking, Oh My Pi ranked first with 22/25, while the original Pi Agent scored 18/25.