#1 · score 24/25
CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
Andrej Leban, Yuekai Sun · 2026-07-09
直接服务代码智能体与自动化开发评测。
Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce CausalDS, a benchmark for evaluating causal reasoning in agentic data-science workflows. Each b
Relevance 5Novelty 5Substance 5Evidence 4Actionability 5
arXiv · PDF
#2 · score 23/25
Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions
Eduardo Almeida Palmieri, Mohamed Chahine Ghanem, Dipo Dunsin, Zubair Baig, Ed de Quincey · 2026-07-03
对 agent 规划、工具使用或长程任务很有启发。
The rapid growth of publicly available digital information has rendered manual open-source intelligence (OSINT) analysis insufficient for modern intelligence, cybersecurity, and cyber investigation. Large language models (LLMs) and agentic AI systems, capable of tool use, multi-step reasoning, and iterative intelligence generation, have emerged as promising solutions, yet evaluation frameworks have not kept pace with reported capabilities. This survey systematically reviews 74 studies and makes four contributions. First, it establishes agentic AI as a distinct analytical category rather than an extension of LLM prompting, organising the literature through an 11-category taxonomy covering LLM
Relevance 5Novelty 4Substance 5Evidence 4Actionability 5
arXiv · PDF
#3 · score 23/25
Reliable and Developer-Aligned Evaluation of Agents for Software Engineering
Razvan Mihai Popescu · 2026-07-08
直接服务代码智能体与自动化开发评测。
Large language models are rapidly moving towards closing the development cycle, transitioning from simple assistive companions to autonomous contributors deeply embedded into collaborative development environments. Despite their accelerated adoption, existing evaluation techniques are limited due to their fragmented nature and distorted projection of true model capabilities, often obtained from hypothetical syntactic scenarios. This research aims to bridge this gap by providing a comprehensive evaluation methodology for LLM-powered agents that is grounded in real-world software development practice. Our evaluation approach focuses on contamination-awareness, in-the-wild agentic behavior asse
Relevance 5Novelty 4Substance 5Evidence 4Actionability 5
arXiv · PDF
#4 · score 23/25
AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation
Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy · 2026-07-07
直接服务代码智能体与自动化开发评测。
We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to dia
Relevance 5Novelty 4Substance 5Evidence 4Actionability 5
arXiv · PDF
#5 · score 21/25
TF-Engram: A Train-Free Engram with SSD-Backed Memory for Large Language Models
Yutang Ma, Kecheng Huang, Xikun Jiang, Zili Shao · 2026-07-08
直接服务代码智能体与自动化开发评测。
Large Language Models (LLMs) store factual knowledge and domain-specific patterns implicitly in dense Transformer parameters, making knowledge expansion costly through pretraining, fine-tuning, retrieval augmentation, or longer contexts. Engram-style memory offers a compact hidden-state injection pathway, but existing GPU-resident designs often rely on hash-based compression, causing unrelated phrases to collide in shared slots and weakening phrase-level semantic fidelity. We present TF-Engram, a train-free Engram system that constructs phrase-specific semantic memory offline from external corpora, stores large memory tables across a GPU--DRAM--SSD hierarchy, and uses Early-Exit Guided Predi
Relevance 5Novelty 3Substance 5Evidence 3Actionability 5
arXiv · PDF
#6 · score 24/25
PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization
Yingyun Cui, Yi Xie, Piaohong Wang, Jiawei Ma, Bo Liu · 2026-07-08
直接服务代码智能体与自动化开发评测。
Coding-agent benchmarks have largely measured whether agents can produce functionally correct patches, but production software also demands measurable speedups on real execution targets. Performance optimization is a distinct agentic task: agents must profile executions, diagnose cross-layer bottlenecks, edit code without breaking correctness, and verify that gains are reproducible rather than measurement artifacts. We introduce PERFOPT-Bench, a benchmark for evaluating this full performance-engineering loop. Each task provides a correct but deliberately suboptimal codebase and asks the agent to improve a target performance metric; scoring requires hidden correctness tests, verified-speedup
Relevance 5Novelty 5Substance 5Evidence 4Actionability 5
arXiv · PDF
#7 · score 23/25
DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
Wenqi Huang, Charley Lee, Leonard Tng, Serena Ge · 2026-07-09
直接服务代码智能体与自动化开发评测。
DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositor
Relevance 5Novelty 4Substance 5Evidence 4Actionability 5
arXiv · PDF
#8 · score 21/25
Harnessing Code Agents for Automatic Software Verification
Shuangxiang Kan, Shuanglong Kan, Sebastian Ertel · 2026-07-07
直接服务代码智能体与自动化开发评测。
Formal verification offers the strongest guarantee of software correctness, but it does not scale: the proofs demanded by interactive theorem provers such as Coq require enormous expert effort. Large language models (LLMs) promise to generate these proofs automatically, yet existing approaches wire a fixed, human-designed proof strategy into the system and constrain the model to follow it (retrieving premises and predicting tactics one step at a time, or splitting goals by divide-and-conquer), and still prove only a fraction of their target theorems. We show that imposing such a strategy is unnecessary and limiting. Handing the whole lemma to a general LLM code agent (for example, Claude Cod
Relevance 5Novelty 3Substance 5Evidence 3Actionability 5
arXiv · PDF
#9 · score 23/25
The Poisoned Chalice of LLM Evaluation Report
Jonathan Katzy, Ali Al-Kaswan, Razvan Mihai Popescu, Zhou Yang · 2026-07-08
直接服务代码智能体与自动化开发评测。
Large language models are increasingly used to evaluate and support software engineering tasks, yet the validity of these evaluations is often undermined by uncertainty about whether benchmark instances were seen during pretraining. This can lead to data contamination, which may inflate performance and result in misleading conclusions about model capability. Despite this, the training corpora of many modern models are only partially disclosed, making direct decontamination infeasible. This creates a need for practical methods that can detect a large language models' prior exposure to training data without access to the full training corpus. To address this challenge, we organize the first Po
Relevance 5Novelty 5Substance 4Evidence 4Actionability 5
arXiv · PDF
#10 · score 23/25
Detecting Vulnerability-Inducing Commits via Multi-Stage Reasoning with LLM-Based Agents
Liyou Chen, Hailong Sun, Xiang Gao, Yue Pan · 2026-07-07
直接服务代码智能体与自动化开发评测。
Detecting vulnerability-inducing commits (VICs) at submission time is critical for improving the security and reliability of software systems. However, this task is highly challenging because it requires reasoning about the semantic impact of code changes from heterogeneous information sources, including code diffs, commit messages, and the surrounding contextual code. Existing approaches often struggle to fully capture these complex interactions, resulting in limited detection performance. In this paper, we propose VIC-RAGENT, an LLM-based multi-agent framework for effective and explainable vulnerability detection. VIC-RAGENT leverages multiple specialized agents to provide complementary pe
Relevance 5Novelty 4Substance 5Evidence 4Actionability 5
arXiv · PDF
#11 · score 21/25
TTHE: Test-Time Harness Evolution
Jun Nie, Yonggang Zhang, Jun Song, Qianshu Cai, Dahai Yu · 2026-07-09
直接服务代码智能体与自动化开发评测。
The behavior of an LLM agent is determined not only by the underlying model, but also by its harness: the executable program that constructs context, invokes tools, verifies intermediate results, and recovers from failures. Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time. This limits adaptation when the test distribution, failure modes, or tool interactions differ from those seen during development. We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs. We introduce Test-Time H
Relevance 5Novelty 3Substance 5Evidence 3Actionability 5
arXiv · PDF
#12 · score 20/25
AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation
Chenyu Zhao, Shenglin Zhang, Wenwei Gu, Yongqian Sun, Dan Pei · 2026-07-07
对 agent 规划、工具使用或长程任务很有启发。
Large language model (LLM) agents are increasingly used for multi-step, stateful tool-use tasks, yet production reliability remains limited. Unlike static software repair, agent repair must recover dynamic trajectories whose early decisions can propagate into later errors and external state changes. Existing automatic remedies address only part of this problem: blind retry adds no diagnosis, outcome feedback says whether a run failed but not where or why, and self-reflection often lacks grounded evidence to prevent the same failure from recurring. We present AgentTether, a run-time repair framework that automates post-run diagnosis and guided recovery without modifying the underlying agent o
Relevance 5Novelty 3Substance 5Evidence 2Actionability 5
arXiv · PDF
数据源说明
Relevance Novelty Substance Evidence Actionability；本次为 arXiv latest-release fallback，不声称 7/11–7/12 有新 arXiv release。