ling/hermes-agent
1
0
Fork 0
mirror of https://github.com/NousResearch/hermes-agent.git synced 2026-05-04 17:57:28 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
4af69097f24a7423fdfc7c2dc6940024078718c8
hermes-agent/environments/benchmarks/terminalbench_2/default.yaml

61 lines
2.9 KiB
YAML
Raw Normal View History

Add new environments and enhance tool context functionality - Introduced new environments: Terminal Test Environment and SWE Environment, each with default configurations for testing and software engineering tasks. - Added TerminalBench 2.0 evaluation environment with comprehensive setup for agentic LLMs, including task execution and verification. - Enhanced ToolContext with methods for uploading and downloading files, ensuring binary-safe operations. - Updated documentation across environments to reflect new features and usage instructions. - Refactored existing environment configurations for consistency and clarity.
2026-02-10 19:39:05 +00:00
# Terminal-Bench 2.0 Evaluation -- Default Configuration
#
# Eval-only environment for the TB2 benchmark (89 terminal tasks).
# Uses Modal terminal backend for per-task cloud-isolated sandboxes
# and OpenRouter for inference.
#
# Usage:
# python environments/benchmarks/terminalbench_2/terminalbench2_env.py evaluate \
# --config environments/benchmarks/terminalbench_2/default.yaml
#
# # Override model:
# python environments/benchmarks/terminalbench_2/terminalbench2_env.py evaluate \
# --config environments/benchmarks/terminalbench_2/default.yaml \
# --openai.model_name anthropic/claude-sonnet-4
env:
Update default.yaml
2026-04-01 17:59:27 -07:00
system_prompt: |
You are an expert software engineer and Linux systems administrator solving a task in an isolated Linux environment. You have terminal and file tools available.
Before each action, briefly analyze what you see and plan your next step. After running a command, read its output carefully before proceeding.
Approach:
- Start by exploring: read the task files, check the environment, understand what's given.
- Break complex problems into steps. Solve and verify each step before moving on.
- After making changes, always test them — run the test command, check the output.
- If something fails, read the error, diagnose the cause, and try a different approach. Do not give up or repeat the same failing command.
- Do not stop until you have verified your solution works.
update trajectory writing
2026-04-01 22:48:22 -07:00
When to stop: Once you believe your solution is complete and you have verified it works (e.g. the program runs correctly, the output looks right, the file is in place), respond with a plain text message summarizing what you did. Do NOT make any more tool calls after that.
Add new environments and enhance tool context functionality - Introduced new environments: Terminal Test Environment and SWE Environment, each with default configurations for testing and software engineering tasks. - Added TerminalBench 2.0 evaluation environment with comprehensive setup for agentic LLMs, including task execution and verification. - Enhanced ToolContext with methods for uploading and downloading files, ensuring binary-safe operations. - Updated documentation across environments to reflect new features and usage instructions. - Refactored existing environment configurations for consistency and clarity.
2026-02-10 19:39:05 +00:00
enabled_toolsets: ["terminal", "file"]
add top_p + user nudges for incorrect format
2026-04-06 10:05:25 -07:00
max_agent_turns: 100
Update terminalbench_2 configuration for enhanced performance and evaluation - Increased max_token_length from 16000 to 32000 to allow for longer inputs. - Adjusted agent_temperature from 0.6 to 0.8 for more varied responses. - Extended test_timeout from 180 to 600 seconds to accommodate longer evaluations. - Updated data directory path for saving evaluations to ensure proper organization.
2026-02-10 19:48:41 +00:00
max_token_length: 32000
add top_p + user nudges for incorrect format
2026-04-06 10:05:25 -07:00
agent_temperature: 1.0
Add new environments and enhance tool context functionality - Introduced new environments: Terminal Test Environment and SWE Environment, each with default configurations for testing and software engineering tasks. - Added TerminalBench 2.0 evaluation environment with comprehensive setup for agentic LLMs, including task execution and verification. - Enhanced ToolContext with methods for uploading and downloading files, ensuring binary-safe operations. - Updated documentation across environments to reflect new features and usage instructions. - Refactored existing environment configurations for consistency and clarity.
2026-02-10 19:39:05 +00:00
terminal_backend: "modal"
wip: run tb2 and fix modal instantiation
2026-04-01 16:55:03 -07:00
terminal_timeout: 300 # 5 min per command (builds, pip install)
tool_pool_size: 128 # thread pool for 89 parallel tasks
add top_p + user nudges for incorrect format
2026-04-06 10:05:25 -07:00
dataset_name: "NousResearch/terminal-bench-2-verified-flattened"
Update terminalbench_2 configuration for enhanced performance and evaluation - Increased max_token_length from 16000 to 32000 to allow for longer inputs. - Adjusted agent_temperature from 0.6 to 0.8 for more varied responses. - Extended test_timeout from 180 to 600 seconds to accommodate longer evaluations. - Updated data directory path for saving evaluations to ensure proper organization.
2026-02-10 19:48:41 +00:00
test_timeout: 600
update trajectory writing
2026-04-01 22:48:22 -07:00
task_timeout: 900 # 15 min wall-clock per task, auto-FAIL if exceeded
Add new environments and enhance tool context functionality - Introduced new environments: Terminal Test Environment and SWE Environment, each with default configurations for testing and software engineering tasks. - Added TerminalBench 2.0 evaluation environment with comprehensive setup for agentic LLMs, including task execution and verification. - Enhanced ToolContext with methods for uploading and downloading files, ensuring binary-safe operations. - Updated documentation across environments to reflect new features and usage instructions. - Refactored existing environment configurations for consistency and clarity.
2026-02-10 19:39:05 +00:00
tokenizer_name: "NousResearch/Hermes-3-Llama-3.1-8B"
use_wandb: true
wandb_name: "terminal-bench-2"
ensure_scores_are_not_same: false
Update terminalbench_2 configuration for enhanced performance and evaluation - Increased max_token_length from 16000 to 32000 to allow for longer inputs. - Adjusted agent_temperature from 0.6 to 0.8 for more varied responses. - Extended test_timeout from 180 to 600 seconds to accommodate longer evaluations. - Updated data directory path for saving evaluations to ensure proper organization.
2026-02-10 19:48:41 +00:00
data_dir_to_save_evals: "environments/benchmarks/evals/terminal-bench-2"
fix: limit concurrent Modal sandbox creations to avoid deadlocks - Add max_concurrent_tasks config (default 8) with semaphore in TB2 eval - Pass cwd: /app via register_task_env_overrides for TB2 tasks - Add /home/ to host path prefixes as safety net for container backends When all 86 TerminalBench2 tasks fire simultaneously, each creates a Modal sandbox via asyncio.run() inside a thread pool worker. Modal's blocking calls deadlock when too many are created at once. The semaphore ensures max 8 concurrent creations. Co-Authored-By: hermes-agent[bot] <hermes-agent[bot]@users.noreply.github.com>
2026-03-07 21:34:06 +00:00
# CRITICAL: Limit concurrent Modal sandbox creations to avoid deadlocks.
# Modal's blocking calls (App.lookup, etc.) deadlock when too many sandboxes
# are created simultaneously inside thread pool workers via asyncio.run().
max_concurrent_tasks: 8
add top_p + user nudges for incorrect format
2026-04-06 10:05:25 -07:00
extra_body:
provider:
order: ["DeepInfra"]
allow_fallbacks: false
Add new environments and enhance tool context functionality - Introduced new environments: Terminal Test Environment and SWE Environment, each with default configurations for testing and software engineering tasks. - Added TerminalBench 2.0 evaluation environment with comprehensive setup for agentic LLMs, including task execution and verification. - Enhanced ToolContext with methods for uploading and downloading files, ensuring binary-safe operations. - Updated documentation across environments to reflect new features and usage instructions. - Refactored existing environment configurations for consistency and clarity.
2026-02-10 19:39:05 +00:00
openai:
base_url: "https://openrouter.ai/api/v1"
add top_p + user nudges for incorrect format
2026-04-06 10:05:25 -07:00
model_name: "nvidia/nemotron-3-super-120b-a12b"
Add new environments and enhance tool context functionality - Introduced new environments: Terminal Test Environment and SWE Environment, each with default configurations for testing and software engineering tasks. - Added TerminalBench 2.0 evaluation environment with comprehensive setup for agentic LLMs, including task execution and verification. - Enhanced ToolContext with methods for uploading and downloading files, ensuring binary-safe operations. - Updated documentation across environments to reflect new features and usage instructions. - Refactored existing environment configurations for consistency and clarity.
2026-02-10 19:39:05 +00:00
server_type: "openai"
health_check: false
wip: run tb2 and fix modal instantiation
2026-04-01 16:55:03 -07:00
timeout: 300 # 5 min per API call (default 1200s causes 20min stalls)
Add new environments and enhance tool context functionality - Introduced new environments: Terminal Test Environment and SWE Environment, each with default configurations for testing and software engineering tasks. - Added TerminalBench 2.0 evaluation environment with comprehensive setup for agentic LLMs, including task execution and verification. - Enhanced ToolContext with methods for uploading and downloading files, ensuring binary-safe operations. - Updated documentation across environments to reflect new features and usage instructions. - Refactored existing environment configurations for consistency and clarity.
2026-02-10 19:39:05 +00:00
# api_key loaded from OPENROUTER_API_KEY in .env
Reference in New Issue Copy Permalink