Trainable modular agent framework (planner, executor, verifier, generator) with web search, Wikipedia and Python tools, plus Flow-GRPO RL training.
github.com/lupantech/agentflow · 2026-10-04 · b940064
Defense-in-depth score
2.1 / 10
As shipped, AgentFlow runs model-written Python directly inside its own process for every tool call, with no sandbox, no approval step and no argument validation. Web pages and search results are fed into the prompt that writes that code, so a malicious page can steer it to read the provider API keys from the environment and send them anywhere, or damage the host. Only step and time caps exist, and timed-out code keeps running.
AgentFlow keeps agent memory to a single run, leaves its response cache off and ends the solve loop after 10 steps or 300 seconds. Beyond that the defaults are weak: model-written Python runs inside the agent's own process with every provider key, with no sandbox or approval step to turn on, so the safest setup narrows the tools and builds the boundaries around the process yourself.
| Setup | How | What it gives you |
|---|---|---|
| Dedicated, low-limit keys | Put only the provider keys this task needs in the .env file, each a dedicated key with a spending limit set at the provider | Whatever runs in the process reaches only the services and spend you chose C1 |
| Only the tools needed | Pass enabled_tools to construct_solver naming just the tools the task uses, leaving out the Python and web tools where you can | Web content and the Python tool stay out of runs that have no use for them C3 |
| The response cache off | Keep the default use_cache=False on the LLM engines | Each run starts from fresh model output, with nothing replayed from earlier sessions C6 |
| Step and time limits | Keep the default max_steps=10 and max_time=300, or lower them for short tasks | Every run stops planning new steps after a fixed count and time C10 |
| Its own container or VM | Run the agent in a container or VM holding no other credentials or files, with network access limited to the hosts the task needs | An outer boundary around model-written code and a narrow path for data to leave C4 |
| Quiet console output | Pass verbose=False to construct_solver where output lands in shared logs | Prompts, commands and tool results stay out of shared console logs C8 |
| A saved run record | Write the dict that solve returns to storage outside the container after each run | A record of every command and tool result that outlives the process C9 |
0.05 / 1.00
AgentFlow has no agent identity or authorization layer. Importing the engines and tools loads a .env file into the process environment, and every engine and tool builds its own client from those environment keys. Because model-written Python runs inside the same process, any hijacked step can use every provider key and the OS user's full authority.
0.00 / 1.00
There is no approval step anywhere. The only check before running a tool is that the model chose a registered tool name; the command itself is model-written Python that runs straight away. Consequential actions, including arbitrary code on the host and outbound requests, happen with no human involved.
0.00 / 1.00
Tool 'commands' are not structured arguments: the executor runs whatever Python the model writes, as long as it ends with an execution = tool.execute(...) line, so any code can be prefixed. The web tools fetch any URL with no host allowlist or internal-address block, and the Python tool's only filter strips exit/quit calls. The default constructor enables every bundled tool, including code execution and web fetch.
0.00 / 1.00
Model-generated code runs inside the agent's own Python process twice over: the executor exec()s every tool command with the executor module's globals, and the Python coder tool exec()s generated code (once with no namespace at all). There is no container, subprocess, or even an import filter, and the process holds every API key and full network access.
0.00 / 1.00
Web pages, Wikipedia articles, and search results are stored as step results and pasted straight into the planner's and executor's prompts with the same standing as the user's question. The executor then writes Python from that context and runs it. A page that hijacks the model can therefore exfiltrate keys through an arbitrary request and run destructive code, unattended.
0.30 / 1.00
Agent memory lives only for one solve() call, and there are no auto-loaded instruction files. The one persistent store is an opt-in disk cache of LLM responses (off by default in the engine factory) that, when enabled, replays stored responses, including generated tool commands, for any identical prompt in later sessions without validation. The .env file is found relative to the package source, not the working directory (python-dotenv behaviour, inferred).
N/A · full credit 1.00 / 1.00
The inference agent loads no third-party code at runtime: tools are imported only from the package's own tools directory, there is no MCP or plugin loader, and LLM engines are API clients. Training code (verl) defaults trust_remote_code to False. This is absence of the surface, not a control.
0.05 / 1.00
Secrets are plain environment variables loaded from .env at import, with no masking anywhere. Verbose mode, on by default, prints every prompt, command and tool result to stdout. Model-written code runs in the same process and can read every provider key from os.environ.
0.28 / 1.00
The solver keeps a per-run dictionary of prompts, generated commands and tool results and returns it to the caller, and prints each step to stdout. Nothing is written to a durable or tamper-resistant log, there is no actor attribution, and the record lives in the same process that runs model code.
0.40 / 1.00
The solve loop stops after 10 steps or 300 seconds by default, checked between steps, and each tool block waits at most 120 seconds. But timeouts only stop waiting: the daemon thread running model code keeps going, and the Python tool's 10-second timer raises in the wrong thread so it never interrupts anything. There is no token or cost budget, and in-process model code could alter the limits.
SafetyCard