📜 Summary of the AI Agent Architecture and Evaluation Report

(≈ 4,000 words – written in Markdown for easy reading)

1. Introduction

The article under review details a next‑generation AI‑agent framework that blends large‑language‑model (LLM) reasoning, secure remote execution, and formal assertion handling into a single, modular system. The central innovation is the “LLM‑as‑Judge” concept: the LLM is treated as an adjudicative component that validates and interprets agent actions against a set of pre‑defined assertions and policy rules.

Alongside this, the report presents a security‑first evaluation—named AgentSec—which benchmarks the framework against realistic attack scenarios and operational use cases. The architecture relies heavily on YAML configuration, JSON/Markdown assertions, and a lightweight SSHExecutor that bridges LLM decision‑making to remote command execution.

This summary unpacks the main technical ideas, evaluates the experimental results, and discusses the implications for secure AI‑driven automation.

2. Background & Motivation

| Topic | Key Points | Why It Matters | |-------|------------|----------------| | AI‑Agent Automation | Modern workflows (e.g., DevOps, IT support, data engineering) increasingly rely on autonomous agents to carry out tasks. | Reduces manual toil but raises questions about safety, compliance, and trust. | | LLMs as Decision Makers | LLMs can generate complex code, compose documents, and plan actions. | However, they lack deterministic guarantees and can hallucinate. | | Formal Assertions & Policies | Assertion languages (JSON Schema, Markdown) enable formal verification of agent actions. | Provide a safety net that enforces business rules and regulatory constraints. | | SSH & Remote Execution | Secure shell remains the lingua franca for remote command execution on Unix‑like systems. | Integrating LLM output with SSH commands requires rigorous input validation to prevent injection attacks. | | Security Evaluation | AgentSec introduces a methodology for systematically assessing agent security. | Helps practitioners gauge real‑world risk before deployment. |

The article positions its architecture as a response to the gap between powerful LLM capabilities and the need for rigorous, auditable agent behavior.

3. Core Components of the Architecture

The system is a composition of seven tightly‑coupled modules, each responsible for a distinct phase in the agent lifecycle.

1. LLM‑as‑Judge – interprets and validates assertions.
2. AgentSec – evaluates the system under adversarial conditions.
3. YAML AgentAdapter – translates user‑defined workflows into internal messages.
4. Agent Observation – collects runtime telemetry.
5. JudgeRouter – dispatches tasks to specialized judges (LLM, rule‑engine, external API).
6. Authorization & SSHExecutor – controls access and executes commands on target hosts.
7. CommandPolicy – enforces fine‑grained constraints on remote actions.

These components cooperate through JSON‑over‑HTTP messages that carry command, metadata, and assertion payloads. The flow can be visualized as:

User  →  YAML AgentAdapter  →  JudgeRouter  →  LLM‑as‑Judge
          ↕               ↕             ↕
      Observation  ←  AgentSec  ←  Authorization/SSHExecutor

4. The “LLM‑as‑Judge” Concept

4.1 What Is a Judge?

In software verification, a judge is a component that examines claims against a set of rules. Traditionally, these are static checkers (e.g., type checkers, theorem provers). In this framework, the LLM itself serves as the judge:

• Input: A request (e.g., “Create a user on host X”) and a set of assertions (e.g., “User must belong to group admin”).
• Processing: The LLM generates an explanation and a decision (approve, reject, modify).
• Output: A JSON object indicating accept/reject, why, and any modifications.

4.2 Advantages

| Benefit | Explanation | |---------|-------------| | Adaptability | The LLM can handle evolving policies without code changes. | | Human‑readable Feedback | The justification is often in natural language, aiding auditors. | | Multi‑modal Reasoning | Can combine textual rules with structured JSON schemas. |

4.3 Challenges

• Hallucination: LLMs may fabricate policy violations.
• Trustworthiness: Need to validate LLM output against deterministic checks.
• Latency: Large models can add significant round‑trip time.

4.4 Mitigation Strategies

1. Dual‑Layer Validation – Use a deterministic rule‑engine as a fallback for high‑risk decisions.
2. Confidence Thresholds – Reject LLM decisions that fall below a specified probability.
3. Prompt Engineering – Embed policy snippets directly in the prompt to constrain the LLM’s output.

5. AgentSec: Security Evaluation Methodology

AgentSec is a structured test harness that probes the system from multiple attack vectors. The evaluation is broken into 10 key findings, each addressing a different threat category.

| Finding # | Threat Category | Summary | |-----------|-----------------|---------| | 1 | Command Injection | LLM-generated commands were sanitized, but complex shell syntax could bypass basic regex filters. | | 2 | Privilege Escalation | SSHExecutor used sudo only when explicitly allowed; no accidental elevation observed. | | 3 | Data Leakage | JSON telemetry was encrypted in transit; however, local logs still contained raw command outputs. | | 4 | Model Misuse | A malicious user could feed the LLM fabricated assertions to bypass policy checks. | | 5 | Denial‑of‑Service | Repeated high‑volume prompts caused LLM throttling; no crash, but latency increased. | | 6 | Credential Theft | SSH keys were stored in HSM; no compromise detected. | | 7 | Policy Drift | Automated comparison of current policy schemas with legacy policies detected 3 drift incidents. | | 8 | Unintended Automation | Agents sometimes executed optional actions (e.g., backup) without explicit user consent. | | 9 | Logging Misconfiguration | Audit logs were not rotated automatically, leading to potential information retention beyond compliance windows. | | 10 | API Abuse | External API calls were rate‑limited, preventing brute‑force exploitation. |

5.1 Experimental Setup

• Hardware: 4 × RTX 3090 GPUs for LLM inference; 2 × Dell PowerEdge R740 for orchestration.
• Software Stack: Python 3.10, FastAPI, OpenAI API (gpt‑4‑32k), Paramiko for SSH.
• Attack Simulations: Crafted via a combination of rule‑based fuzzers and human‑crafted exploit vectors.

5.2 Key Takeaways

• The primary risk remains in incomplete sanitization of shell commands.
• Authorization layers (policy, role‑based access) effectively mitigate privilege escalation.
• Logging and audit trails need better lifecycle management to meet regulatory standards.

6. YAML AgentAdapter – Declarative Workflow Engine

The YAML AgentAdapter is a lightweight parser that turns human‑friendly YAML definitions into structured commands the system can execute.

6.1 YAML Syntax

steps:
  - name: gather_system_info
    type: agent_query
    query: |
      import platform, socket
      return {
        "os": platform.system(),
        "hostname": socket.gethostname()
      }

  - name: create_user
    type: agent_action
    action: create_user
    params:
      username: "{{steps.gather_system_info.hostname}}_admin"
      groups: ["admin", "sudo"]

  - name: verify_user
    type: agent_assert
    assertion:
      kind: json
      schema: |
        {
          "type": "object",
          "properties": {
            "username": {"type": "string"},
            "groups": {"type": "array"}
          },
          "required": ["username", "groups"]
        }

6.2 How It Works

1. Parsing – The adapter uses pyyaml to read the file.
2. Templating – Jinja2-like interpolation allows step results to feed subsequent steps.
3. Message Generation – Each step is transformed into a JSON payload with metadata (step ID, type, dependencies).
4. Dispatch – The payload is sent to JudgeRouter for evaluation.

6.3 Benefits

• Human‑readable: Teams can design workflows without writing code.
• Version‑controlled: YAML files are Git‑friendly.
• Extensible: Custom step types can be added via plugin modules.

6.4 Limitations

• Schema Drift: If underlying data structures change, YAML steps may break.
• Error Propagation: Failure in an early step can cascade; the adapter currently retries only for idempotent actions.

7. Agent Observation – Runtime Telemetry

The Agent Observation module collects in‑flight data and exposes it via an API and a Prometheus exporter.

7.1 Telemetry Fields

| Field | Type | Description | |-------|------|-------------| | agent_id | string | Unique identifier for the agent instance. | | step_id | string | Current step being executed. | | status | enum | queued, running, success, failure. | | latency_ms | integer | Time taken for the step. | | assertion_result | boolean | Pass/fail of the latest assertion. | | sensitive_payload | masked | Any data that matched a sensitive regex. |

7.2 Exporter Configuration

prometheus:
  enabled: true
  port: 9090
  scrape_interval: 15s

The module exposes metrics like:

agent_step_latency_seconds{agent_id="a1",step_id="create_user"} 0.42
agent_assertion_pass_total{agent_id="a1"} 42

7.3 Privacy Considerations

• Masking: All telemetry is passed through a masking filter that replaces values matching patterns like (?i)(password|token) with ***.
• Encryption: The telemetry channel uses TLS 1.3 with mutual authentication.

8. JudgeRouter – Dispatcher & Orchestrator

The JudgeRouter is the linchpin that selects the appropriate judge for each request.

8.1 Decision Logic

| Criterion | Judge | |-----------|-------| | Type of request (agent_action vs. agent_assert) | LLM‑as‑Judge | | Policy severity (high vs. low) | Deterministic rule‑engine | | External API call | API Judge (proxy to OpenAI or internal LLM) | | Resource constraints | Queue manager (to prevent overload) |

8.2 Routing Table

{
  "agent_action": "llm_judge",
  "agent_assert": "llm_judge",
  "external_api": "api_judge",
  "high_severity": "rule_engine"
}

8.3 Failure Recovery

• Timeouts: If a judge exceeds timeout_ms, the request is marked failure and logged.
• Fallback: On judge failure, the Router retries with a simpler judge (e.g., rule engine).

9. Authorization & SSHExecutor – Secure Remote Actions

9.1 Authorization Layer

Authorization is role‑based and enforced in two stages:

1. Policy Check – A JSON Policy document specifies allowed actions per role.
2. Token Validation – Each request carries a JWT signed by a central Identity Provider (IdP).

Example Policy

{
  "role": "operator",
  "allow": [
    {"action": "create_user", "resource": "host/*"},
    {"action": "execute_script", "resource": "host/*", "script": ".*\\.sh"}
  ],
  "deny": [
    {"action": "delete_user", "resource": "host/*"}
  ]
}

The authorization engine returns allow/deny along with a confidence score.

9.2 SSHExecutor Implementation

| Component | Description | |-----------|-------------| | SSHClient | Uses Paramiko for SSH connections. | | CommandSanitizer | Performs whitelisting of shell commands based on the policy. | | ExecutionContext | Keeps track of cwd, user, and environment variables. | | ResultParser | Converts stdout/stderr into structured JSON. |

class SSHExecutor:
    def execute(self, host, command, env=None):
        self._sanitize(command)
        with paramiko.SSHClient() as client:
            client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
            client.connect(host, username=env['user'])
            stdin, stdout, stderr = client.exec_command(command)
            return {
                "stdout": stdout.read().decode(),
                "stderr": stderr.read().decode(),
                "exit_code": stdout.channel.recv_exit_status()
            }

9.3 Security Controls

• Strict Path Binding – The executor resolves paths relative to a sandbox directory.
• Rate Limiting – Each host is limited to 10 req/min to mitigate brute‑force.
• Audit Logging – Every command executed is logged with a hash of the input for replayability.

10. CommandPolicy – Fine‑grained Control

CommandPolicy is a JSON‑based language that describes what commands can run, when they can run, and under what circumstances they may be modified.

10.1 Policy Syntax

{
  "policy_name": "user_management",
  "rules": [
    {
      "action": "create_user",
      "condition": "role == 'operator'",
      "script_pattern": "^useradd",
      "max_retries": 3,
      "notify": "email"
    },
    {
      "action": "delete_user",
      "condition": "role == 'admin'",
      "script_pattern": "^userdel",
      "max_retries": 1,
      "notify": "slack"
    }
  ]
}

10.2 Enforcement Flow

1. Policy Load – At startup, policies are parsed and cached.
2. Request Validation – Each command is matched against policy rules.
3. Execution Decision – If matched, the command proceeds; otherwise it is blocked.
4. Post‑Execution Hook – If notify is defined, a webhook is triggered with the execution outcome.

10.3 Dynamic Policy Updates

Policies can be updated via a REST endpoint that supports CRDT (Conflict‑free Replicated Data Type) updates, ensuring eventual consistency across distributed orchestrators.

11. 10 Findings (Deduped) – In‑Depth Analysis

Below we expand on the 10 findings from AgentSec, adding context, mitigation steps, and open questions.

| # | Finding | Context | Mitigation | Open Question | |---|---------|---------|------------|---------------| | 1 | Command Injection | The LLM produced rm -rf / when given a malformed assertion. | Sanitization using a whitelist of safe shell constructs. | Could a more sophisticated LLM prompt prevent injection entirely? | | 2 | Privilege Escalation | A sudo escalation attempt was caught by the policy. | Least privilege policy, explicit sudo approvals. | What about privilege creep in long‑running sessions? | | 3 | Data Leakage | Raw stdout from commands was stored in plaintext logs. | Encrypt logs at rest; enforce log masking. | Is there a need for data retention policies? | | 4 | Model Misuse | Attackers fed fabricated assertions to bypass policy. | Model watermarking and confidence thresholds. | How to detect fabricated assertions automatically? | | 5 | Denial‑of‑Service | Overloading LLM with prompts caused throttling. | Rate limiting on the API gateway. | Is a local inference engine feasible? | | 6 | Credential Theft | HSM protected SSH keys, but the key retrieval process was unencrypted. | Use Hardware‑Backed Security and end‑to‑end encryption. | Should key usage be token‑based instead of static? | | 7 | Policy Drift | New policy added by an admin unintentionally conflicted with existing rules. | Policy conflict detection tool. | How to automatically resolve conflicts? | | 8 | Unintended Automation | Agents executed optional backup commands without user consent. | Require explicit confirmation tokens. | Can we make optional actions part of the assertion? | | 9 | Logging Misconfiguration | Logs were not rotated, violating GDPR. | Implement log rotation and archival policies. | How to enforce rotation across distributed nodes? | |10 | API Abuse | External API calls were limited to 10 req/min. | Rate limiting, token revocation on abuse. | Is dynamic throttling needed per user? |

12. Implications & Future Directions

12.1 Operational Impact

| Impact | Description | |--------|-------------| | Automation Speed | LLM‑based decision making reduces manual intervention by up to 70 % in typical DevOps pipelines. | | Compliance | Formal assertions and audit logs provide evidence for regulatory compliance (SOC 2, ISO 27001). | | Security Posture | Granular policy enforcement lowers attack surface; however, the LLM layer introduces new risk vectors that must be monitored. |

12.2 Research Opportunities

1. LLM Verification – Formal methods to prove that an LLM’s output satisfies a given policy.
2. Policy Languages – Extending JSON Schema to encode temporal constraints and probabilistic guarantees.
3. Hybrid Judges – Combining symbolic reasoning with LLM inference to create a probabilistic verifier.
4. Federated Deployment – Scaling the architecture across multi‑cloud environments with consistent policy enforcement.
5. Explainability Enhancements – Augmenting LLM justifications with trace diagrams that link assertions to source code changes.

12.3 Open Source Ecosystem

The authors have released AgentSec and related components under the MIT license, encouraging community contributions. The repo includes:

• agent_adapter.yaml – sample workflow definitions.
• policy_manager.py – policy CRUD API.
• ssh_executor.py – core executor with sandboxing.
• test_agentsec.py – full suite of attack simulations.

Community feedback has already led to a plugin architecture for custom judges (e.g., a formal theorem prover plug‑in).

13. Conclusion

The article presents a holistic, security‑first approach to building AI‑powered autonomous agents. By treating the LLM as an adjudicative judge, the system gains flexibility and human‑readable explanations. Yet, the evaluation reveals that traditional security controls (policy enforcement, sanitization, audit logging) remain essential.

Key takeaways:

• LLM‑as‑Judge is a promising paradigm but must be paired with deterministic checks for high‑stakes operations.
• AgentSec demonstrates that systematic, adversarial testing can surface subtle vulnerabilities before deployment.
• YAML AgentAdapter and CommandPolicy provide a declarative, composable workflow layer that eases operational adoption.

For practitioners, the framework offers a blueprint to operationalize AI automation while maintaining compliance and security. For researchers, it opens new questions about verifiable machine reasoning and policy-driven AI control.

Future work should focus on formal verification of LLM outputs, dynamic policy conflict resolution, and efficient on‑prem LLM inference to reduce latency and dependency on external APIs.

Prepared by: [Your Name]
Date: 25 May 2026