🐦‍⬛ Skwaq

What Is Skwaq?

An agentic vulnerability analyzer that uses a team of AI agents to investigate source code and binaries for security vulnerabilities. Unlike traditional static analysis tools that rely on fixed rules, skwaq's agents reason about code like experienced security researchers — tracing data flows, mapping attack surfaces, and debating exploitability.

What makes it unique: skwaq improves itself. A built-in benchmark harness (Skwaq Gym) measures detection accuracy against industry benchmarks, and a self-improvement loop uses AI agents to analyze their own failures and propose better investigation strategies.

Started with a brief specification for a "vulnerability investigation copilot":

• Analyze binaries and source code using a code property graph
• AI agents reason about the results
• No training data, no pre-built rules, no benchmark infrastructure

• No labeled datasets — agents must reason, not memorize
• No manually tuned rules — improvement comes from the loop
• Benchmark infrastructure built alongside the tool, not before

• Built the Skwaq Gym benchmark harness — started with a small fixture suite at F1=50%
• Wired up 5-layer detection: patterns, dataflow, context validation, LLM agents, synthesis
• Connected four industry benchmarks:
  • NIST Juliet — 54K synthetic C/C++ cases across 116 CWEs
  • OWASP Benchmark — 2,740 Java web-app vulnerability cases
  • DARPA CGC — real challenge binaries (patched/unpatched)
  • Meta CyberSecEval — real-world vulnerability patterns
• Development and multi-agent analysis pipelines orchestrated by amplihack-recipe-runner

Built a code property graph in LadybugDB — skwaq's own analysis code extracts and stores:

Agents query this graph with specialized tools — enabling analysis that static patterns alone cannot achieve.

Started with a small set of agents. The self-improvement loop was given instructions to experiment with new agentic workflows and agent types — and evolved the pipeline organically as it learned what worked:

• Phase 1: Basic attack-surface mapping + vulnerability hunting
• Phase 2: Specialized language-specific agents added (Java, Python)
• Phase 3: Critic agent for false-positive reduction
• Phase 4: Offense/defense debate for high-stakes findings

Integrated agent-kgpacks — portable knowledge graph packages that encode:

• Vulnerability patterns per CWE class
• CWE taxonomies and family relationships
• Investigation strategies learned from prior cycles

• Knowledge packs are versioned — improvements accumulate without overwriting
• Different CWE families can load specialized expertise independently

Added durable agent memory (adapted from amplihack-memory-lib's cognitive memory model) so investigation insights persist across cycles:

• Which detection strategies worked for which CWE families
• Which proposals were rejected by the overfitting-reviewer and why
• CWE-specific instructions that improved recall in prior cycles

• Memory is durable — survives restarts and new deployments
• Memory is queryable — agents retrieve relevant past experience before each investigation
• Rejection reasons stored alongside accepted improvements — both inform future proposals

Built the gym improve command — agents started teaching themselves:

• failure-analyst investigates each false negative, diagnoses why detection failed
• Proposes improvements to agent prompts, taint rules, CWE mappings, or patterns
• overfitting-reviewer validates every proposal before it is applied
• Accepted improvements are written back to agent prompts, taint engine, and scoring
• Rejected proposals are stored as lessons in durable memory

Four proposal types (in preference order): AGENT_PROMPT → TAINT_RULE → CWE_MAPPING → NEW_PATTERN

It became critical to ensure improvement loops don't "build to the benchmark" — overfitting to specific test cases rather than improving general detection.

Response: built a specialized overfitting-reviewer agent with explicit rejection criteria:

• Duplicate boilerplate proposals from different test cases
• Benchmark-specific naming conventions
• Narrow detection logic for broad CWE classes
• Generic graph traversal where type/semantic analysis is needed
• Speculative detection based on proximity rather than evidence

Architecture: Five-Layer Detection Pipeline

Each vulnerability investigation passes through five layers, from fast pattern matching to deep semantic reasoning:

Architecture: Code Property Graph

All analysis operates on a code property graph stored in LadybugDB — functions, call edges, taint flows, data sources, data sinks, and symbols. Agents query this graph with specialized tools.

Multi-Agent Analysis

All 18 agents and the pipelines they participate in:

Self-Improvement Loop

Results: Benchmark Suites

Results: Current Baselines — Pattern+Dataflow Mode

Results: Per-CWE Improvements (Self-Improvement Loop)

Results: Industry Comparison

Note: Benchmarks differ across tools — direct comparison is approximate. See IRIS paper for methodology.

Improvement History: Key Milestones

Full benchmark progress and history →

Benchmark Ground-Truth & False-Positive Problem

Usage

# Full agentic analysis
skwaq analyze /path/to/repo

# Quick pattern-only scan
skwaq analyze path/to/src.c --quick

# Analyze compiled binary
skwaq analyze path/to/binary --binary

# Stripped binary with renaming
skwaq analyze path/to/binary \
  --binary --decompile-rename

# Agentic eval
skwaq gym eval \
  --suites juliet,owasp,cyberseceval

# Pattern-only eval (fast)
skwaq gym eval --suites juliet \
  --quick --max-cases 200

# Run improvement cycle
skwaq gym improve juliet \
  --max-cases 30 --max-improvements 5

# Target specific CWE
skwaq gym improve juliet \
  --cwe CWE-78 --max-cases 20

Supports Azure AI Foundry (GPT-5.4), Anthropic Claude (via Azure MaaS), and GitHub Copilot (Claude Opus 4.6) backends — configured in skwaq.toml.