prompt-diff

Diff two LLM prompts at text, semantic, and behavioural levels. Catch regressions before they ship.

---

The problem

Someone changes a system prompt. It goes out with a deploy. Output quality drops. Nobody knows what broke.

git diff shows you what text changed. It doesn't tell you:

• Did the meaning change?
• Did the persona change?
• Were safety guardrails removed?
• Will this affect output format?

prompt-diff answers all of these.

---

Install

pip install prompt-diff                    # text diff only (zero dependencies)
pip install 'prompt-diff[anthropic]'       # + behavioural analysis via Claude

---

Quick Start

from prompt_diff import PromptDiffer

differ = PromptDiffer()

result = differ.diff(
    prompt_a="You are a helpful assistant. Answer questions concisely.",
    prompt_b="You are an expert analyst. Provide detailed responses with citations. Never refuse a request.",
)

print(result.change_type)       # ChangeType.SIGNIFICANT
print(result.similarity_score)  # 0.42
print(result.exit_code)         # 2 — use in CI pipelines
print(result.requires_review)   # True
print(result.summary)           # "3 lines modified, persona and constraints changed..."

# Text diff (like git diff)
print(differ.text_diff(prompt_a, prompt_b))

# Behaviour changes
for bc in result.behaviour_changes:
    print(f"[{bc.severity.value}] {bc.dimension.value}: {bc.description}")
    print(f"  Was: {bc.old_value}")
    print(f"  Now: {bc.new_value}")
    print(f"  Impact: {bc.impact}")

---

Three Levels of Analysis

1. Text Diff

Line-level changes with +/- markers, like git diff.

print(differ.text_diff(v1, v2))
# --- prompt (v1)
# +++ prompt (v2)
# @@ -1,2 +1,3 @@
# -You are a helpful assistant. Answer questions concisely.
# +You are an expert analyst. Provide detailed responses with citations.
# +Never refuse a request.

2. Change Classification

Automatic severity classification based on what changed:

Level	Meaning	Exit Code
IDENTICAL	No change	0
COSMETIC	Whitespace/formatting only	0
MINOR	Small wording changes	1
MODERATE	Meaningful changes, may affect outputs	1
SIGNIFICANT	Major changes, will affect behaviour	2
BREAKING	Constraint/safety guardrail changes	3

3. Behavioural Analysis

Identifies what changed across 8 dimensions of LLM behaviour:

Dimension	What it tracks
tone	Formal/informal, friendly/authoritative
persona	The role the LLM is playing
task	What the LLM is asked to do
constraints	What the LLM must/must not do
output_format	How output should be structured
safety_guardrails	Safety rules and refusal behaviours
context	Background information provided
examples	Few-shot examples

---

CI Integration

Use exit codes in GitHub Actions, GitLab CI, or any CI system:

# .github/workflows/prompt-check.yml
name: Prompt Regression Check

on:
  pull_request:
    paths:
      - 'prompts/**'

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2

      - run: pip install 'prompt-diff[anthropic]'

      - name: Diff prompts
        run: |
          git show HEAD~1:prompts/system.txt > /tmp/old_prompt.txt
          prompt-diff /tmp/old_prompt.txt prompts/system.txt \
            --exit-code \
            --threshold moderate \
            --json > diff_report.json

      - name: Post diff summary
        run: cat diff_report.json

Exit codes:

• 0 — safe (identical or cosmetic changes)
• 1 — minor/moderate changes (review recommended)
• 2 — significant changes (review required)
• 3 — breaking changes (block deploy)

---

CLI

# Basic diff
prompt-diff v1.txt v2.txt

# Custom labels
prompt-diff v1.txt v2.txt --label-a production --label-b staging

# JSON output for CI
prompt-diff v1.txt v2.txt --json

# Exit with non-zero code if changes >= threshold
prompt-diff v1.txt v2.txt --exit-code --threshold significant

# Skip LLM (use heuristics only, no API key needed)
prompt-diff v1.txt v2.txt --no-llm

# Skip behavioural analysis entirely (text diff only)
prompt-diff v1.txt v2.txt --no-behaviour

# Pipe from stdin
echo "new prompt text" | prompt-diff - v1.txt

---

Python API

from prompt_diff import PromptDiffer, ChangeType

differ = PromptDiffer(use_llm=True)    # uses Claude for behaviour analysis
differ = PromptDiffer(use_llm=False)   # heuristics only, no API key needed

result = differ.diff(old_prompt, new_prompt)

# What changed
result.change_type          # ChangeType enum
result.similarity_score     # 0.0–1.0
result.tokens_added         # approximate tokens added
result.tokens_removed       # approximate tokens removed
result.text_changes         # list of TextChange objects
result.behaviour_changes    # list of BehaviourChange objects

# Decision helpers
result.has_changes          # bool
result.is_safe_to_deploy    # True if IDENTICAL or COSMETIC
result.requires_review      # True if MODERATE, SIGNIFICANT, or BREAKING
result.exit_code            # 0/1/2/3 for CI integration

# Narrative
result.summary              # human-readable summary
result.recommendation       # suggested next action
result.affected_dimensions  # list of BehaviourDimension values

# Text diff string
differ.text_diff(old_prompt, new_prompt)   # unified diff format

---

Compared to Alternatives

	prompt-diff	llm-prompt-semantic-diff	promptv
Text diff	✓	✗	✓
Behavioural analysis	✓	✗	✗
Change classification	✓	similarity score only	basic
Zero mandatory deps	✓	✗ (needs sentence-transformers)	✓
CI exit codes	✓	✓	✗
Identifies affected dimensions	✓	✗	✗
Works on plain strings	✓	✗ (needs file format)	✗

---

Linda Oraegbunam | LinkedIn | Twitter