🔄 Topic

Building and benchmarking the local LLM coding stack — a test harness, routing rules, and fixing what code review found.

🎯 Goal

Know, with evidence, which local model to trust for which coding task — and clean up the reliability bugs review surfaced.

🛠 What I Did

I documented and benchmarked the local coding stack: a harness for running models against real tasks, routing rules for which model handles what, and workflow docs so the setup survives model swaps. Then I ran a review pass over my own orchestration code and fixed what it found: a routing fallback that could pick the wrong provider, unsafe provider defaults, a cancellation path that could race, and orphaned processes left behind after failures. I also bootstrapped a proper operating layer for the workstation's coding agents.

Main areas covered:

benchmark harness for local coding models
routing rules based on measured capability
workflow documentation for model swaps
routing fallback and provider default fixes
cancellation safety and orphan process cleanup
operating layer bootstrap for coding agents

🔗 Key Cybersecurity Connections

Reliability bugs are security bugs waiting for context: an unsafe fallback is an unintended trust decision, a cancellation race is a state-integrity flaw, and orphaned processes are unaccounted-for execution. Reviewing my own automation like hostile code is defensive practice.

🔍 Investigation Questions

Which model does each task type actually route to, and why?
What happens when the preferred provider is unavailable?
Can a cancelled task leave work half-applied?
Are there processes running that no active task owns?
Do the benchmark results justify the routing table?

🚨 Detection Opportunities

Potential monitoring ideas:

fallback routing activating unexpectedly
orphaned agent processes accumulating
tasks cancelled but still producing output
provider configuration drift from defaults
benchmark regressions after model updates

Example:

project=local-coding-stack
change_type=orchestration_reliability_fix
risk_area=unintended_execution_paths
triage=audit_fallbacks_cancellation_and_orphans

🧭 MITRE ATT&CK Techniques

Possible mappings depending on confirmed behavior:

T1059 — Command and Scripting Interpreter
T1057 — Process Discovery
T1565 — Data Manipulation

🗺 Visual Investigation Diagram

Benchmark harness
    ↓

Measured capability ↓ Routing rules ↓ Review findings ↓ Fixed, documented stack

⚠ Challenges

The challenge was reviewing my own orchestration code honestly. The bugs were all in the unhappy paths — fallback, cancellation, failure cleanup — exactly the paths I had tested least.

📚 What I Learned

I learned that unhappy paths are where automation rots. Everything works in the demo; the fallback, the cancel, and the crash are where wrong trust decisions and stray processes live.

➡ Next Steps

Re-run benchmarks after every model change
Add checks for orphaned processes to routine maintenance
Keep routing decisions written down with their evidence
Extend the harness with security-flavored coding tasks

🧠 Reflection

This was useful because it applied the audit mindset to my own tooling: measure first, route on evidence, and treat every unhappy path as a finding.

🧩 Lessons Learned

What worked

Benchmarking before trusting, and reviewing the orchestrator like third-party code.

What broke

Fallback routing, provider defaults, cancellation, and process cleanup.

Why it broke

I had only exercised the happy path during development.

Fix / takeaway

Test the failure paths deliberately; that is where reliability and security overlap.

📈 Skill Progression Context

This supports my cybersecurity progression because evidence-based trust decisions and unhappy-path analysis are the same skills used in code review and incident investigation.

😄 TL;DR

Benchmarked the stack; the bugs were hiding in the failure paths.