> ## Documentation Index > Fetch the complete documentation index at: https://meganharrisonconsulting.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # rag-strategy-council > Run a structured multi-agent strategy debate before implementing recurring or high-risk Alleato RAG, AI assistant, packet-first project intelligence, source lookup, AI SDK provider/tool-calling, or retrieval architecture changes. Use this when the user asks for consensus, expert debate, "best strategy", "stop fixing RAG forever", "RAG council", "agents debate", or when a RAG fix has failed more than once. This page is generated from the actual `SKILL.md` file on disk. ## Metadata | Field | Value | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Source group | Workspace Skills | | Source root | `/Users/meganharrison/Documents/alleato-pm/.codex/skills` | | Relative skill path | `rag-strategy-council/SKILL.md` | | Actual file | `/Users/meganharrison/Documents/alleato-pm/.codex/skills/rag-strategy-council/SKILL.md` | | Description | Run a structured multi-agent strategy debate before implementing recurring or high-risk Alleato RAG, AI assistant, packet-first project intelligence, source lookup, AI SDK provider/tool-calling, or retrieval architecture changes. Use this when the user asks for consensus, expert debate, "best strategy", "stop fixing RAG forever", "RAG council", "agents debate", or when a RAG fix has failed more than once. | | Docs overview | [Workspace Skills](/skills/workspace-skills) | ## Why this exists Use this page when you want to review the real instructions behind a skill without leaving the docs site. The content below is copied from the actual `SKILL.md` file, not rewritten by hand. ## Raw SKILL.md ````md theme={null} --- name: rag-strategy-council description: Run a structured multi-agent strategy debate before implementing recurring or high-risk Alleato RAG, AI assistant, packet-first project intelligence, source lookup, AI SDK provider/tool-calling, or retrieval architecture changes. Use this when the user asks for consensus, expert debate, "best strategy", "stop fixing RAG forever", "RAG council", "agents debate", or when a RAG fix has failed more than once. argument-hint: --- # RAG Strategy Council Use this skill when the right answer is not "patch the first suspicious file." The goal is to force expert disagreement, evidence collection, and a consensus implementation sequence before changing the Alleato AI assistant. This skill is repo-specific. It must be used with `alleato-rag-implementation` for Alleato retrieval/project-intelligence behavior. Use the `ai-sdk` skill whenever the council touches AI SDK, AI Gateway, model IDs, provider routing, streaming, tool calling, structured output, or embeddings. ## What This Produces Create a council report under: ```text docs/ai-plan/councils/YYYY-MM-DD-rag-strategy-council-.md ``` Use `templates/council-report.md` as the starting structure. The report is the deliverable unless the user explicitly asks to proceed into implementation. Do not let the debate disappear into chat. ## When To Run A Council Run this before implementation when any of these are true: - A RAG/assistant fix has failed before or the user says prior fixes did not hold. - The issue could sit in multiple layers: intent routing, provider/tool behavior, packet freshness, source lookup, ingestion, attribution, embeddings, or prompt synthesis. - The user asks for consensus, top expertise, a strategy debate, or a durable plan. - A proposed fix would change the assistant architecture, provider path, packet-first behavior, eval gates, or ingestion/vectorization pipeline. - You are tempted to blame embeddings first. Do not run a council for a narrow, evidenced bug with an obvious local fix. Use the normal `alleato-rag-implementation` checklist instead. ## Council Roles Use five roles. If sub-agents are available and the user asked for debate/consensus, spawn them in parallel with bounded read-only scopes. If sub-agents are unavailable, simulate the same roles in the main thread but clearly mark that it was an inline council. ### 1. Repo Architect Defends the current Alleato architecture. Focus: - packet-first project intelligence - current chat route and intent router - existing packet, source lookup, tool, and eval surfaces - what should be reused instead of rebuilt Required evidence: - relevant files - existing verifier commands - current guardrails that already exist ### 2. RAG Architect Challenges retrieval quality and source-grounding. Focus: - chunking and metadata quality - hybrid retrieval, reranking, and filtering - source-specific lookup behavior - evaluation coverage - attribution and project linkage Required evidence: - likely retrieval failure layer - metrics or verifier gaps - proposed retrieval changes only after routing/provider/packet causes are ruled out ### 3. AI SDK And Provider Specialist Checks whether the model/provider path can actually do the expected work. Focus: - AI SDK docs from `frontend/node_modules/ai/docs/` or provider package docs - AI Gateway versus direct provider behavior - tool calling availability - `finishReason`, empty output, streaming behavior, and tool traces - model IDs and provider matrix checks Required evidence: - exact docs/source references used - provider/tool matrix command when relevant - whether the failure is caused before retrieval is even reached ### 4. Failure-Mode Reviewer Argues how the proposed fix could still fail. Focus: - silent fallbacks - stale packet paths - generic answer fallthrough - missing metadata - weak evals that pass while behavior is still bad - operational degradation in ingestion/vectorization Required evidence: - detection gap - fail-loud requirement - guardrail or verifier that prevents recurrence ### 5. Product Advisor Judges whether the strategy creates the right user experience. Focus: - project advisor behavior, not chunk dumps - exact evidence when the user asks for source lookup - confidence and freshness language - whether the answer helps Megan/Brandon make decisions - whether output belongs in chat, a durable packet, or a normal app page Required evidence: - representative user prompts - success/failure examples - acceptance criteria stated in plain English ## Operating Protocol ### 1. Frame The Question Write the exact council question in one sentence. Examples: - "Why does source lookup keep returning generic project answers instead of exact meeting/email evidence?" - "What is the lowest-risk strategy for making packet-first project intelligence reliable without replacing the assistant stack?" - "Should we change retrieval, provider/tool routing, or packet compilation first?" ### 2. Build An Evidence Packet First Before opinions, gather the smallest relevant evidence packet: - failing prompt(s), route, project, and expected behavior - relevant files from the Alleato RAG skill - recent verifier results, if already available - relevant live DB counts or health checks when the failure may be data-side - AI SDK docs/source excerpts when provider/tool behavior is in scope Do not run long checks in the main thread. Delegate expensive verification to a cheaper sub-agent and ask for a compact pass/fail report. ### 3. Dispatch Role Briefs Each role must answer this structure: ```text Role: Position: Evidence: Risk in the other strategies: Minimum viable next step: Guardrail required: Confidence: ``` Keep each role bounded. The point is disagreement and evidence, not five essays. ### 4. Force Disagreement After initial positions, identify: - where roles agree - where roles disagree - which disagreement can be resolved by a targeted verifier or DB query - which disagreement is a product judgment Do not average the opinions. Prefer the strategy with the strongest evidence, smallest blast radius, and best recurrence guardrail. ### 5. Synthesize The Decision The main agent writes the final decision, including: - confirmed root-cause layer, or the first layer to disprove - implementation sequence - verification gates - fail-loud behavior - what not to change yet - follow-up issues if the work spans multiple owners ### 6. Stop Before Code Unless Asked If the user asked for a strategy, stop at the report. If the user asked to implement after the council, implement only the first agreed slice and add the guardrail before moving to the next layer. ## Required Output Sections Every council report must include: 1. Executive decision 2. Council question 3. Evidence packet 4. Role positions 5. Disagreements and resolution 6. Consensus implementation sequence 7. Verification gates 8. Fail-loud and recurrence guardrails 9. Open questions 10. Recommended next step ## Default Verification Menu Start narrow. Do not run the full suite unless the council decides it is necessary. ```bash npm run rag:verify:chat-architecture npm run rag:verify:source-specific node scripts/verify/verify_ai_tool_calling_provider_matrix.mjs node scripts/verify/verify_ai_intelligence_packet_contract.mjs node scripts/verify/verify_ai_advisor_quality.mjs npm run rag:verify:eval-suite:case -- npm run rag:verify:meetings npm run rag:verify:render-ai ``` Long-running commands belong in a cheaper verification sub-agent with a compact report: ```text pass/fail: command: concise error lines: likely owner files: related to current task or unrelated repo debt: ``` ## Anti-Patterns Do not: - treat council mode as a prettier chat answer - let agents vote without evidence - implement a retrieval rewrite before checking intent/provider/packet behavior - declare consensus when the roles simply repeated the same assumption - skip the report file - make "HTTP 200" the success metric - let a verifier fail silently or remain optional when it protects the chosen strategy ## Done Criteria The council is complete when: - the report exists under `docs/ai-plan/councils/` - each role gave a position with evidence and a guardrail - disagreements are explicitly resolved or left as named open questions - the recommended implementation sequence is ordered by risk and dependency - at least one recurrence-prevention guardrail is named for the first implementation slice ````