type: axiom

author: RLM Gold Team Lead

axiom: RLM infrastructure is complete; the learning loop is blocked by missing human feedback collection, not missing code.

evidence: 9,109 lines of RLM code exist across 5 modules. PostgreSQL has 12 tables. Telnyx call.hangup is wired to process_interaction(). But pl_preference_pairs has 0 rows. No feedback = no training = no improvement.

action: Build Telegram inline button feedback mechanism FIRST before any other RLM work.


axiom: Use DPO (Module 4) for first training run, not PPO (Module 3).

evidence: DPO trains directly from preference pairs. No rollout buffer. No Redis dependency. 10x faster iteration than PPO. PPO requires rollout buffer (Redis is currently broken). DPO produces comparable quality for preference-based alignment tasks.

action: After 100 preference pairs collected, trigger rlm_04_dpo_trainer. PPO is Phase 2 only.


axiom: Shadow mode is correct until 200+ interactions are logged AND reward model is validated against held-out preferences.

evidence: Shadow mode = RLM processes and logs WITHOUT altering AIVA responses. This is safe. Flipping to live mode before reward model validation risks degrading AIVA's real call performance, which has revenue implications.

action: Keep shadow_mode=True until (a) 200 interactions logged, (b) reward model trained on 100+ pairs, (c) A/B test shows +5% reward improvement.


axiom: Treat RLM training data as a searchable environment, not context to stuff into prompts. O(n) not O(n²).

evidence: MIT arXiv:2512.24601. Genesis parallel Task spawning IS async RLM in practice — each task result is a micro-chunk processed independently, not appended to a growing context window.

action: When designing RLM data pipelines, prefer Qdrant semantic search over PostgreSQL full-scans for preference pair retrieval.


axiom: Preference feedback must arrive within 60 seconds of call end to maximize Kinan's recall and labeling accuracy.

evidence: Human memory degrades rapidly. A feedback request sent 24h after a call will produce unreliable labels. Immediate post-call Telegram message with summary + thumbs up/down achieves maximum signal quality.

action: Telegram notification fires inside process_interaction() after call analysis completes — not via a separate scheduled job.


axiom: Constitutional AI is the safety guardrail that runs on EVERY interaction regardless of reward model state.

evidence: rlm_05 runs inside process_interaction() via critique_output(). Even with mock reward scoring, 20 constitutional principles are checked on every voice call. This cannot be disabled.

action: Never gate CAI checks on reward model readiness. CAI runs unconditionally.


axiom: Module dependency chain: human feedback → rlm_01 → rlm_02 → rlm_04. Unblocking rlm_01 unblocks everything downstream.

evidence: rlm_01 produces preference pairs → rlm_02 trains reward model → rlm_04 trains policy via DPO. rlm_03 (PPO) depends on Redis (broken). rlm_05 (CAI) is independent. The single P0 blocker is 0 preference pairs.

action: All engineering effort should point at getting preference pairs to 100 as fast as possible.


axiom: Fix Redis AFTER human feedback is flowing, not before.

evidence: RedisConfig.get_client_params missing is P2 non-blocking. Redis only matters for PPO rollout buffer, which is not the first training method. Fixing Redis now delays the P0 fix (human feedback collection).

action: Add get_client_params classmethod to RedisConfig in elestio_config.py when starting PPO Phase 2 work.


axiom: The heuristic reward mock (good=0.80, bad=0.30) is safe for shadow mode but must not be used to gate AIVA's responses.

evidence: The mock scores every 'good' output at 0.80 and 'bad' at 0.30 based on keyword heuristics. This is directionally correct but lacks calibration. In shadow mode this is fine for logging. If used to gate responses, it would introduce systematic bias.

action: Fix RewardInference checkpoint_path signature before activating live reward gating.


axiom: AIVA advances from Level 2 (Knight) to Level 3 (Advisor) via verified interaction count + RLM-validated performance, not time elapsed.

evidence: trust_progression.json: Level 2 requires 100 interactions with 4/6 criteria met. Currently 38/100. Level 3 criteria include reward model score improvement, CAI violation rate < 5%, and positive call outcome rate > 70%.

action: Every voice call handled by AIVA increments the counter. RLM validation prevents gaming by requiring quality metrics, not just volume.


axiom: RLM modules are the Cerebral Cortex (planning + learning). Execution agents (OpenClaw, task runners) are the Motor Cortex. The MCP bridge is the synapse between them.

evidence: Gemini Deep Think DT22: biological superorganism model. RLM thinks and validates. Execution agents act. The bridge ensures RLM output (validated action manifest) is consumed by execution agents. Never let Motor Cortex act without Cerebral Cortex validation in high-stakes operations.

action: Future: wire agent_task_completion events as RLM inputs so agent success/failure feeds the preference learning loop.


category: monitoring

axiom: The RLM health dashboard query is: SELECT date_trunc('day', created_at) AS day, COUNT(*) AS calls, AVG(reward_score) AS avg_reward, SUM(CASE WHEN outcome_label='positive' THEN 1 ELSE 0 END)::float/COUNT(*) AS positive_rate FROM aiva_rlm.aiva_feedback_scores GROUP BY 1 ORDER BY 1 DESC LIMIT 14;

evidence: This 14-day trend query is the single most important diagnostic. If avg_reward is rising and positive_rate is rising, training is working. If flat after 100+ interactions, reward model needs retraining.

action: Run this query daily as part of AIVA morning status check.


category: naming

axiom: AIVA/logging.py must be renamed to AIVA/aiva_logging.py to prevent Python stdlib shadowing.

evidence: AIVA/logging.py shadows Python stdlib logging module. When AIVA directory is in sys.path, asyncio and all stdlib modules that import logging crash with AttributeError: partially initialized module. This is a P1 time-bomb that will manifest in production.

action: Rename file, update all imports inside AIVA/ directory. This is a 30-minute fix. Do not defer.


category: testing

axiom: The 3-test RLM Gateway smoke test (Constitutional Check + Preference Collection + Reward Scoring) must pass before any deployment or code change to RLM modules.

evidence: All 3 tests pass as of 2026-02-19 per rlm_pg_connection_report.md. These tests validate: (1) CAI principles load from constitution.json, (2) a preference pair can be written to PostgreSQL, (3) reward scoring returns valid float values.

action: Run python AIVA/rlm_gateway.py after any RLM code change. 3/3 PASS is the minimum bar.


category: velocity

axiom: Time to 100 preference pairs is the single velocity metric for RLM Phase 1. Everything else is a trailing indicator.

evidence: Until 100 preference pairs exist, nothing trains. The BradleyTerry model, neural reward model, and DPO trainer are all gated on this number. Kinan labeling 3 calls/day = 33 days to 100. Telegram automation makes this zero-friction.

action: Build Telegram feedback collection. Target: 100 pairs within 30 days of launch.