'i֚ dZddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z ejjdejjdejjdddlmZmZdd lmZdd lmZmZmZdd lmZmZmZ ddlZdd l m!Z!m"Z" ddl%Z%ejLejNdejPdZ)GddZ*GddZ+GddZ,GddZGddZ-da.de-fdZ/d$de0de e0de e0e ffdZ1d$de0d e0de e0fd!Z2d"Z3e4d#k(rejje3yy#e#$r dZe$d YwxYw#e#$r dZ%e$dYwxYw)%a RLM Gateway - Integration Layer for Reinforcement Learning from Memory This module connects the 5 RLM modules to AIVA's live decision-making loop: 1. Preference Learning (rlm_01) 2. Reward Model (rlm_02) 3. PPO Engine (rlm_03) 4. DPO Trainer (rlm_04) 5. Constitutional AI (rlm_05) The gateway: - Intercepts AIVA outputs before sending to users - Scores outputs using the trained reward model - Revises outputs using Constitutional AI - Collects human feedback for training - Triggers training when sufficient data accumulated - Manages A/B testing for safe policy deployment Author: Genesis-OS RLM Integration Version: 1.0.0 Date: 2026-02-16 N)datetime)DictListOptionalTuplez/mnt/e/genesis-systemz)/mnt/e/genesis-system/data/genesis-memoryz,/mnt/e/genesis-system/AIVA/queen_outputs/rlm)PostgresConfig RedisConfig)PreferenceDataset) RewardModelr RewardInference) Constitution SelfCritique RevisionLoop)JsonRealDictCursorzCWarning: psycopg2 not installed. RLM Gateway will run in mock mode.z@Warning: redis not installed. RLM Gateway will run in mock mode.z4%(asctime)s - %(name)s - %(levelname)s - %(message)s)levelformat RLMGatewayceZdZdZddefdZdefdZdedeee eeffdZ dedede efd Z d ede fd Zded ed e edefdZy)_PlaceholderConstitutionalAIz8Constitutional AI module for self-critique and revision.constitution_pathcJ||_|j|_d|_y)NT)r_load_constitution constitutionenabledselfrs )/mnt/e/genesis-system/AIVA/rlm_gateway.py__init__z%_PlaceholderConstitutionalAI.__init__Gs"!2 335 returnc t|jd5}tj|cdddS#1swYyxYw#t$r*t j d|jgidcYSwxYw)z.Load constitutional principles from JSON file.rNzConstitution file not found: ) principles enforcement)openrjsonloadFileNotFoundErrorloggerwarning)rfs rrz/_PlaceholderConstitutionalAI._load_constitutionLsm 9d,,c2 $ayy| $ $ $  9 NN:4;Q;Q:RS T"$R8 8 9s*A7 AAAA0A65A6outputcjKg}g}d}|jjdgD]q}|j||}|s|j|d|jd|dd||j |d|j |kDsm|d}s|rdj |nd }|||fSw) z Critique output against constitutional principles. Returns: (critique_text, violated_principle_ids, severity) lowr$id Violates name: severity No violations detected)rget_check_principleappend_severity_rankjoin)rr-violated critiques max_severity principle violation critique_texts rcritique_outputz,_PlaceholderConstitutionalAI.critique_outputUs  **..|R@ 9I--fi@I $0  9Yv->,?r)!MN&&y'<=@S@ST`@aa#,Z#8L  91: ),?W h 44s:B3AB3!B3r?ch|d}|dk(rd|vsd|vryd|vsd|vry|d k(r3gd }|D]*}|j|jvs$d |d cS|d k(r3gd}|D]*}|j|jvs$d|d cS|dk(r!t|dkDrd|jvryy)z Check if output violates a specific principle. Returns violation description if violated, None otherwise. r0p003zC:zC:\z-Output references C: drive which is forbiddenz152.53.201.152z aiva-miniz2Output references AIVA server which is untouchablep002)zprobably costsztypically aroundzusually priced atzbased on industry standardszmost competitorsz I estimatez(Output contains fabrication indicator: ''p005)zlet's refactor for elegancez internal toolz1% efficiency gainzno customer valuezpurely for maintenancez*Output suggests revenue-negative action: 'p004izdispatching agentz2Output is >5000 chars without delegating to agentsN)lowerlen)rr-r? principle_idfabrication_keywordskeywordrevenue_negativepatterns rr8z-_PlaceholderConstitutionalAI._check_principlels !  6 !v~6!1F6)[F-BK 6 !$ 0 Q==?flln4EgYaPP Q 6 !  , S==?flln4GyPQRR S 6 !6{T!&9&OKr r4c0dddddj|dS)z(Convert severity string to numeric rank.)r/mediumhighcritical)r7)rr4s rr:z+_PlaceholderConstitutionalAI._severity_ranksAqa@DDXqQQr critiqueviolated_principlescXK|s|Sd|d|ddj|d}d|d|Sw)a3 Revise output to comply with constitutional principles. In production, this would call an LLM (Claude/Gemini) with: - Original output - Critique - Violated principles - Examples of compliant outputs For now, returns a revision instruction. z4 CONSTITUTIONAL REVISION REQUIRED Original output: z Critique: z Violated principles: , z5 Please revise to comply with Genesis Constitution. z[CONSTITUTIONAL WARNING: ] )r;)rr-rXrYrevision_prompts r revise_outputz*_PlaceholderConstitutionalAI.revise_outputs^#M   ii 3456  +8*E&BB(*Nz,/mnt/e/genesis-system/AIVA/constitution.json)__name__ __module__ __qualname____doc__strrrrrrrBrr8intr:r^r rrrDsB# 9D95C5E#tCy#:M4N5.(s(t( (TRsRsRC#CCSWX[S\CadCr rc jeZdZdZd defdZd dededeeeeeffdZd deded eededef d Z y )ConstitutionalAIzEWrapper for Constitutional AI modules for self-critique and revision.rct||_t|j|_t |j|_y)N) config_path)r)critique_engine)r rrrlr revision_looprs rrzConstitutionalAI.__init__s6(5FG+9J9JK)$:N:NOr r-promptr!cK|jj||}g}g}d}|D]}|js|j|j|jd|j d|j |$|jj|jkDs||j}|rdj|nd}|r|jjnd} | dk(r|rd} ||| fSw) z Critique output against constitutional principles using the actual RLM module. Returns: (critique_text, violated_principle_ids, severity) rnresponseNr1r3r5r6noner/) rlrXr<r9rKprinciple_name explanationr4valuer;r2rI) rr-rncritique_resultsr<critiques_textmax_severity_enumc_resultcritique_summary severity_strs rrBz ConstitutionalAI.critique_outputs //88QW8X ( :H   5 56%% (2I2I1J"XMaMaLb&cd$,0A0A0G0GJ[JaJa0a(0(9(9%  :9G499^4Ld9J(--335PV 6 !h L<77s5C3A+C3$AC3rXrYcK|jj||}|jr |jSdt |j d|j d}||jzSw)ze Revise output to comply with constitutional principles using the actual RLM module. rpz4[CONSTITUTIONAL WARNING: Remaining issues detected: z. Final verdict: r\)rmrevise is_compliantrevised_responserJremaining_issues final_verdict)rr-rXrYrnrevision_resultr+s rr^zConstitutionalAI.revise_outputs ,,336F3S  ' '"33 3MSQ`QqQqMrLstEFUFcFcEddijG_=== =sA+A-Nr`)) rarbrcrdrerrrrBr^rgr rririsrOP#P 8C88eCQUVYQZ\_L_F`88 ># > >SWX[S\ >fi >sv >r ricFeZdZdZd deefdZdZd dedeedefd Z y) _PlaceholderRewardModelz+Reward model inference for scoring outputs.Ncheckpoint_pathc<||_|j|_yN)r _load_modelmodel)rrs rrz _PlaceholderRewardModel.__init__s.%%' r c|jr)tjj|jstj dyy)z*Load trained reward model from checkpoint.z5No reward model checkpoint found. Using mock scoring.N)rospathexistsr*r+rs rrz#_PlaceholderRewardModel._load_models5##277>>$:N:N+O NNR Sr r-contextr!cK|jid}gd}tfd|Dr|dz }tdDr|dz }gd}tfd|Dr|dz}td td |Syw) z Score output quality using trained reward model. Returns: Float score 0.0 to 1.0 (higher = better) g?) dispatchingcreating executing deployingbuildingc3BK|]}|jvywrrI).0verbr-s r z7_PlaceholderRewardModel.score_output.. sCd46<<>)C皙?c3<K|]}|jywr)isdigit)rchars rrz7_PlaceholderRewardModel.score_output..s5d4<<>5sg?)mightcouldmaybeperhapszI thinkc3BK|]}|jvywrr)rphraser-s rrz7_PlaceholderRewardModel.score_output..sH6V\\^+Hr?)ranymaxmin)rr-rscore action_verbs vague_phrasess ` r score_outputz$_PlaceholderRewardModel.score_outputs :: E]LClCC 5f55 NMH-HH sCUO, ,sA7A:r) rarbrcrdrrerrfloatrrgr rrrs95( (x}PUr rc@eZdZdZd deefdZd dedeedefdZy) r z0Wrapper for RewardInference for scoring outputs.Nrc td}t||_d|_tj dy#t $r:}tjd|t||_d|_Yd}~yd}~wwxYw) Ni) embedding_dim)rTz3Real RewardModel (via RewardInference) initialized.z=Failed to initialize Real RewardModel, falling back to mock: )rF) RealRewardModelr inference_enginerr*info Exceptionerrorr)rr dummy_modeles rrzRewardModel.__init__ sn !)L;;..D IIK KK, -K  LL7s; < sAA.. B7BBNr-rr!cKtj}t|ddid|jd}|jj ||d{}||d<|j j ||xsdd{\}}}|||d |d <|} |rn|j jjrN|j j||||xsdd{} d |d <|j||| || d{|jr0tj|jkr|j|d<tj|z |d<t| |d<| |fS7777sw)a Main entry point: process AIVA output before sending. Steps: 1. Score output with reward model 2. Check constitutional compliance 3. Revise if necessary 4. Log processing metadata Args: output: AIVA's raw output context: Optional context (input that led to this output) Returns: (final_output, metadata) rrF)original_lengthprocessing_time reward_scoreconstitutional_checkrevisedpolicy_versionNrr)rn)rYr4rXrTroriginal_outputrXrevised_outputrYr4rr final_length)timerJrrrrrBrlrr^_log_constitutional_violationrrandomrr) rr-r start_timemetadatarrXrYr4 final_outputs rprocess_outputzRLMGateway.process_output}s"YY[ "6{ $&"99  "..;;FGLL #/ 9=8N8N8^8^_enun{y{8^8|2|/%x#6  , '( 4#9#9#I#I#Q#Q!%!7!7!E!EfhXkt{uB@B!E"CCL"&HY 44 &!+$7! 5     FMMOd6S6S$S)-)@)@H% &&*iikJ&>"##&|#4 X%%CM3}C sJAE5E,-E5E/AE5E1"E5?E3A-E5/E51E53E5feedbackcK|jtjdy |jj5}|j d|j |xsd||fddd|jj tjd||jd{y#1swYTxYw7#t$r"}tjd|Yd}~yd}~wwxYww)a Collect human feedback on an output. Args: output: The output that was shown to the user feedback: User feedback ('good', 'bad', 'regenerate', or free text) context: Optional context (what led to this output) Nz&No DB connection. Feedback not stored.z INSERT INTO rlm_ab_test_results (policy_version, input_text, output_text, user_feedback, created_at) VALUES (%s, %s, %s, %s, NOW()) rzFeedback collected: zFailed to collect feedback: ) rr*r+cursorexecutercommitr_check_training_triggerrr)rr-rrcurrs rcollect_feedbackzRLMGateway.collect_feedbacks <<  NNC D  =$$& #  //Mr    LL   ! KK.xj9 :..0 0 0   1 = LL7s; < < =sS#C4C%B8%A C2C3C7C48C=C C1C,'C4,C11C4output_aoutput_bchoice input_text confidence annotator_idrc K|jtjdy |jj5}|j d|xsd||||||rt j |ndfddd|jjtjd|d|d|jd{y#1swYXxYw7#t$r"} tjd| Yd} ~ yd} ~ wwxYww) a Collect preference comparison: which output is better? Args: output_a: First output option output_b: Second output option choice: 1 (A preferred), -1 (B preferred), 0 (tie) input_text: Optional input that generated these outputs confidence: Confidence in preference (0.0 to 1.0) annotator_id: ID of the annotator (default: "auto") metadata: Optional dictionary for additional metadata Nz(No DB connection. Preference not stored.z INSERT INTO pl_preference_pairs (input_text, output_a, output_b, preference, confidence, annotator_id, metadata, created_at) VALUES (%s, %s, %s, %s, %s, %s, %s, NOW()) rzPreference collected: z (confidence: )zFailed to collect preference: ) rr*r+rrr'dumpsrrrrr) rrrrrrrrrrs rcollect_preferencezRLMGateway.collect_preferences <<  NNE F  ?$$& #  $" ,4DJJx($   LL   ! KK0zlRST U..0 0 0%  $ 1 ? LL9!= > > ?sS#DC5C 5ACCC D CC D#D;DDDrrXrrYr4c K|jy |jj5}|jd|||||f|D]}|jd|f ddd|jjtj d|ddj ||dk(rtjd|yy#1swYlxYw#t$r"}tjd |Yd}~yd}~wwxYww) z)Log constitutional violation to database.Nz INSERT INTO cai_critique_log (original_output, critique, revised_output, violated_principles, severity, created_at) VALUES (%s, %s, %s, %s, %s, NOW()) aA INSERT INTO cai_principle_violations (principle_id, violation_date, count) VALUES (%s, CURRENT_DATE, 1) ON CONFLICT (principle_id, violation_date) DO UPDATE SET count = cai_principle_violations.count + 1 z!Constitutional violation logged: z - r[rWz#CRITICAL CONSTITUTIONAL VIOLATION: z(Failed to log constitutional violation: ) rrrrr*r+r;rWrr) rrrXrrYr4rrKrs rrz(RLMGateway._log_constitutional_violation s <<   I$$& *#  $"'  %8*LKK! ' ** *, LL   ! NN>xjDIIViLjKkl m:%"EhZ PQ&5 * *< I LLCA3G H H IsFDC2C A)CD CC DC;6D;DDcK|jy |jj5}|jd|jd}ddd|jk\r2t j d|d|jd{yy#1swYKxYw7#t$r"}t jd|Yd}~yd}~wwxYww)z8Check if training should be triggered based on new data.Na_ SELECT COUNT(*) FROM pl_preference_pairs WHERE created_at > ( SELECT COALESCE(MAX(created_at), '1970-01-01') FROM rlm_training_triggers WHERE training_module = 'reward_model' AND status = 'completed' ) rzTraining threshold met: z& new preferences. Triggering training.z"Failed to check training trigger: ) rrrfetchonerr*rtrigger_trainingrr)rr new_prefsrs rrz"RLMGateway._check_training_trigger2s <<   C$$& .#  LLN1-  .D333 6ykAghi++---4 . .. C LL=aSA B B CsRCB(%BAB(B&B(CB#B(( C1C CCCmodulemanualcK|jtjdy|rdnd} |jj5}|j d|d||f|j d}ddd|jj tjdd |y#1swY?xYw#t$r"}tjd |Yd}~yd}~wwxYww) z Trigger RLM training run. Args: module: Which module to train ('reward_model', 'ppo', 'dpo', 'cai') manual: Whether this is a manual trigger (vs automatic threshold) Nz*No DB connection. Cannot trigger training.r thresholdz INSERT INTO rlm_training_triggers (trigger_type, trigger_reason, training_module, status, created_at) VALUES (%s, %s, %s, 'queued', NOW()) RETURNING id zTraining triggered: rzTraining trigger created: ID=z , module=zFailed to trigger training: ) rr*r+rrrrrrr)rrr trigger_typer trigger_idrs rrzRLMGateway.trigger_trainingKs <<  NNG H #)x{  =$$& /#  !*<.9 !\\^A.  / LL   ! KK7 |9VHU V / /( = LL7s; < < =sF)C*B<,B02=B</C*0B95B<< C'C"C*"C''C*call_id transcript caller_numbercall_duration_secondsoutcomec Kd|dddgddd} |jj|d{\}}} ||d<| |d<|r|j||||| d{|jj |d|d{} | |d <|j |||} | |d <|j l|j|||||| | || d{} | |d <|jd{tjd |d| dd| dt||Stjd|d| d |S7777t7Y#t$r;} t| |d<d|d<tjd|d| dYd} ~ |Sd} ~ wwxYww)a Process a completed voice call interaction through the full RLM pipeline. Called by the Telnyx webhook handler on call.hangup events. Runs in SHADOW MODE: logs data only, does not alter call behaviour. Pipeline: 1. Store interaction record in aiva_interactions 2. Run Constitutional AI check on transcript 3. Compute heuristic reward score 4. Infer outcome label (positive/negative/neutral) 5. Store scored record in aiva_feedback_scores 6. Check training trigger threshold Args: call_id: Telnyx call session ID transcript: Full concatenated transcript text caller_number: Caller phone number (for context) call_duration_seconds: Total call length outcome: 'completed', 'transferred', 'voicemail', 'error' Returns: Dict with rlm_processed=True and metadata TN) rlm_processedr shadow_modeinteraction_idrcai_violations outcome_labelrr cai_severityrzcall_id=)rrr) rrrr r rrrrrzRLM shadow processed call z : reward=z.3fz , outcome=z, cai_violations=zRLM shadow mode (no DB): call z scored rFr z(RLM process_interaction failed for call r3)exc_info)rrBrrr_infer_outcome_labelr_store_aiva_interactionrr*rrJr+rrer)rrrrr r resultrXrYr4rrrrs rprocess_interactionzRLMGateway.process_interactionqs*B""  !  4 c<@ ""88$.%#-(;% 9"&!2!2!?!? V^_f^gTh!?!iiL%1F> "!55j'K`aM&3F? #||''+'C'C#)"/*?#!-"/#6!)(D ( ",:'(22444 0 :*3/z-I&&)*=&>%?A !?yQ]^aPbcd i7o j "5 c!!fF7O&+F? # LLCG9BqcR]aL b b  cs FED<-ED?'EEA EEE+E,/EFE:F<E?EEEE F 0FFF  Fduration_secondsc|jgd}gd}tfd|D}tfd|D}|dkr|dz }|dkDr |dkDr|dz }|d k(ry ||kDry ||kDry y ) z Infer a positive/negative/neutral label from call signals. Positive signals: caller says thank you, booking confirmed, long call Negative signals: caller frustrated, hung up quickly, escalation keywords )z thank youthanksperfectgreat excellentbook appointmentscheduleconfirmz yes pleasezthat's helpfulz sounds good brilliantamazing) z forget itz never mindz not helpfulterribleuselesswaste frustratedangrycancelz wrong numberz do not callc3,K|] }|vsd ywrQNrgrkwtexts rrz2RLMGateway._infer_outcome_label..I2bDjQI c3,K|] }|vsd ywr*rgr+s rrz2RLMGateway._infer_outcome_label..r.r/rQxrrnegativepositiveneutral)rIsum) rrr rpositive_keywordsnegative_keywordspositive_countnegative_countr-s @rrzRLMGateway._infer_outcome_labels!    I):III):II b a N c !nq&8 a N g  n , n ,r rrrrc Kd} |jj5} | jd| jd| jd| jd| jd||||||f| j} | r| dnd} | jd| |||| |jfddd|jj | S#1swY%xYw#t $rQ} tjd |d |  |jjn#t $rYnwxYwYd} ~ | Sd} ~ wwxYww) zIPersist interaction to aiva_interactions and aiva_feedback_scores tables.Nz$CREATE SCHEMA IF NOT EXISTS aiva_rlma$ CREATE TABLE IF NOT EXISTS aiva_rlm.aiva_interactions ( id SERIAL PRIMARY KEY, call_id VARCHAR(255) UNIQUE NOT NULL, caller_number VARCHAR(50), transcript TEXT, call_duration_seconds INTEGER DEFAULT 0, outcome VARCHAR(50), outcome_label VARCHAR(20), created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ) a CREATE TABLE IF NOT EXISTS aiva_rlm.aiva_feedback_scores ( id SERIAL PRIMARY KEY, interaction_id INTEGER REFERENCES aiva_rlm.aiva_interactions(id), call_id VARCHAR(255) NOT NULL, reward_score DECIMAL(6,4), cai_violations TEXT[], cai_severity VARCHAR(20), policy_version VARCHAR(50), shadow_mode BOOLEAN DEFAULT TRUE, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ) a CREATE TABLE IF NOT EXISTS aiva_rlm.aiva_preference_pairs ( id SERIAL PRIMARY KEY, call_id_a VARCHAR(255), call_id_b VARCHAR(255), preferred VARCHAR(10), confidence DECIMAL(4,3) DEFAULT 1.0, annotator VARCHAR(50) DEFAULT 'auto', created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ) a INSERT INTO aiva_rlm.aiva_interactions (call_id, caller_number, transcript, call_duration_seconds, outcome, outcome_label) VALUES (%s, %s, %s, %s, %s, %s) ON CONFLICT (call_id) DO UPDATE SET outcome = EXCLUDED.outcome, outcome_label = EXCLUDED.outcome_label RETURNING id rz INSERT INTO aiva_rlm.aiva_feedback_scores (interaction_id, call_id, reward_score, cai_violations, cai_severity, policy_version) VALUES (%s, %s, %s, %s, %s, %s) z!Failed to store AIVA interaction r3) rrrrrrrr*rrollback)rrrrr r rrrrrrrowrs rrz"RLMGateway._store_aiva_interactionspR $$&H # BC       }j:OQXZghjlln+.QD  # " //  {H T LL   !gH H X  LLs renable_ab_testzRLMGateway.enable_ab_testbsB $"4(5% (s):(;=I[H\]^s57hourscK|jiS |jjt5}|jd|j }dddDcgc] }t |c}|j |dS#1swY4xYwcc}w#t$r$}tjd|icYd}~Sd}~wwxYww)z*Get A/B test results for the last N hours.N)cursor_factoryzH SELECT * FROM v_ab_test_comparison )resultswinnerz Failed to get A/B test results: ) rrrrfetchalldict_determine_ab_winnerrr*r)rrArrDr=rs rget_ab_test_resultszRLMGateway.get_ab_test_resultsos << I $$N$C )s ,,.  )299#DI933G<   ) ):  LL;A3? @I sWC B"B B B2BCB B C!C:C;CCCrDct|dkrytfd|Dd}tfd|Dd}|r|sy|jdd}|jdd}||dzkDry ||dzkDry y ) z(Determine which policy won the A/B test.rRinsufficient_datac3HK|]}|djk(s|ywrN)rrr#rs rrz2RLMGateway._determine_ab_winner..s$d3C1DHcHc1c1d""Nc3HK|]}|djk(s|ywrM)rrNs rrz2RLMGateway._determine_ab_winner..s$`3C1DH_H_1_1`rO avg_rewardrg? new_policy old_policy inconclusive)rJnextr7)rrDrSrR old_score new_scores` rrHzRLMGateway._determine_ab_winners wrs.  ..'(;<>?7VCCMMQ4 N  ,, A   < (uCuCn1>1>h,,\MM>i $i $`Z 9c 9HSM 9USVX\S\M] 9 >3 ># > > *Z zGKK[QH OPQ N E LMNs$% E"2E7"E43E47F F