# Deep Think: Memory Upgrade — Auto-Capture Trigger Pattern Expansion

**Topic**: `auto_capture`
**Cycle**: 2
**Model**: gemini-2.5-pro
**Generated**: 2026-02-26T14:56:35.948078+00:00
**Elapsed**: 46.3s
**Thinking tokens**: 16384

---

## Generated Prompt (Phase 1 — Gemini Flash)

AIVA's current auto-capture mechanism relies heavily on the `MEMORY_TRIGGERS` regex array in `enhanced_index.ts` and the `shouldCapture` function. This array contains approximately 30 patterns covering explicit mentions of preferences, decisions, contacts, dates, locations, skills, goals, projects, URLs, and numbers. The `detectCategory` function also uses regex patterns for initial categorization. Messages are filtered by `shouldCapture` (length between 10 and `DEFAULT_CAPTURE_MAX_CHARS=500`, no prompt injection, no system content, etc.) before being passed to `splitFacts` and then stored. Importance is computed by `computeImportance` based on explicit "remember" commands, entity counts (emails, phones, dates, URLs, proper nouns detected by `extractEntities`), and general preference/decision/fact statements. The `AUTO_CAPTURE_LIMIT` is set to 5 facts per `agent_end` event.

While effective for explicit information, the current `MEMORY_TRIGGERS` array is limited in its ability to capture implicit personal information, evolving relationship dynamics, nuanced health/financial states, recurring frustrations, user values/beliefs, professional history, cultural preferences, or language code-switching signals. This results in an estimated **30-40% recall gap** for critical, yet implicitly stated, user context during `before_agent_start` `autoRecall` events, leading to less personalized interactions. Furthermore, the `computeImportance` function, while enhanced, still primarily focuses on explicit entities and keywords, potentially undervaluing implicitly derived personal information. The current system captures up to `AUTO_CAPTURE_LIMIT` memories per turn, but often misses subtle, long-term pattern-based insights.

We need to significantly expand the `autoCapture` trigger patterns to encompass a broader spectrum of personal and behavioral insights, moving beyond explicit mentions to implicit signals. This requires a multi-faceted approach involving advanced NLP techniques, sentiment analysis, and pattern generalization.

**Specific Weaknesses & Optimization Opportunities:**

1.  **Limited Implicit Pattern Recognition (Recall Gap ~30-40%):** The current `MEMORY_TRIGGERS` are too literal. We miss information like "I always find myself getting frustrated when X happens" (recurring frustration), "My family values honesty above all else" (user values/beliefs), or "I grew up speaking both English and Spanish" (cultural/linguistic context). This leads to `memory_recall` missing crucial context.
2.  **Static Importance for Implicit Memories:** `computeImportance` doesn't differentiate well for implicit information. A statement like "I hate waiting on hold" should gain higher importance if it surfaces repeatedly, even if it lacks explicit entities. We need a mechanism to dynamically boost importance for recurring implicit patterns.
3.  **Lack of Contextual Category Detection for Nuance:** `detectCategory` uses strict regex for `contact`, `event`, `location`, etc. Implicit information often falls into "other" or a miscategorized bucket, hindering effective `memory_search_advanced` and `memory_list` by category.

**Proposals:**

1.  **Proposal: Hybrid Trigger System with Embedding-based Pattern Matching for Implicit Signals.**
    *   **Objective**: Reduce the "recall gap" for implicitly stated preferences, frustrations, values, and relationship dynamics by 20% within 3 months, as measured by A/B tests on user satisfaction scores for personalization.
    *   **Mechanism**: Introduce a secondary, embedding-based trigger mechanism within `shouldCapture`. When a message doesn't match `MEMORY_TRIGGERS`, embed the user message and compare it against a curated set of "implicit pattern embeddings" (e.g., embeddings of "user values are important," "recurring frustration," "relationship status update"). If a high cosine similarity (`>0.75`) is found, trigger capture.
    *   **TypeScript Sketch (within `shouldCapture`):**
        ```typescript
        // existing MEMORY_TRIGGERS check
        if (MEMORY_TRIGGERS.some((r) => r.test(text))) {
          return true;
        }

        // NEW: Embedding-based implicit pattern check
        const implicitPatternEmbeddings = await getImplicitPatternEmbeddings(); // Load once, cache
        const messageVector = await embeddings.embed(text);

        for (const patternVector of implicitPatternEmbeddings) {
          if (cosineSimilarity(messageVector, patternVector) > IMPLICIT_CAPTURE_SIM_THRESHOLD) { // e.g., 0.75
            return true;
          }
        }
        return false;
        ```
    *   **Evaluation**: A/B test groups. Group A (current system), Group B (hybrid trigger). Measure `memory_recall` hit rate for queries related to implicit user facts (e.g., "what frustrates me?"), and qualitative user feedback on personalization. Target: increase implicit recall rate by 20%.

2.  **Proposal: Sentiment and Frequency-Boosted Importance Scoring.**
    *   **Objective**: Improve the accuracy of `importance` scores for frequently mentioned or emotionally charged implicit memories by 15%, leading to better `memory_recall` ordering.
    *   **Mechanism**: Enhance `computeImportance` to incorporate sentiment analysis and a rolling frequency count. For any captured memory, if it expresses strong negative sentiment (e.g., "hate," "frustrated," "annoyed") or positive sentiment (e.g., "love," "thrilled") *and* similar sentiments/topics have been mentioned multiple times (e.g., 3+ times in the last 7 days), apply an `IMPORTANCE_SENTIMENT_BOOST` (e.g., +0.15) or `IMPORTANCE_FREQUENCY_BOOST` (e.g., +0.10). This requires a lightweight, in-memory frequency counter keyed by embedding clusters or coarse topics.
    *   **TypeScript Sketch (within `computeImportance`):**
        ```typescript
        function computeImportance(text: string, recentCaptures?: Array<{text: string, vector: number[]}>): number {
          let baseImportance = // existing logic based on entities, explicit commands
          // ...

          const sentimentScore = analyzeSentiment(text); // Placeholder for a new utility
          if (sentimentScore < -0.6 || sentimentScore > 0.6) { // Strong negative/positive
            baseImportance += IMPORTANCE_SENTIMENT_BOOST; // e.g., 0.15
          }

          // NEW: Frequency check (requires tracking recent message embeddings/topics)
          if (recentCaptures) { // Passed from `agent_end` context
            const currentVector = await embeddings.embed(text);
            let similarMentions = 0;
            for (const rc of recentCaptures) {
              if (cosineSimilarity(currentVector, rc.vector) > RECENT_MENTION_SIM_THRESHOLD) { // e.g., 0.8
                similarMentions++;
              }
            }
            if (similarMentions >= MIN_FREQUENCY_FOR_BOOST) { // e.g., 3
              baseImportance += IMPORTANCE_FREQUENCY_BOOST; // e.g., 0.10
            }
          }

          return Math.min(1.0, baseImportance);
        }
        ```
    *   **Evaluation**: Monitor `memory_recall` results. Compare the `finalScore` of implicitly captured memories against baseline. Track the frequency of "high importance" memories (>=0.8) that stem from implicit triggers. Target: 15% increase in highly relevant implicit memories retrieved.

3.  **Proposal: Dynamic Categorization with LLM Assistance for Nuanced Categories.**
    *   **Objective**: Improve the precision of `detectCategory` for nuanced, implicit memories by 25%, ensuring better organization and retrieval via category filters in `memory_list` and `memory_search_advanced`.
    *   **Mechanism**: When `detectCategory` cannot assign a specific category (falls to "other" or a generic "fact"), or for particularly complex statements, use a lightweight, local LLM call (e.g., via Ollama and `api.chat.completions.create` if available, or a pre-trained small model) to suggest a more specific category beyond the current `MEMORY_CATEGORIES`. This could introduce sub-categories or dynamically assign a more appropriate primary category. Define a new `NUANCED_CATEGORIES` enum in `enhanced_config.ts` (e.g., `relationship-dynamic`, `health-fact`, `financial-state`, `user-value`).
    *   **TypeScript Sketch (within `detectCategory`):**
        ```typescript
        export function detectCategory(text: string): MemoryCategory | NuancedMemoryCategory {
          const lower = text.toLowerCase();
          // ... existing regex-based category detection ...

          // If still generic or 'other', invoke LLM for nuance
          if (currentCategory === "other" || currentCategory === "fact" && text.length > MIN_LLM_CATEGORY_LEN) {
            const prompt = `Categorize the following text into one of these categories: ${[...MEMORY_CATEGORIES, ...NUANCED_CATEGORIES].join(", ")}. Text: "${text}"\nCategory:`;
            const llmResponse = await api.llm.categorize(prompt); // Placeholder for LLM call

---

## Thinking Process (Phase 2 — Gemini Pro)

**My Evaluation of AIVA's Auto-Capture Enhancement Proposals**

Okay, here's my take on this. AIVA's current auto-capture mechanism, relying heavily on `MEMORY_TRIGGERS` and regex, has a well-identified recall gap of 30-40%. That's significant. The analysis provided is solid, clearly outlining the problem and the proposed solutions. I've taken the time to analyze the key components: `enhanced_index.ts`, the filtering logic in `shouldCapture`, message splitting via `splitFacts`, categorization using `detectCategory`, importance scoring via `computeImportance`, auto capture limit `AUTO_CAPTURE_LIMIT` and auto recall `autoRecall`. The proposals are well-structured, and the detailed problem description provides a strong foundation. I'm seeing three well-defined proposals for improvement, and I need to evaluate them as a senior engineer/architect. My task is to assess their strengths, identify the weaknesses and suggest any improvements.

Here's my immediate overall recommendation: These proposals are excellent and directly address the core weaknesses. I recommend a phased implementation starting with Proposal 1 (hybrid trigger system), followed by Proposal 2 (boosted importance scoring), and then Proposal 3 (dynamic LLM-based categorization). This staged approach minimizes risk, allows for iterative improvements, and provides quick wins.

**Detailed Evaluation of Each Proposal:**

*   **Proposal 1: Hybrid Trigger System (Regex + Embeddings)**
    *   *Summary:* This proposes keeping the existing regex-based `MEMORY_TRIGGERS` and supplementing it with an embedding-based similarity search to capture implicit patterns. This would be integrated into the `shouldCapture` filtering function.
    *   *Strengths:* This directly tackles the "implicit pattern recognition" weakness. It has great potential for identifying subtle user cues, frustrations, and cultural context that regex misses. Leveraging existing embedding technology reduces development time. Caching the `implicitPatternEmbeddings` is key for performance, and the hybrid approach provides a good balance between speed (regex) and accuracy (embeddings).
    *   *Potential Risks & Challenges:* Embedding lookups can introduce latency, so optimization is crucial. We need to define and curate an initial "implicit pattern" corpus, which will require ongoing effort to maintain. False positives are a concern; a high similarity score doesn't guarantee relevance.
    *   *Refinements & Recommendations:* We need to implement a negative example set alongside the positive patterns. This helps to filter out irrelevant matches. Carefully curate and categorize the implicit patterns. Consider tooling to manage this corpus and track its performance over time. A/B test the performance of the new approach.

*   **Proposal 2: Boosted Importance Scoring (Sentiment & Frequency)**
    *   *Summary:* This suggests enhancing the `computeImportance` function by adding sentiment analysis and frequency boosts. This targets the "static importance" weakness.
    *   *Strengths:* Intuitively appealing. Emotional and recurring information is more valuable. Sentiment analysis will prioritize emotionally charged memories, which are important for personalization. Frequency tracking highlights key themes in the conversation.
    *   *Potential Risks & Challenges:* Introducing sentiment analysis can increase latency. We need to choose a fast, lightweight sentiment analysis library. How do we implement the frequency counter? There are risks of over-boosting (e.g., a single "I hate traffic" comment).
    *   *Refinements & Recommendations:* For the frequency counter, a persistent but lightweight cache (e.g., Redis, in-memory LRU cache) keyed by semantic clusters is an excellent idea. Implement decay/cool-down mechanisms for sentiment boosts to prevent over-emphasis on a single statement. Monitor the distribution of the importance scores to ensure they're not skewed.

*   **Proposal 3: Dynamic LLM-Based Categorization**
    *   *Summary:* This proposes using an LLM to categorize messages that are poorly handled by regex, addressing the "poor categorization" weakness. This relies on an LLM to dynamically categorize nuanced information.
    *   *Strengths:* The LLM should be effective at categorizing the more nuanced and complex user information. Leveraging an LLM for categorization has a powerful potential for improving memory recall and personalization. The suggested use of a fixed enum `NUANCED_CATEGORIES` is smart; this provides a safety net and structure for the output.
    *   *Potential Risks & Challenges:* This is the most expensive and potentially slowest component. LLMs are prone to hallucinations, unexpected costs, and increased latency. We need robust fallbacks and strict rate limits. Hardware requirements might become a bottleneck if using a local LLM like Ollama.
    *   *Refinements & Recommendations:* Strongly emphasize using a lightweight LLM and robust error handling. Implement fallbacks to regex if the LLM fails or times out. Implement stringent cost monitoring and alerts. Make sure to define

---

## Deep Think Response

Excellent analysis and a set of well-defined, actionable proposals. This document clearly identifies the current system's limitations—the "recall gap" for implicit information—and proposes a sophisticated, multi-pronged strategy to address it. The combination of embedding-based triggers, dynamic importance scoring, and LLM-assisted categorization represents a significant architectural evolution for AIVA's memory system.

Here is a detailed evaluation of each proposal, along with recommended refinements and a strategic implementation plan.

### **Executive Summary**

The proposals are sound and directly target the core weaknesses in the current `autoCapture` mechanism. They correctly identify that a purely regex-based approach is insufficient for capturing the nuance of human conversation. The proposed hybrid system (regex + embeddings), enhanced scoring, and dynamic categorization will significantly close the 30-40% recall gap and lead to a more intuitive and personalized user experience.

I recommend a phased implementation, prioritizing the proposals in the order they are presented:
1.  **Hybrid Trigger System:** The highest priority, as it directly addresses the primary problem of capturing missed information.
2.  **Sentiment/Frequency-Boosted Importance:** A critical follow-up to ensure newly captured implicit memories are correctly prioritized.
3.  **Dynamic LLM Categorization:** The most complex and costly proposal, best implemented last to refine the organization of the newly captured data.

This approach will deliver incremental value while managing technical risk.

---

### **Detailed Proposal Evaluation**

#### **Proposal 1: Hybrid Trigger System with Embedding-based Pattern Matching**

This is the cornerstone of the entire initiative. Moving beyond literal string matching to semantic similarity is the right strategic move.

*   **Strengths:**
    *   **Directly Addresses Recall Gap:** This is a direct, high-impact solution to the problem of missing implicit context. It will capture statements like "I get so drained by back-to-back meetings" which regex would almost certainly miss.
    *   **Future-Proof:** An embedding-based system is far more flexible and scalable than manually curating hundreds of regex patterns. The "implicit pattern embeddings" can be updated and expanded over time without code changes.
    *   **Technically Feasible:** The proposed TypeScript sketch is logical. The use of a similarity threshold (`0.75`) is a standard and effective approach.

*   **Potential Risks & Challenges:**
    *   **Latency:** The `embeddings.embed(text)` call on every message that fails the regex check will introduce latency. This must be benchmarked to ensure it doesn't degrade the user's interactive experience. The call should be asynchronous and non-blocking.
    *   **Cost:** If using a third-party embedding API, this will incur costs for every message analyzed.
    *   **False Positives:** A similarity threshold might incorrectly trigger capture on irrelevant chatter that is semantically close to a pattern. For example, a user discussing a movie character's values might be mistaken for stating their own.
    *   **Corpus Curation:** The effectiveness of this system depends entirely on the quality and breadth of the "implicit pattern embeddings." This corpus needs careful creation, curation, and maintenance.

*   **Refinements & Recommendations:**
    1.  **Optimize Performance:**
        *   Ensure `getImplicitPatternEmbeddings()` is truly a singleton that loads and caches the pattern vectors on service startup.
        *   Run the embedding check *after* all other `shouldCapture` filters (length, system content, etc.) to minimize unnecessary API calls.
    2.  **Refine the Pattern Corpus:**
        *   The corpus should contain not just positive examples (e.g., "I find X frustrating") but also canonical examples of different *types* of implicit information (values, beliefs, recurring events, relationship statements).
        *   Consider adding a small set of "negative pattern embeddings" (e.g., embeddings of "hypothetical scenario," "quoting someone else"). If a message is highly similar to a negative pattern, we can explicitly block capture, reducing false positives.
    3.  **Implement a Kill Switch:** Use a feature flag to enable/disable the embedding-based trigger mechanism for safe deployment and A/B testing.

---

#### **Proposal 2: Sentiment and Frequency-Boosted Importance Scoring**

This is a crucial enhancement. Capturing more information is only useful if we can accurately gauge its importance. This proposal correctly identifies that emotion and repetition are strong signals of significance.

*   **Strengths:**
    *   **Dynamic Prioritization:** Moves `computeImportance` from a static, entity-based model to one that reflects the user's emotional state and recurring concerns.
    *   **Surfaces Key Frustrations/Preferences:** A recurring complaint about "waiting on hold" will rightly be boosted, making it more likely to be recalled when the user is about to make a phone call.
    *   **Efficient:** A lightweight sentiment analysis library and a simple frequency counter should not add significant overhead.

*   **Potential Risks & Challenges:**
    *   **Frequency Tracking Complexity:** The sketch `if (recentCaptures)` is a good start but is limited to the current context. Tracking frequency across sessions requires a more robust mechanism (e.g., a Redis cache or a dedicated table) which adds infrastructure complexity.
    *   **Defining "Similar":** The `RECENT_MENTION_SIM_THRESHOLD` is another sensitive hyperparameter. Setting it too high will miss variations of the same topic; setting it too low will incorrectly group unrelated topics.
    *   **Sentiment Ambiguity:** Sarcasm and complex sentences can fool simple sentiment analyzers, potentially leading to incorrect importance boosts.

*   **Refinements & Recommendations:**
    1.  **Phased Frequency Tracking:**
        *   **Phase 1 (Simple):** Start with the proposed `recentCaptures` approach, limiting the scope to the current session. This is low-cost and will capture in-conversation repetition.
        *   **Phase 2 (Advanced):** Implement a persistent, time-windowed cache (e.g., Redis with TTLs). Store a hash or a quantized representation of the message embedding as the key and a counter/timestamp as the value. This enables cross-session frequency tracking (e.g., "mentioned 3 times in last 7 days").
    2.  **Use a Fast Sentiment Library:** Integrate a small, performant sentiment analysis library (e.g., `sentiment` for Node.js) to avoid adding latency to the capture process.
    3.  **Apply Importance Decay:** To prevent a single topic from becoming permanently high-importance, consider applying a decay factor to the frequency boost over time.

---

#### **Proposal 3: Dynamic Categorization with LLM Assistance**

This proposal tackles the final piece of the puzzle: organization. Using an LLM for nuanced categorization is a powerful technique when regex fails.

*   **Strengths:**
    *   **Solves the "Other" Bucket Problem:** This is an elegant solution for categorizing complex, implicit statements that don't fit into predefined regex-based categories.
    *   **Improved Retrieval:** Well-defined categories (`user-value`, `health-fact`) make targeted `memory_search_advanced` and `memory_list` queries significantly more effective.
    *   **Adaptable:** The LLM's reasoning capability can handle novel phrasings and topics much better than a rigid rules-based system.

*   **Potential Risks & Challenges:**
    *   **Latency & Cost:** This is the most significant concern. An LLM call, even to a local model, is orders of magnitude slower and more resource-intensive than regex. This could severely slow down the `agent_end` processing pipeline.
    *   **Reliability:** The LLM might fail, return a malformed response, or hallucinate a category not in the provided list. The system must be resilient to these failures.
    *   **Model Management:** If using a local model via Ollama, this introduces a new operational dependency. The model needs to be managed, updated, and scaled.

*   **Refinements & Recommendations:**
    1.  **Strict Guardrails:**
        *   Use the LLM *only* as a fallback, as proposed. The prompt must be highly structured (few-shot prompting with examples can help) to constrain the output to the predefined `NUANCED_CATEGORIES` enum.
        *   Implement aggressive timeouts and a circuit-breaker pattern for the LLM call. If it fails or times out, gracefully fall back to the default category ("other" or "fact").
    2.  **Asynchronous Processing:** The LLM categorization step should ideally be offloaded to a background job or queue. The memory can be saved initially with a generic category (`fact-uncategorized`) and then updated asynchronously by the LLM worker. This prevents blocking the main event loop.
    3.  **Cost/Performance Budgeting:** Define a strict budget (e.g., max 2 LLM calls per `agent_end` event) to prevent runaway costs or performance degradation. Monitor latency and cost metrics closely.

### **Recommended Implementation Roadmap**

**Phase 1: Foundation - Capture (Next 3 Months)**

1.  **Task:** Implement **Proposal 1 (Hybrid Trigger System)**.
2.  **Steps:**
    *   Curate the initial v1 corpus of 15-20 canonical "implicit pattern" sentences.
    *   Set up the embedding infrastructure (either API client or local model).
    *   Implement the caching for `implicitPatternEmbeddings`.
    *   Modify `shouldCapture` with the new logic behind a feature flag.
    *   Establish the A/B test framework to measure the recall gap reduction and user satisfaction.
3.  **Goal:** Reduce implicit information recall gap by 20%. Validate that latency is within acceptable limits (<150ms overhead per message).

**Phase 2: Refinement - Prioritization (Month 4-5)**

1.  **Task:** Implement **Proposal 2 (Sentiment/Frequency-Boosted Importance)**.
2.  **Steps:**
    *   Integrate a lightweight sentiment analysis library.
    *   Implement the *session-based* frequency counter first.
    *   Update `computeImportance` with the sentiment and frequency boosts, tuning the boost values.
    *   Monitor the distribution of importance scores to ensure they are being applied meaningfully.
3.  **Goal:** Increase the retrieval relevance of implicit memories by 15%, as measured by the `finalScore` in `memory_recall` for test queries.

**Phase 3: Organization - Categorization (Month 6+)**

1.  **Task:** Implement **Proposal 3 (Dynamic LLM Categorization)**.
2.  **Steps:**
    *   Define the `NUANCED_CATEGORIES` enum.
    *   Set up the LLM integration (e.g., Ollama client) with strict timeouts and error handling.
    *   Implement the fallback logic in `detectCategory`. Consider the async/background job approach for performance.
    *   Deploy with heavy monitoring on latency, error rates, and cost/resource usage.
3.  **Goal:** Improve the precision of nuanced category assignment by 25%, measured by manual review of categorized memories.
