Category-Specific Memory Decay System

System that implements different decay models for memory entries based on category, configured via CATEGORY_DECAY_PARAMS. Affects importanceScore and recencyScore in hybridSearch.


DecayParams Configuration Interface

Interface for defining decay parameters (decayFunction, decayRate, halfLifeDays, minimumImportance, eventCompletionDecayMultiplier, freshnessBoost) for different memory categories in CATEGORY_DECAY_PARAMS.


Use Memory Category to Adjust Importance/Recency

The system will differentiate decay models based on a `MemoryCategory` property, improving recall precision.


Spaced Repetition Memory Reinforcement

Implements spaced repetition by reinforcing memories used in recall or context loading, increasing their importance and recall count to combat decay.  `reinforceMemory` method in `MemoryDB` boosts importance and recall count. Triggered during memory recall (`hybridSearch`) and agent start. Affects decay calculation through `lastRecalledAt` and `recallCount` fields.


ReinforceMemory Method

A method within the MemoryDB class to increase memory importance and recall count based on usage. Increases importance by 15% of remaining room to 1.0 and increments recallCount.  Updates lastRecalledAt timestamp.


A/B Test for Memory Upgrade Proposals

A/B/C test setup to evaluate category-specific decay and spaced repetition, comparing old system (A), category decay + spaced repetition (B), and category decay only (C). Metrics: Recall Redundancy Rate, Long-Term Context Retention, Unnecessary Recall Cycles.


Context-Aware Memory Injection

Strategy for memory upgrade utilizing context-aware injection techniques.


Context-Aware Memory Injection Layer

A proposed system to improve memory injection by dynamically adapting to the conversation context, user state, and prompt structure, addressing the limitations of the current memory injection mechanism. It aims to improve precision, optimize injection placement, suppress irrelevant memories, and integrate real-time user state.


Dynamic Conversation Topic Modeling & Intent Detection

A mechanism to identify the current conversation topic and user intent from recent turns, improving memory retrieval relevance. It involves an auxiliary embedding call or keyword/phrase extraction heuristic.


Relevance Re-ranking and Suppression

A post-retrieval step that leverages the detected conversation context (topic, intent) to filter or re-order candidate memories before MMR, including a dynamic suppression mechanism for irrelevant memories.


Adaptive Injection Placement

Logic to determine optimal memory injection placement (prepend, append, or in-line) based on memory content and prompt structure, improving prompt flow and LLM performance.


Context-Aware Memory Injection Layer

A system designed to improve memory injection in the AIVA plugin by using dynamic context, relevance filtering, injection flexibility, and user state integration. It involves topic modeling, intent detection, relevance re-ranking, adaptive injection placement, and user state personalization.


A hybrid approach using embedding models to generate topic vectors and lightweight NLP to extract keywords and infer user intent. It aims to capture conversation context efficiently and inform query enhancement.


Relevance Re-ranking and Suppression

Filtering candidate memories after initial retrieval using the topic vector to eliminate irrelevant results, computing relevance scores (cosine similarity) and applying suppression thresholds.


Adaptive Injection Placement

Improving the agent's behavior to offer more injection options (prepend, append, inject) based on the prompt content and memory relevance, using if/else rules to determine the optimal placement strategy.


User State Integration

Personalizing recall by modifying candidate scores based on the user's current state, accessing user state from the API, and applying score boosts using a configuration file.


Context-Aware Memory Injection

Implements dynamic conversation topic modeling, intent detection, relevance re-ranking, and suppression to improve memory recall precision.


getCurrentConversationContext Function

Analyzes recent conversation history to generate a topic vector, extract keywords, and infer intent (question, task, reflection, unknown).


Relevance Re-ranking and Suppression

Re-ranks and suppresses candidate memories based on their relevance to the current conversation context using cosine similarity and intent mismatch penalties.


Adaptive Injection Placement Strategy

A strategy to inject memories into prompts based on prompt structure and retrieved memory content. Uses 'prepend', 'append', or 'inline' placement. Implemented with a `determineInjectionPlacement` function that considers prompt length and content (e.g., coding-related).


determineInjectionPlacement Function

Function that analyzes the prompt's structure and nature of the retrieved memories to determine the optimal placement for injecting memories into the context ('prepend' or 'append'). Rules include short prompts ending with '?' being appended and coding prompts being prepended.


User State Integration for Personalization

A strategy to personalize memory retrieval by mapping user states (e.g., 'task_domain') to score multipliers for specific memory categories. An `applyStateBoosts` function applies these multipliers to promote relevant memories.


applyStateBoosts Function

Function that applies score boosts to memory candidates based on the current user state and a configuration map (`CATEGORY_BOOSTS_BY_STATE`). Boosts are applied based on the user's `task_domain` and the category of the memory.


Context-Aware Memory Injection System

This system enhances the agent's memory recall by incorporating conversation context, user state, and diversification techniques to retrieve and inject relevant memories into the conversation.


Hybrid Search for Initial Retrieval

Combines vector-based similarity search (using blended prompt and topic vectors) with keyword-based BM25 search to retrieve candidate memories.


Personalization Boost using User State

Applies boosts to candidate memories based on the user's current state, allowing the agent to prioritize information that is most relevant to the individual user.


Adaptive Memory Injection Placement

Dynamically determines whether to prepend or append injected memories to the agent's context based on the current prompt and content.


Relevance Re-Ranking & Suppression

Re-ranks and suppresses candidate memories based on conversation context to filter out irrelevant information.


Memory Diversification with MMR

Applies Maximal Marginal Relevance (MMR) and category diversity to ensure a diverse set of memories is injected into the context.


Memory Compression and Hierarchical Storage Strategy

A strategy to address scaling and performance issues related to memory storage by implementing progressive summarization & hierarchical memory clusters and vector quantization for cold storage.


MemoryCluster

A meta-memory structure that groups related MemoryEntry instances and stores a compressed summary (text and vector) of its constituent memories, facilitating hierarchical memory organization.


Vector Quantization for Cold Storage

A technique to compress vector data for less frequently accessed memories designated for cold storage, reducing storage footprint and enabling efficient cold storage management.


Tiered Memory Storage with LanceDB Product Quantization

Strategy to reduce memory storage costs and improve retrieval efficiency by using hot and cold storage tiers in LanceDB. Cold storage utilizes Product Quantization for high compression.


Hierarchical Memory Clustering for Efficient Retrieval

Strategy to improve retrieval by grouping related memories into clusters with summaries, enabling a two-stage search process (cluster search then memory search).


memory-lancedb plugin refactoring

Refactor monolithic memory architecture into a multi-tiered, hierarchical system to address the scaling and performance issues.


Memory Clustering System

A two-stage memory system using LanceDB with `memory_clusters` table for high-level semantic indexing and a modified `memories` table with `clusterId` for individual entries. The system clusters memories based on semantic similarity, summarizes them with an LLM, and uses clusters to bias retrieval.


clusterAndSummarizeMemories Process

Background process that identifies candidate memories without a `clusterId`, finds similar neighbors, forms clusters if the size is adequate, generates summaries and key concepts using an LLM, and stores the cluster and updates member memories.


hybridSearch

Modified search process that first searches the `memory_clusters` table for relevant clusters and then searches the `memories` table, biased toward memories within those clusters.


Tiered Storage with Vector Quantization for Memory

Implements a tiered storage system using LanceDB with hot (full precision) and cold (quantized) tables. Older, less frequently used memories are compressed using Product Quantization (PQ) and moved to the cold table to reduce storage footprint and improve search latency.


MemoryDB with Hot/Cold Tables

Modified MemoryDB class uses two LanceDB tables: `memories_hot` for recent memories and `memories_cold` for archived, quantized memories. Methods like `store`, `update`, and `vectorSearch` are updated to work across both tables.


Memory Compression System

The memory compression system involves archiving old memories to cold storage, clustering similar memories, and implementing a two-stage search process (cluster -> memory) for intelligent retrieval. It uses hot and cold storage tiers.


Hot/Cold Memory Tiering

Strategy to improve memory retrieval performance by separating frequently accessed (hot) and infrequently accessed (cold) memories.  This involves archiving older memories to a separate, potentially slower, storage (cold) and optimizing search for the hot memories.


Clustered Memory Search

Strategy to improve the efficiency of memory retrieval using clustering and summarization of memories. The search uses a two-stage process: first identify relevant clusters, then search within those clusters.


Multi-Agent Shared Memory Protocol

A protocol for managing shared memory between a 'Queen AIVA' (master) and multiple 'SubAIVA' (worker) instances, addressing challenges like redundant learning, lack of aggregated behavioral patterns, inefficient capability propagation, and privacy concerns.


Privacy Partitioning and Anonymization

A strategy to identify and filter PII/sensitive data at the SubAIVA level, preventing its propagation to Queen AIVA, while still allowing for aggregate pattern extraction. Involves extending `shouldCapture` logic or introducing a `shouldPropagate` function, and aggregating/anonymizing behavioral patterns before sending to the Queen.


Federated Memory Training Loop

A system where SubAIVAs periodically contribute to a global model maintained by Queen AIVA, impacting `computeImportance`, `detectCategory`, or `CATEGORY_DECAY_PARAMS`.  Includes a conflict resolution mechanism to handle contradictory preferences or facts propagated by different SubAIVAs.


Multi-Agent Shared Memory Protocol

A two-tiered memory architecture with SubAIVAMemoryDB (Local/Edge) and QueenMemoryDB (Global/Core) facilitating federated learning. Uses privacy filtering, data aggregation, and asynchronous messaging.


SubAIVAMemoryDB

Local memory store for each SubAIVA, maintaining private, high-fidelity data. Responsible for initial data processing and privacy filtering before data propagation.


QueenMemoryDB

Centralized memory store managed by Queen AIVA, holding aggregated, anonymized insights, generalized skills, and global configuration parameters derived from SubAIVAs.


Memory Propagation with Privacy Levels

Implementation of privacy levels ('private', 'anonymizable', 'public') using a `shouldPropagate` function before sending data from SubAIVAs to the Queen. Anonymization performed via `aggregateAndAnonymize`.


Confidence-Weighted Voting Scheme

Conflict resolution mechanism within the federated training loop. If multiple SubAIVAs suggest different outputs, the one with the most support wins; conflicts are logged for review.


Shared Memory Architecture (SubAIVA & Queen)

Hierarchical memory structure using SubAIVAMemoryDB and QueenMemoryDB for raw and aggregated knowledge respectively. Includes communication protocol for upward and downward propagation of memory.


Asynchronous Memory Propagation

SubAIVAs propagate anonymized, aggregated learnings to Queen, while Queen broadcasts capability updates to SubAIVAs using asynchronous events to minimize latency.


MemoryEntry Enhancement

Extends MemoryEntry with privacyLevel and aggregationState to control propagation and lifecycle within multi-agent environment. Introduces AggregatedMemoryEntry for Queen.


Privacy Partitioning and Anonymization Strategy

Strategy for achieving 100% PII non-propagation through privacy level classification, anonymization, and aggregation of memory entries.  Leverages a privacyLevel field to categorize entries as private, public, or anonymizable. Anonymizable entries are aggregated and generalized before propagation to the Queen agent.


MemoryDB Privacy Classification System

System within MemoryDB responsible for classifying the privacy level of memory entries. It utilizes rule-based classification based on content, category, and PII regex patterns to determine if data is private, public, or anonymizable.


Aggregation and Anonymization Pipeline

Pipeline for aggregating and anonymizing memory entries flagged as 'pending_aggregation'. This process groups entries by category, generalizes content to remove specific PII, and creates a new aggregated entry for the Queen agent, reducing redundancy.


Federated Memory Training Loop

A distributed learning process where SubAIVAs capture local learnings, propagate aggregated data to a Queen, the Queen analyzes global patterns, and broadcasts capability updates back to the SubAIVAs, adapting local configurations.


The Queen's method for handling conflicting AggregatedMemoryEntry objects. It uses vector search to find similar entries, merges them based on confidence-weighted importance (source count), and updates the Queen's global knowledge base.


Tiered Search Strategy

A search system where SubAIVAs first search their local memory, and if insufficient, query the Queen for globally relevant information.  This prevents local context pollution and enables access to collective intelligence.


Batch Memory Propagation

Propagating memory updates on `agent_end` or a timer amortizes the cost of aggregation and network communication, preventing the Queen from being overwhelmed and improving scalability.


Surprise-Based Learning Integration in memory-lancedb

AIVA's memory-lancedb plugin incorporates a computeSurprise function to boost importance based on novelty. The goal is to integrate with Genesis Titan Memory for contextual surprise calculation, dynamic category reclassification, and axiom generation, using reinforcement learning for surprise-driven importance.


Contextual Surprise Calculation Enhancement

Augment computeSurprise with contextual distance calculation, considering semantic deviation from the conversation topic and weighting surprise based on contradiction or confirmation of existing high-importance memories. Introduce SURPRISE_CONTEXT_WEIGHT.


Dynamic Category Reclassification Process Design

Design an asynchronous process, reclassifySurprisingMemories, triggered by high-surprise memory captures. Examine nearby memories, leverage an LLM for alternative MemoryCategory labels, and implement a threshold for automatic reclassification with CATEGORY_RECLASSIFICATION_LOG and CATEGORY_RECLASSIFICATION_LLM_PROMPT_TEMPLATE.


Axiom Generation Integration Design

Propose a new MemoryDB method, flagForAxiomGeneration, to store a genesisAxiomFlag on surprising memories and send them to Genesis for axiom generation.


Intelligent Memory Management System

System to improve detection, understanding, and leveraging of surprising information by incorporating semantic context, contradiction detection, and prior expectations into memory management.


Contextual Surprise Calculation

Calculation of surprise score based on vector novelty, contextual deviation, and contradiction/confirmation checks with existing memories. It weights vector novelty and contextual deviation using SURPRISE_CONTEXT_WEIGHT.


Memory Reclassification

Dynamic reclassification of memories into predefined categories using an LLM, triggered by high surprise scores and incorporating memory text, context, and agent feedback.


Contextual Surprise Calculation in Enhanced MemoryDB

Implements a new surprise score calculation method that considers both vector novelty and conversational context. It also identifies contradictions with high-importance existing memories and boosts the surprise score accordingly.


Dynamic Category Reclassification

An adaptive mechanism using an LLM to correct and discover memory categories, improving the accuracy of memory categorization based on surprise scores, agent feedback, and semantically similar memories.


Genesis KG Axiom Generation Pipeline

A system for escalating high-surprise memories to a higher-level reasoning system (Genesis) for axiom generation, enabling the system to learn from its most profound surprises.


Reinforcement Learning for Surprise-Driven Memory Importance

Implements a feedback loop to reinforce the importance of useful surprising memories and decay irrelevant ones based on recall frequency and surprise score. Uses constants REINFORCEMENT_SURPRISE_MULTIPLIER, SURPRISE_DECAY_DAYS, and SURPRISE_IRRELEVANCE_DECAY_RATE.


Differential Decay of Irrelevant Surprise Memories

A background process that decays the importance of memories that were initially surprising but have not been recalled after a certain period (SURPRISE_DECAY_DAYS). It targets memories with high initialSurpriseScore and zero recallCount.


Memory Upgrade - Hybrid Retrieval Weight Optimization

Strategy for optimizing hybrid retrieval weights within the Memory Upgrade project.


Dynamic Hybrid Weights for Memory Retrieval

Implement dynamic adjustment of `HYBRID_WEIGHTS` based on query `MemoryCategory` and semantic intent, using keywords to boost weights for `vector`, `bm25`, `recency`, or `importance` scores within `MemoryDB.hybridSearch`.


Memory Retrieval Weighting System

Current memory retrieval in AIVA uses static weights for `vector`, `bm25`, `recency`, and `importance`. The system lacks dynamic adaptation based on conversation, user feedback, or memory characteristics, leading to suboptimal recall precision and relevance.


Hybrid Memory Retrieval Strategy

A strategy to improve AIVA's memory retrieval by dynamically adjusting hybrid weights based on context, category, and reinforcement learning. Includes three proposals: contextual weight adjustment, RL for adaptive weights, and category-specific base weights.


AdaptiveWeightsLearner

A reinforcement learning system to adaptively adjust hybrid weights based on positive and negative feedback signals. Uses a multi-armed bandit approach and persists learned weights.


A/B Test for Dynamic Weights

A/B test to compare dynamic weights (Group B) with static weights (Group A) using the 'Memory Utilization Score' as a metric. Hypothesis: Group B will show a statistically significant increase in the memory utilization score compared to Group A.


Contextual & Category-Specific Hybrid Weight Adjustment

A layered weighting system that adjusts hybrid search weights based on context, category, and keywords, using default weights, category-specific base weights, and dynamic keyword-based adjustments.  It enhances the configuration in `enhanced_config.ts` and implements dynamic weight logic in `enhanced_index.ts`.


enhanced_config.ts Weight Configuration

Configuration file that defines default hybrid weights, category-specific base weights, and dynamic keyword-based adjustments for hybrid search.


enhanced_index.ts Dynamic Weight Logic

Implements the logic to calculate effective hybrid search weights based on query context, category, and keywords, normalizing the weights before applying them in the hybrid search.


AdaptiveWeightsLearner

A system that dynamically adjusts the weights of different memory retrieval components (vector, BM25, recency, importance) based on feedback signals. It persists weights to disk and normalizes them after updates.


Feedback Instrumentation for Memory Recall

Capturing implicit positive and negative feedback signals related to memory recall to improve the quality and relevance of memories used by the LLM. Examples include inclusion, citation, pruning, irrelevance, and correction.


Auto-Capture Trigger Pattern Expansion

Expansion of the auto-capture trigger pattern to enhance memory upgrade capabilities.


AIVA Memory System Categorization and Importance Scoring

AIVA's memory system relies on regex patterns and keyword matching for category detection and assigns importance scores to captured memories. It uses MEMORY_TRIGGERS, detectCategory, and computeImportance, with categories including preference, fact, decision, entity, event, contact, location, skill, goal, and other.


Expand Implicit Pattern Recognition in Memory System

Introduce new regex patterns and SLM-based classifiers within `detectCategory` to capture implicit user values, emotional states, recurring frustrations, health/wellness, financial sentiment, and relationship dynamics to increase the capture rate of high-value personal context by 25%.


Fine-Grained Categorization for Complex Personal Data

Add at least 5-7 new, distinct `MemoryCategory` types (e.g. 'health/medical', 'financial', 'relationship') to enhance the precision of `_calculateDynamicScores` and `hybridSearch` by allowing boosted `freshnessBoost` or lower `minimumImportance` thresholds post-event for specific categories.


Hybrid Approach for Implicit Pattern Recognition

Leverage regex for fast pattern recognition and integrate a small language model (SLM) for more nuanced context-aware recognition.


Nuanced Importance Scoring System

Modify computeImportance function. Assign higher baseline scores (0.85-0.9) to high-impact categories and use a multiplier system.


Fine-Grained Categorization

Define new memory categories (health, financial, relationship, etc.) and update MEMORY_TRIGGERS with regex patterns.


Code-Switching Detection Integration

Integrate a library like `franc` to identify language shifts and potentially boost memory importance for code-switching instances.


Enhanced Memory Categorization System

System to expand implicit pattern recognition by creating a new, more nuanced set of `MEMORY_CATEGORIES` and corresponding `MEMORY_TRIGGERS`. Utilizes expanded regex patterns and a hook for an optional, asynchronous SLM-based classifier for complex cases.


Prioritize specific regex triggers over general triggers

The implementation will combine existing triggers with the new implicit triggers, ensuring that the new, more specific triggers are prioritized during category detection.


A/B testing for memory categorization improvements

Deploy the enhanced `detectCategory` system to 50% of users in an A/B test to measure improvements in memory categorization accuracy.


Category-Based Importance Scoring

Prioritize new, high-value implicit information by assigning higher baseline importance scores to new, high-impact categories using a lookup map. This moves away from a one-size-fits-all score for non-explicit memories.


Language Code-Switching Detection

Integrate a client-side language detection library to analyze user input. If code-switching is detected, add metadata and boost the memory's importance, as it often signals personal significance.


A Typescript map defining baseline importance scores for different memory categories such as 'health', 'financial', etc.


Memory Consolidation Algorithm Refinement

Refine the memory consolidation algorithm to address scaling and quality issues. The current algorithm performs a full scan of all memories. Improvements include incremental consolidation, smarter merge strategy, adaptive staleness, and contradiction-aware importance adjustment.


Prune Score Calculation for Adaptive Staleness

The system should calculate a `pruneScore` that factors in `ageDays`, `importance`, `minimumImportance`, `decayFunction`, and `eventCompletionDecayMultiplier`, making `STALENESS_DAYS` and `STALENESS_IMPORTANCE_THRESHOLD` dynamic per category. This addresses the issue of arbitrary and inflexible staleness thresholds.


Synthesize-and-Update Merge Strategy

Implement a `synthesize-and-update` strategy for merging memories within a defined similarity range. Generate a new combined `text` field that integrates information from both memories, preserving nuance and generating a new combined vector.


Memory Consolidation Strategy using ANN and LLM

A multi-stage memory consolidation process using Approximate Nearest Neighbor (ANN) search for efficient merging, LLM for intelligent synthesis of memories, and adaptive pruning based on dynamic scoring.


Prune Score Calculation System

Defines a `pruneScore` calculation that dynamically incorporates category-specific decay rates, event completion status, and overall memory importance to determine memory staleness.


Contradiction-Aware Memory Decay

Implement a mechanism to flag contradictory memories and decay their importance over time, reducing the likelihood of recalling outdated or incorrect information.


Incremental ANN-Powered Memory Consolidation

Leverages ANN search (LanceDB) to find nearest neighbor memories for potential merging, focusing on new or high-churn memories. Reduces complexity from O(N²) to O(M * k * log N).


MemoryDB Class - Enhanced Index

Implements memory consolidation logic, including finding merge candidates, merging memories (direct and synthesized), and pruning stale memories.  Tracks metrics like candidates checked, merges performed, and execution time.


Merge Thresholds (High & Low)

Defines thresholds for cosine similarity to determine merge strategy. MERGE_THRESHOLD_HIGH triggers direct merging, MERGE_THRESHOLD_LOW triggers intelligent synthesis.


Intelligent, Synthesis-Based Memory Merging

Leverages an LLM to combine related memories into a single, coherent, and comprehensive memory, preserving unique facts and nuances while eliminating redundancy. Replaces the lossy 'keep the longer one' strategy. Merged memories are tracked via the 'mergedFrom' metadata field.


MemoryDB enhanced_index.ts LLM merging

A method, synthesizeAndMergeMemories, within the MemoryDB class that invokes an LLM to merge two memories, update the database and handle error cases.


Adaptive Staleness Pruning Strategy

Replaces inflexible, global pruning thresholds with a dynamic, per-category scoring system based on memory importance, age, and category-specific decay parameters. This strategy aims to improve the accuracy and quality of memory pruning.


Prune Score Calculation System

A system that calculates a prune score for each memory entry based on its category, importance, and age. The score is then compared to a global threshold to determine if the memory should be pruned. Includes exponential and linear decay functions, and acceleration for completed event memories.


Contradiction-Aware Importance Decay Strategy

Introduces a mechanism to rapidly de-prioritize outdated or incorrect information by flagging contradicted memories and applying an exponential decay to their importance based on the time since they were contradicted.


Cross-Session Memory Lifecycle Protocol

A comprehensive protocol for managing memory across session boundaries, including session boundary detection, pre-session load, post-session flush & consolidation, and long-term archival with priority boosting. Aims to address cold starts, inefficient consolidation, and lack of topic shift detection.


SessionContext Management

Creation of a `SessionContext` interface (`sessionId`, `lastActivity`, `mainTopicVector`, `sessionMemories`) and a mechanism to manage it across `before_agent_start` and `agent_end` events using `OpenClawPluginApi` or a new internal service.


SessionManager Service

Central service to track session state, crucial for managing conversational context across agent runs and enabling cross-session memory transfer. Manages sessionId, lastActivity, mainTopicVector, and sessionMemories.


Topic-Aware Memory Recall

Strategy to detect shifts in conversation topics by comparing the current prompt's vector with the previous session's mainTopicVector, adjusting memory recall based on similarity.


Session-Specific Memory Consolidation

Lightweight automated cleanup at the end of each session to reduce memory redundancy, focusing only on memories within the just-ended session.


Dynamic Priority Boosting

Temporarily boosting the importance of memories from the most recent session to ensure continuity, modifying scoring logic to interact gracefully with decay mechanisms.


MemoryEntry Schema Modification

Add `sessionBoostExpiry` and `sessionRecallCount` to the MemoryEntry schema, allowing for dynamic prioritization and temporary boosting of memory importance based on session context.


SessionManager

Centralized service to persist and manage session state across hook calls. Responsible for session lifecycle (start, end, timeout) and maintaining session context (ID, activity, topic vector, memory IDs).


Session Context and MemoryEntry Extensions

Extending the SessionContext and MemoryEntry interfaces to include session-specific data such as sessionId, newMemoryIds, sessionRecallCount and sessionBoostExpiry.


Session-Aware Memory Recall

Modifies memory recall to consider session context. Boosts memories from the preceding session if the current session is a continuation. Implements session consolidation to merge near-duplicate memories at the end of each session, improving memory quality and preventing redundancy.


Session Consolidation

Performs lightweight consolidation at the end of each agent run by merging near-duplicate memories within the just-ended session. This aims to maintain memory quality by removing redundant information.


Session Manager

A system for managing session context including starting, ending, and updating sessions. The session manager keeps track of the current session and the main topic vector associated with each session.


Memory Reinforcement with Session-Specific Boost

System for boosting memory recall probability for memories created and recalled within the same session using session recall count and session boost expiry.


Hybrid Search with Session Boost

The `hybridSearch` function will accept `sessionContext` and apply `SESSION_BOOST_FACTOR` based on both `isContinuation` and `sessionBoostExpiry` to enhance recall of session-relevant memories.


Episodic vs Semantic Memory Separation

Deep Think initiative focused on separating episodic and semantic memory within the AI system.


AIVA Memory System

AIVA's memory system, currently implemented with a single LanceDB table, is being redesigned to use a dual-store architecture (`episodic_memories` and `semantic_memories`) to improve memory management and retrieval performance. This involves routing queries and memories to appropriate stores and implementing tailored decay and consolidation strategies.


Dual-Store Memory Architecture

Proposed strategy to enhance AIVA's memory management by separating episodic and semantic memories into distinct LanceDB tables. This allows for optimized decay parameters, retrieval mechanisms, and consolidation strategies tailored to each memory type.


Implement Episodic Decay Parameters

Decision to introduce `EpisodicDecayParams` in `enhanced_config.ts`, including a `rapidInitialDecayFactor` and `episodicRetentionDays`, for more granular control over episodic memory decay.


Query Routing Function

Design a new `_routeQueryToStores` function that intelligently routes or fuses queries to the episodic or semantic memory stores based on query characteristics.


Dual-Store MemoryDB Architecture

A MemoryDB architecture with two distinct stores: `episodic_memories` and `semantic_memories`. The `store` method routes entries based on `entry.category`, e.g., 'event' and 'decision' to episodic, others to semantic. The tables include episodic decay, semantic structure, retrieval, and consolidation.


Enhanced Episodic Decay Mechanism

Introduce `EpisodicDecayParams` in the config file, extending `DecayParams` as a union type. Within `_calculateDynamicScores`, apply `rapidInitialDecayFactor` during a 'rapid decay window' for memories with `EpisodicDecayParams`. Uses a modified half-life formula.


Semantic Graph Integration via Relational Hops

Update `MemoryEntry` with `relatedMemoryIds?: string[]`.  During storage of semantic memories, perform a similarity search, populate `relatedMemoryIds`. During retrieval, leverage the 'relational hop' via `_routeAndFuseSearch` (or `hybridSearch`) by performing a secondary query for related IDs and fusing results.


Dynamic Query Routing and Fusion

The `_routeAndFuseSearch` method orchestrates query routing. `_determineQueryIntent` employs keyword heuristics to decide the primary store (episodic vs. semantic).  Call `hybridSearch` on both stores. `fuseResults` combines results, normalizing scores, applying boosts based on query intent, and re-ranking.


Granular Consolidation Method

Modify `consolidateMemories` by adding an optional `targetStore` parameter ('episodic', 'semantic', or 'all'). Branch based on `targetStore`, applying consolidation logic only to the specified store (e.g., prune older episodic memories if `targetStore` is 'episodic').


Dual-Store MemoryDB Architecture

Splits the single `memories` table into `episodic_memories` and `semantic_memories` to optimize memory management and retrieval. Uses LanceDB for storage and provides methods for storing, retrieving, deleting, and updating memories in the appropriate store based on the memory category. Includes mechanisms for semantic linking and enhanced episodic decay.


Enhanced Episodic Decay

Implements a rapid initial decay mechanism for episodic memories using `rapidInitialDecayFactor` and `rapidDecayWindowMs`.  This allows for modeling faster forgetting of short-term memories while maintaining longer-term semantic memory.


Semantic Graph Integration

Enrich semantic memories with relational links to enable multi-hop queries using the `enhanced_schema.ts` extension and modification of `hybridSearch` with relational hop.


Dynamic Query Routing and Fusion

Central retrieval system that directs traffic and merges results intelligently by intent detection and score boosting.


Hybrid Search with Relational Hop Implementation

Implements a hybrid search that fetches related memories based on their IDs and fuses them with initial search results, using a decay score to rank contextually relevant hopped memories.


Intent-Based Query Routing

Routes queries to either episodic or semantic memory stores based on keyword intent detection and fuses results with intent-based score boosting.


Dual-Store Memory Architecture

A memory system with separate episodic and semantic stores, each with specific retention and consolidation strategies, designed for improved precision, latency, and efficiency compared to a monolithic store.


Granular Consolidation

Store-specific consolidation logic, allowing for different retention and merging strategies for episodic and semantic memories. Episodic memory uses decay model while semantic memory uses staleness and importance logic.


Similarity-based Consolidation

Consolidation that merges similar memories within each store, improving efficiency and reducing redundancy. Implemented using O(N^2) or approximate nearest neighbor logic.


Semantic Linking Process

A batch job to find and create relationships between memories in the semantic store. This improves the retrieval of relevant information for complex queries.


Episodic Memory Decay Model

Aggressive pruning of episodic memories based on retention policies defined by `episodicRetentionDays` within `CATEGORY_DECAY_PARAMS`.


Memory Importance Decay Curve Engineering

Engineering memory importance decay curves within the context of a memory upgrade. This involves understanding and designing how the importance of different memories degrades over time to optimize memory allocation and resource utilization.


AIVA memory importance decay system

AIVA's `memory-lancedb` plugin implements memory importance decay and recency scoring. It uses category-specific DecayParams, recallResilience, and FRESHNESS_BOOST to manage memory decay. It has limitations in spaced repetition, contextual decay, event-based decay, and negative reinforcement.


Improve memory recall precision with adaptive spacing

Improve the P90 precision of `memory_recall` by 5% for memories with `recallCount > 3` by introducing an adaptive spaced repetition factor (easinessFactor) based on recall scores.


Interference-aware decay modifier

Implement a mechanism to assess "semantic interference" or "relatedness density" during memory capture, storing a count of related memories and using it to reduce the effectiveDecayRate for memories with higher semanticDensity.


AIVA Memory Decay System

The current memory decay in AIVA has weaknesses. The goal is to significantly improve it by addressing the shortcomings of the existing `memory-lancedb` plugin with adaptive spaced repetition, interference-aware decay, and refined event-based decay.


Adaptive Spaced Repetition for Memory Decay

Incorporate spaced repetition principles, mirroring the logic of algorithms like SM-2, using an 'easiness factor' updated based on recall success. Easier memories are reviewed less frequently, harder ones more often. Implemented by adjusting the decay calculation based on `easinessFactor`.


Interference-Aware Decay

Memories that are semantically related (high density) should reinforce each other and decay more slowly. The `semanticDensity` of a memory is determined by the number of similar memories.


Adaptive Spaced Repetition (SM-2 Inspired)

Replaces the static `recallResilience` with a dynamic `easinessFactor` that adapts based on recall success. The `easinessFactor` reduces the perceived age of the memory. It is updated based on the quality of the recall, proxied by `finalScore`.


Interference-Aware Decay Modifier

Introduces a `semanticDensity` field to each memory, slowing decay for memories that are part of a dense, coherent cluster of information. The `semanticDensity` is used to create a decay reduction factor.


A/B Test for Spaced Repetition

A/B test comparing the existing system with static `recallResilience` (Group A) to the new system with adaptive `easinessFactor` (Group B). Metric: P90 `finalScore` for memories retrieved via `memory_recall`. Cohort: Memories with `recallCount > 3` and `createdAt` more than 30 days ago. Target: A 5% increase in the P90 `finalScore` for Group B compared to Group A.


Event-Based Memory Decay Curve with Multi-Phase Decay

Implements a three-phase decay curve (pre-event, post-event, long-term) for event memories, using configured decay rates and durations to refine memory importance over time. Configurable via `enhanced_config.ts`.


Negative Reinforcement for Memory Retrieval

A mechanism to penalize memories that are frequently retrieved with high `vectorScore` but consistently low `finalScore`, indicating 'false positives'. Implemented by tracking `irrelevantRetrievalCount` and applying an aggressive decay multiplier.


Context-Aware Memory Injection Strategy

A strategy for upgrading memory by injecting information based on the current context.


Context-Aware Memory Injection Layer

A system designed to improve memory injection by dynamically re-ranking memories based on conversation context, suppressing irrelevant memories, boosting relevant ones, and optimizing injection placement to enhance agent comprehension and response quality.


postRetrievalReRank Function

A function `postRetrievalReRank(candidates: HybridSearchResult[], conversationHistory: string[], currentPrompt: string): HybridSearchResult[]` designed to dynamically re-rank memory candidates based on the semantic relationship between each candidate memory and the current conversation turn or detected user intent.


determineContextualFactors Function

A function `determineContextualFactors(conversationHistory: string[], timeOfDay: Date): { topic: string; sentiment: string; taskDomain: string }` to determine the current conversation topic, user sentiment, and task domain from the conversation history.


applyContextualModifiers Function

A function `applyContextualModifiers(memory: HybridSearchResult, contextualFactors: { topic: string; sentiment: string; taskDomain: string }): number` that returns a multiplier to the memory's final score based on contextual factors, enabling suppression and boosting of memory relevance.


Context-Aware Memory Injection Layer

A strategy to enhance AIVA's memory system by making it contextually aware, improving retrieval precision, token efficiency, and agent response quality. It moves from reactive to proactive.


A flexible configuration structure (defined in enhanced_config.ts) as an array of rules that allow for targeted suppression or boosting of memories based on context. Conditions and actions make it extensible.


determineContextualFactors Function

A function (in enhanced_index.ts) to analyze context, currently a placeholder using keyword matching. It aims to evolve to a more advanced implementation such as LLM call.


postRetrievalReRank Function

A function (in enhanced_index.ts) to dynamically re-rank memories using a contextualRelevanceScore. Combines original relevance with context (keyword overlap from prompt and history).


applyContextualModifiers Function

A function (in enhanced_index.ts) that applies multipliers to memory scores based on the CONTEXTUAL_MODIFIERS_CONFIG rules. If conditions match, score is multiplied.


formatAndPlaceMemoriesContext Function

A formatter that returns a structured object { prependContext, appendContext } for flexible memory placement based on memory category and other factors.


Optimized Injection Placement

Strategy for optimizing memory injection by placing different memory types (e.g., decisions, preferences) at specific locations (prepend, append) in the context.


Context-Aware Memory Injection Layer

A system to modify memory scores based on conversational context using configurable rules, influencing memory re-ranking and injection.


Contextual Memory Score Modification Strategy

Apply a multiplier to a memory's final score based on a set of configurable rules that match the current conversational context (topic, sentiment, task domain, memory category).  The first matching rule's multiplier is applied.


Determine Conversational Context Function

A function `determineContextualFactors` that analyzes recent conversation history to derive the current conversational context (topic, sentiment, taskDomain). A placeholder implementation is provided which uses keyword based detection.


Context-Aware Memory Injection for Improved Agent Performance

The strategy involves re-ranking memories based on contextual relevance, applying contextual modifiers to adjust memory scores, and strategically placing memories in context to improve agent response quality and token efficiency.


postRetrievalReRank function

Re-scores memory candidates based on keyword overlap with the immediate turn to optimize for turn-level relevance, preventing injection of stale memories.


applyContextualModifiers function with CONTEXTUAL_MODIFIERS_CONFIG

Applies a multiplier to memory scores based on relevance to the current taskDomain, reducing the likelihood of irrelevant memories being injected.


formatAndPlaceMemoriesContext function

Places memories into prependContext (high-priority instructions) or appendContext (useful background) to improve agent response relevance and avoid distraction.


Memory Compression for Long-Term Storage

Memory compression as a strategy for extending the capacity of long-term storage solutions within the Deep Think system.


Memory Compression and Hierarchical Storage Strategy

A comprehensive strategy to compress memory entries and implement hierarchical storage to reduce storage footprint, improve retrieval latency, and optimize operational costs.  It includes progressive summarization of older memories, vector quantization for embeddings, and hierarchical memory clustering into meta-memories with a cold-storage tier for infrequently accessed data.


Progressive Summarization for Memory Entries

Implement a strategy to summarize older, less important memory entries to reduce storage and I/O costs. This involves adding a `summarizedText` field to `MemoryEntry`, defining a summarization process (heuristic or LLM-based), and triggering summarization based on importance and age.  `hybridSearch` should utilize `summarizedText` and re-hydrate the full text only when needed.


Vector Quantization for Memory Embeddings

Apply vector quantization (VQ) to `MemoryEntry.vector` embeddings to reduce storage costs.  Choose a VQ method (Scalar or Product Quantization), define the vector field as `number[] | Uint8Array` (or similar), and detail the quantization/de-quantization process in `db.store` and `_calculateDynamicScores`.  Quantize vectors based on memory age/importance.


Hierarchical Memory Clustering (Meta-Memories)

Implement hierarchical clustering of related memories into 'meta-memories' to improve search precision and reduce latency. Define a `MetaMemory` type, implement a clustering algorithm (e.g., k-means), and modify retrieval to first retrieve relevant meta-memories before drilling down to individual memories.


Cold Storage Tier for Infrequently Accessed Memories

Introduce a cold-storage tier (e.g., S3-backed LanceDB) for infrequently accessed `MemoryEntry` objects to reduce hot storage costs.  Move memories with low importance, high age, and infrequent recall to cold storage, re-hydrating them on demand.


Memory Compression & Hierarchical Storage for memory-lancedb

A multi-pronged strategy to evolve the AIVA memory system into a tiered, efficient, and scalable architecture by cold storage, progressive summarization, vector quantization, and hierarchical clustering.


Progressive Summarization of Memory

Summarize aging, less important memories using LLMs (with heuristic fallback) to reduce text storage. Stored in `summarizedText` field, triggered in `memory_consolidate`.


Vector Quantization (VQ) using LanceDB

Using LanceDB's Product Quantization (PQ) to reduce vector storage for aging memories, triggered during `consolidate`.  LanceDB handles quantization automatically.


Hierarchical Memory Clustering with HDBSCAN

Clustering the memory space into 'meta-memories' using HDBSCAN for efficient two-stage search. A `MetaMemory` structure contains clustered memory IDs.


Cold-Storage Tier using separate LanceDB instance

Migrating old, less relevant memories to a cheaper, slower cold-storage tier managed by a separate LanceDB instance, configured using `coldDbPath` in `MemoryConfig`.


Progressive Summarization for Memory Compression

Summarize older, less important memories using OpenAI's Chat Completions API, with a fallback to extractive summarization if the API is unavailable or fails.  The `summarizedText` field is used in `hybridSearch` and `listAll`, while `memory_recall(id)` retrieves the full original text.


MemoryDB Class for Memory Management

The `MemoryDB` class manages memory storage and retrieval, including progressive summarization and vector quantization. It uses LanceDB for efficient storage and search, with methods for initialization, consolidation, summarization and vector quantization.


Vector Quantization (VQ) for Embedding Compression

Use LanceDB's Product Quantization (PQ) to reduce storage size of vector embeddings. The application code still deals with number[] arrays, and LanceDB handles the quantization/de-quantization transparently during writes and reads. Index rebuild/compaction is triggered as part of the `consolidate` function.


Introduces a `meta_memories` table for clustering similar memories.  Clustering is done using HDBSCAN. `hybridSearch` is rewritten to first search `meta_memories`, then filter the main `memories` table, drastically reducing the search space.


Cold-Storage Tier

Implements a two-tiered memory system with a hot (fast) and cold (slower, cheaper) storage.  Old, unimportant memories are moved to the cold store. `get(id)` checks both tiers, and `rehydrate(id)` moves memories from cold to hot.


Multi-Stage Retrieval

`hybridSearch` function is rewritten to first search the `meta_memories` table, then filter the main `memories` table. Reduces search space and computational cost.


Memory Consolidation Pipeline

Comprehensive workflow orchestrating memory migration to cold storage, progressive summarization, compaction using Vector Quantization (VQ), and hierarchical clustering to optimize memory storage and retrieval.


Hierarchical Search

A two-stage retrieval logic system that leverages clustered memories and meta-memory index to improve search efficiency and accuracy.


Multi-Agent Shared Memory Protocols

Exploration of multi-agent shared memory protocols to enhance coordination and collaboration among agents.


AIVA Federated Memory Protocol

A system for sharing knowledge between a Queen AIVA and SubAIVA agents while preserving privacy. Includes mechanisms for aggregating behavioral patterns upwards, propagating capabilities downwards, and resolving conflicting preferences.


Upward Aggregation of Behavioral Patterns (SubAIVA to Queen)

SubAIVA agents aggregate anonymized insights and report them to the Queen AIVA. This includes anonymized preference trends (using vector centroids) and "surprise" event reporting (generalized summaries of novel events).


MemoryAggregator Class

A class within `enhanced_index.ts` or `federated_memory.ts` responsible for periodically generating anonymized, aggregated insights from a SubAIVA's local MemoryDB. Aggregates preference trends and surprise events.


Anonymized Preference Trends using Vector Centroids

SubAIVAs aggregate changes in preference category memories by computing vector centroids for clusters of preference memories and reporting these centroids along with their frequency and average importance decay rates.


"Surprise" Event Reporting

SubAIVAs report instances where computeSurprise yields a SURPRISE_BOOST_MAX value, along with the category and a generalized summary of the novel text, but without the original text or vector.


Federated Memory Protocol

A multi-agent memory system for the AIVA, enabling learning, privacy, and conflict resolution between the Queen and SubAIVAs by upward aggregation, downward propagation, and collaborative intelligence.


Upward Aggregation (Queen Learns)

The Queen learns from the fleet by aggregating preference trends using vector clustering and reporting surprise events using keyword extraction. This protects privacy by incorporating differential privacy and transmitting only summarized insights.


Downward Propagation (Queen Updates SubAIVAs)

The Queen updates SubAIVAs using a GlobalMemoryConfig with versioning and merge strategies. Security is enforced through authentication and a fallback/retry mechanism.


Federated Training and Conflict Resolution

Collaborative intelligence using federated learning for model updates and weighted consensus for resolving conflicting preferences based on user base size and preference strength.


MemoryAggregator Class

The MemoryAggregator class in federated_memory.ts aggregates memory data from MemoryDB, clustering preferences and reporting surprise events in an anonymized manner. It uses k-means clustering for preference trends and keyword extraction for surprise events.


Upward Aggregation of Behavioral Patterns

Mechanism for aggregating individual agent (SubAIVA) behavioral patterns to a central 'Queen' agent by clustering preferences and reporting surprise events.


MemorySyncService for Global Configuration Updates

Defines a MemorySyncService that polls the Queen AIVA for global memory configuration updates and applies them to the local MemoryDB instance, including category decay overrides, new memory triggers, and reinforcement boost factors.


Downward Propagation of Global Capabilities

The strategy of propagating global memory configurations from the Queen AIVA to SubAIVAs, ensuring consistency and allowing for fine-tuning of memory parameters fleet-wide.


Using versioning for global memory configuration

Employ a versioning scheme (e.g., timestamp or hash) for the `GlobalMemoryConfig` to allow SubAIVAs to determine if they need to sync with the Queen AIVA.


Federated Memory Training Loop with Conflict Resolution

A closed-loop learning mechanism employing Federated Averaging (FedAvg) for model training across SubAIVAs and a weighted consensus model for resolving conflicting preference information. The Queen maintains a global model and updates it based on anonymized weight updates from SubAIVAs.


Federated Averaging (FedAvg) Implementation

The system implements FedAvg by distributing global model weights, training locally on SubAIVAs without data exposure, aggregating weight updates, and updating the global model. The Queen orchestrates the process.


Weighted Consensus Model for Preference Conflict Resolution

The Queen resolves conflicts in preference reports from SubAIVAs by weighting centroid vectors by member count and average importance. It maintains multiple dominant preference vectors mapped to different contexts, identified by SubAIVA metadata.


Surprise-Based Learning Integration in Memory System

Integrates surprise-based learning by introducing dynamic surprise score refinement and decay, surprise-triggered category reclassification, Genesis KG axiom generation, and reinforcement of surprise-based utility in AIVA's memory system.


Implement decaying surpriseScore in MemoryEntry

Introduce a new `surpriseScore` field in `MemoryEntry` with a decay mechanism (decay rate similar to `IMPORTANCE_DECAY_PER_DAY` but with `SURPRISE_DECAY_HALF_LIFE_DAYS` of 14 days).


Dynamic Category Reclassification

Implement a mechanism to trigger category reclassification based on sustained high surprise and cross-category retrieval patterns within the `agent_end` hook.


Genesis Axiom Suggestion Hook Integration

Connect high-surprise memories (surpriseScore > 0.7 and importance > 0.8) to a `api.on('genesis_axiom_suggestion')` hook to inform and update Genesis's broader knowledge graph, occurring for 1-5 memories per agent_end cycle where new memories are stored.


Dynamic Surprise Mechanism in memory-lancedb plugin

The core change is the introduction of a dynamic surprise mechanism in the `memory-lancedb` plugin, replacing the static importance boost with a decaying `surpriseScore` field in the `MemoryEntry` data model. This score will be influenced by decay, reinforcement, and used in memory retrieval.


Surprise-Triggered Reclassification via detectCategory

High surprise scores from frequently recalled memories should trigger re-evaluation of their categories using the `detectCategory` function in the context of the user's current query. This will happen in `agent_end` hook. This aims to reduce miscategorization.


Integration with Genesis KG via genesis_axiom_suggestion

A mechanism will be added to trigger the `api.on('genesis_axiom_suggestion')` hook when a memory's `surpriseScore` exceeds 0.7 *and* its `importance` exceeds 0.8. This aims to link surprising memories to axiom generation in the Genesis KG, limited to 1-5 memories per cycle.


Reinforce Surprise during memory recall

Modify the reinforcement logic in `reinforceMemoryUtility` to boost the `surpriseScore` (in addition to `importance`) when a recalled memory is actually *used* in generating the final response. Boost will be capped at `SURPRISE_BOOST_MAX_REINFORCEMENT` (0.1).


Surprise-Based Memory Scoring System

A system for dynamically scoring memories based on novelty and unexpectedness.  It uses a surprise score that decays over time and can be boosted when a surprising memory is proven useful.  It includes configuration constants, data model updates to track surprise, and modifications to the memory capture and retrieval logic.


Dynamic Hybrid Retrieval Weights

Implement dynamic adjustment of hybrid retrieval weights (vector, bm25, recency, importance) based on query context, memory category, and retrieval feedback to improve memory recall precision and relevance in Genesis.


Context-Aware Dynamic Weighting

A mechanism to adjust HYBRID_WEIGHTS dynamically based on query, conversation history, and user intent. Leverages LLM-based classifier, rule-based system, or combination. Function `getDynamicHybridWeights(query: string, conversationHistory: Message[]): HybridWeights` is introduced.


Category-Specific Weight Overrides

HYBRID_WEIGHTS configured per MemoryCategory, similar to CATEGORY_DECAY_PARAMS. Constant `CATEGORY_HYBRID_WEIGHTS: Record<MemoryCategory, typeof HYBRID_WEIGHTS>` defined in `enhanced_config.ts`.


Learned Weights from Retrieval Feedback (Reinforcement Learning)

A feedback loop where successful memory_recall and autoRecall events contribute to adjusting HYBRID_WEIGHTS. Considers gradient-descent or multi-armed bandit approach. Tracks weight component contribution to relevance.


Adaptive Hybrid Search Weights for Memory Retrieval

Implements dynamic weights for hybrid search in memory retrieval, adapting to query intent, memory categories, and retrieval feedback. This includes context-aware weighting, category-specific overrides, and learning from retrieval feedback.


Context-Aware Dynamic Weighting System

Uses a rule-based system and an LLM to classify query intent and adjust hybrid search weights. It defines QueryIntent enum and IntentWeights to map query intents to hybrid weights.


Category-Specific Weight Override System

Introduces per-category weights to refine retrieval. It uses a CATEGORY_HYBRID_WEIGHTS constant and detects query categories to apply appropriate weights, improving relevance for specific memory types.


Learned Weights from Retrieval Feedback System

A long-term system for learning from reinforceMemory signals. Uses a gradient-descent approach to update hybrid weights based on retrieval success, logging data and using A/B testing to validate improvements.


Context-Aware Dynamic Weighting for Hybrid Search

Introduces a dynamic weighting mechanism for hybrid search (vector, BM25, recency, importance) based on query intent. Uses a rule-based intent classifier with potential LLM extension. Implemented in `getDynamicHybridWeights` and integrated into `MemoryDB.hybridSearch`.


Intent Classification using Keyword Rules

Implements initial intent classification using a set of keyword-based rules (INTENT_RULES) for speed and efficiency. Fallbacks to default weights if no rules match. Future enhancement involves using an LLM for more complex classification.


Hybrid Search Weighting Strategy

Defines a strategy for dynamically adjusting hybrid search weights based on query intent. Different query intents (FACTUAL_RECALL, EVENT_REMINDER, PREFERENCE_QUERY, RELATIONAL_QUERY, GENERAL_CONVERSATION) have corresponding weight profiles (INTENT_WEIGHTS) defined in `enhanced_config.ts`.


Category-Specific Hybrid Weight Overrides

Implements category-specific weight overrides (CATEGORY_HYBRID_WEIGHTS) in hybrid search to allow fine-grained control over retrieval based on memory category.  Uses a per-candidate approach for scoring during the final scoring loop.


Hybrid Search Weighting Fallback

Prioritizes category-specific hybrid weight overrides.  If no category-specific weight exists, gracefully falls back to intent-based weights for hybrid search.


WeightsManager

A service/class to manage and persist hybrid search weights, allowing for online learning via reinforcement feedback. It loads, saves, gets, and updates weights based on query intent and reinforced memory scores, using a learning rate to adjust weights incrementally.


Reinforcement Learning for Hybrid Weights

Implement a feedback loop using `db.reinforceMemory` events to adjust hybrid search weights. Capture full score breakdowns for reinforced memories and adjust weights (vector, bm25, recency, importance) based on their contribution to the final score. A/B testing to compare performance against static weights.


Expand Memory Triggers for Implicit Personal Information

Expand the `MEMORY_TRIGGERS` regex patterns and the auto-capture mechanism in AIVA's `memory-lancedb` plugin to better capture implicit, nuanced personal information like health facts, financial sentiment, relationship dynamics, cultural preferences, and recurring frustrations.  This includes contextual trigger activation and semantic inference for importance and category assignment.


Contextual Trigger Activation Mechanism

A system to dynamically adjust the sensitivity of `MEMORY_TRIGGERS` based on recent conversational turns and identified user intent.  This involves a `contextualCaptureScore` function to assess relevance for implicit capture using factors like keyword proximity, sentiment analysis, NER, and temporal proximity of similar topics.


Semantic Inference for Auto-Capture

An algorithm that goes beyond regex matching to perform lightweight, semantic inference for detecting implicit patterns and assigning `importance` and `category` more accurately. This may involve a local pre-trained model or leveraging existing embeddings and vector search.


Enhanced Memory Capture Strategy for AIVA

A multi-stage process within the `agent_end` function to move beyond simple regex and build a more intelligent, context-aware system for capturing and storing information in AIVA's `memory-lancedb` plugin. Involves smarter triggers, contextual analysis, semantic inference via embeddings, and code-switching detection.


Contextual Capture Score Function

A function that analyzes conversation history, using keywords, sentiment analysis, and NER to calculate a score determining the significance of a message based on context. This score is used to decide whether to trigger more complex semantic analysis.


Semantic Pattern Matching via Embeddings

Utilizing the existing vector database and a curated list of 'exemplar' statements with associated category and importance. New messages are embedded and compared against exemplars for category and importance inference based on similarity scores.


Code-Switching Detection

A heuristic approach to detect multiple languages within a conversation by counting English and Czech words, flagging the conversation as code-switched if counts exceed a threshold and the ratio isn't extreme. Used to select correct trigger sets.


Memory Creation Pipeline

1.  Initial user message check.
2.  Code-switching detection to choose triggers.
3.  Calculate `contextualCaptureScore`.
4.  If the score is high enough, proceed.
5.  Run initial `shouldCapture` (with the selected triggers).
6.  If triggered, run `inferImplicitMemoryDetails` for semantic enrichment.
7.  Build the memory document with an informed `category` and `importance`.
8.  Save to the database.


enhanced_config.ts Memory Category Expansion

Expanding the `enhanced_config.ts` file with new implicit information categories and trigger patterns to improve the capture of user's implicit memory.


Implicit Information Regex/Keyword Patterns

Defines new regex/keyword patterns for categories: HEALTH_WELLBEING, FINANCIAL_STATE, RELATIONSHIP_DYNAMICS, VALUES_FRUSTRATIONS, CULTURAL_HABITS.  These patterns are used to detect and categorize implicit information in user input.


Contextual Trigger Activation Mechanism

A mechanism to precede the main trigger check, using sentiment analysis and contextual history to avoid running expensive logic on every message. It uses a `contextualCaptureScore` function that considers sentiment, topic categories, named entities, and introspective language.


Semantic Inference for Importance and Category

Uses a 'Semantic Pattern Matching' approach via embeddings and a curated list of example sentences to infer the category and importance of an implicitly stated memory. It relies on vector search against a set of pre-defined exemplars.


Implicit Pattern Exemplars

A curated list of example sentences with their ideal category and importance, used for semantic inference.


Code-Switching Detection Heuristic

A regex-based heuristic to identify and handle mixed-language messages (code-switching).  Uses keyword lists for different languages to determine if a message contains multiple languages and the dominant language.


Enhanced Memory Capture Logic

A multi-stage pipeline for memory capture, incorporating context analysis, trigger checking, semantic refinement, and importance evaluation. Aims to improve implicit information capture, adapt to context, leverage semantic understanding, and handle multilingual input.


Memory Consolidation Algorithm (consolidateMemories)

The `consolidateMemories` function in AIVA's `memory-lancedb` plugin (`enhanced_index.ts`) handles memory consolidation. The current implementation suffers from O(N²) complexity, suboptimal merge strategies, arbitrary staleness pruning, and lack of contradiction handling.


Incremental Consolidation using Approximate Nearest Neighbor Search

Proposes using locality-sensitive hashing (LSH) or a two-stage approach with LanceDB's vector search to identify merge candidates for memory consolidation, significantly reducing the O(N²) complexity to near O(N log N) or O(N).


Synthesis Merge Strategy for Memory Consolidation

Develops a 'synthesis' merge strategy that intelligently combines the text of two similar memories into a single, more comprehensive one using LLM calls or heuristic-based merging.


Adaptive Staleness Pruning based on Category Decay

Integrates the `CATEGORY_DECAY_PARAMS` (specifically `minimumImportance`) into the pruning logic. Memories should be pruned if their dynamically calculated current importance falls below their category's `minimumImportance` threshold and they haven't been recalled recently.


Contradiction-Aware Decay for Memories

Proposes a mechanism to identify and accelerate the decay of contradicted memories by increasing the decay rate or lowering the minimum importance when new information contradicts existing memories.


ConsolidateMemories Function Enhancement Strategy

A strategy to enhance the consolidateMemories function by addressing slow merges, information loss, static pruning, and lack of contradiction handling. This involves using k-NN search with LanceDB, LLM-powered memory synthesis, dynamic pruning based on importance scores, and proactive contradiction handling during memory storage.


LLM-Powered Memory Synthesis Function

Creation of an LLM-powered `synthesizeMemories` function to combine related memories into a richer statement, preserving information and context during the consolidation process.


Contradiction Detection in Memory Store Pipeline

Implementation of a contradiction detection mechanism within the `MemoryDB.store` function. New information will be checked against existing memories using an LLM. If contradictions are found, the importance of outdated facts will be reduced, accelerating their pruning.


k-NN Search with LanceDB for Memory Consolidation

Use k-Nearest Neighbors search with LanceDB to optimize memory consolidation, reducing complexity from O(N²) to O(M * k * log N). This involves processing subsets of memories and finding neighbors with LanceDB's search function.


Dynamic Pruning based on Importance Scores

Replace static staleness rules with dynamic pruning based on dynamically-calculated importance. The existing dynamic scoring system will determine if a memory should be pruned by comparing its importance against category-specific `minimumImportance` thresholds.


Incremental k-NN Memory Consolidation

Moves core merge logic from full table scan to targeted, batched operation using k-NN search to find similar memories for merging, improving efficiency from O(N^2) to approximately O(M * k * log N). This involves a consolidation batch size and k-NN parameter to control the search.


Memory Consolidation Process

A two-stage process that includes incremental semantic merge using k-NN to consolidate similar memories, followed by adaptive staleness pruning to remove outdated memories. Implemented in the `consolidateMemories` function of the `MemoryDB` class.


Intelligent Synthesis Merging via LLM

Introduces an LLM-powered function, `synthesizeMemoryTexts`, to merge similar memories into a single, comprehensive memory, minimizing information loss compared to simply discarding one. Utilizes a system prompt to guide the LLM in combining unique facts and eliminating redundancy.


Cross-Session Memory Transfer Protocol for AIVA

A comprehensive protocol for managing memory across user sessions in AIVA, addressing suboptimal pre-session memory priming, lack of coherent end-of-session memory consolidation, inadequate session-specific importance boosting, and absence of topic tracking and session continuity detection. It includes session boundary detection, context persistence, smart pre-session memory priming, context injection, end-of-session consolidation & archival trigger, and session-critical memory boosting & decay adjustment.


Session Context Persistence

Mechanism to persist a minimal `SessionSummary` (e.g., `lastNMemories`, `activeTopicEmbeddings`, `sessionGoals`) to a dedicated LanceDB table or a temporary file. Includes defining a `SessionContext` type and integrating it with `api.on("agent_end")` and `api.on("before_agent_start")`.


Smart Pre-Session Memory Priming

Enhance `api.on("before_agent_start")` to leverage `SessionContext` from the previous session to actively incorporate `activeTopicEmbeddings` to perform an initial, broader hybrid search. Prioritize memories with a "session-critical" flag.


Automatic End-of-Session Consolidation

Implement an automatic, background `consolidateMemories` trigger after a period of inactivity. The process should perform a more efficient, possibly chunked, consolidation process to avoid blocking.


Session-Critical Memory Management

Introduce a `sessionCritical` boolean flag or a `sessionImportanceBoost` field to `MemoryEntry`. Memories involved in a session get this flag set or boost applied.


Memory Plugin Session Management System

Complete session management for the memory plugin, addressing cold start priming, automatic session consolidation, memory boosting, and topic continuity. Uses SessionContext to maintain state.


Session Boundary Detection using Inactivity Threshold

Defines a session as the period between `before_agent_start` and `agent_end`, extended by an inactivity window (`SESSION_INACTIVITY_THRESHOLD_MS`).


Smart Pre-Session Priming with Hybrid Search

Uses `SessionContext` to build a hybrid search in `before_agent_start`, combining user input with session's background, improving cold start relevance.


End-of-Session Consolidation with Batching

Consolidates similar memories in the background at the end of a session using a `setTimeout` timer to trigger batched consolidation, preventing memory bloat.


Session-Critical Boosting with Decaying Importance

Boosts important memories of the session temporarily using `sessionImportanceBoost` in `MemoryEntry`, decaying over time, with a `lastBoostedAt` timestamp.


Session Manager

The Session Manager tracks user interaction sessions, persists context (recent memory IDs, topic embeddings, etc.) to a `session_context.json` file, and resumes sessions if the inactivity threshold isn't exceeded. It determines session boundaries using `SESSION_INACTIVITY_THRESHOLD_MS` and manages session state.


Implement `_primeSessionMemories` for cross-session context

Creates a `_primeSessionMemories` function in `session_manager.ts` to load context from previous sessions using a hybrid search (user prompt and previous active topic embeddings). This function is integrated into the `enhanced_index.ts` `before_agent_start` hook to prime the agent's memory with relevant information before the conversation begins.


Implement Inactivity-Based Trigger for Memory Consolidation

Refactor consolidateMemories to run asynchronously in batches triggered by inactivity, reducing main thread latency. Integrate the scheduling in enhanced_index.ts agent_end hook.


Asynchronous Batched Memory Consolidation

Refactor of consolidateMemories to run in batches asynchronously using _consolidateBatch function. The function identifies duplicate and stale memories within each batch, deletes them and logs the deleted redundant memories.


Session-Critical Memory Boosting and Decay

Implements session-specific importance boosting for memories with rapid decay after session end. It adds `sessionImportanceBoost` and `lastBoostedAt` to `MemoryEntry`, updating them on memory usage and modifying `_calculateDynamicScores` to include the boost with decay.


Update MemoryEntry Type

Adds `sessionImportanceBoost` and `lastBoostedAt` fields to the `MemoryEntry` type in `enhanced_index.ts` to support session-specific boosting.


Modify _calculateDynamicScores

Updates the `_calculateDynamicScores` function to incorporate session-critical boost with accelerated decay, using `sessionImportanceBoost` and `lastBoostedAt` to adjust memory scores based on session context.


Dual-Store Memory Architecture (Episodic/Semantic Separation)

Design a dual-store architecture separating episodic and semantic memory, leveraging LanceDB with distinct schemas, indexing, and decay policies for each.


Memory Category-Based Routing of Memory Entries

Automatically route incoming memory_store calls and autoCapture events to the appropriate episodic or semantic store based on the MemoryCategory.


Differentiated Memory Decay & Consolidation Policies

Implement distinct decay functions and consolidation routines for episodic and semantic memories, optimizing for rapid episodic decay and long-term semantic retention.


Unified Hybrid Search with Cross-Store Result Fusion

Design a unified hybridSearch entry point that intelligently queries both episodic and semantic memory stores, using a fusion algorithm to combine results.


Semantic Graph-Based Retrieval

Introduce graph-based retrieval for semantic memory to enable more sophisticated inferential retrieval by modeling relationships between entities and concepts.


Dual-Store Memory Architecture (Episodic & Semantic)

A dual-store architecture separating episodic and semantic memories to improve retrieval, decay, and indexing efficiency within AIVA's memory system, mirroring cognitive science principles.


MemoryManager Class with EpisodicMemoryDB & SemanticMemoryDB

Using a `MemoryManager` class containing `EpisodicMemoryDB` and `SemanticMemoryDB` instances for a cleaner, more maintainable implementation (composition over inheritance).


Category-based Routing via detectCategory Function

Routing memory entries to the appropriate `MemoryDB` instance based on content analysis performed by the `detectCategory` function.


Differentiated Decay Parameters for Episodic/Semantic

Employing distinct decay parameters for episodic (fast decay) and semantic (near-permanent) memories, controlled by a `CATEGORY_DECAY_PARAMS` object, to optimize memory footprint and recall.


Unified hybridSearch Entry Point in MemoryManager

A unified `hybridSearch` entry point within the `MemoryManager` to query both memory stores in parallel, using weighted sum fusion and re-ranking for combined results.


A/B Testing for Dual-Store Memory Evaluation

Proposing A/B testing to compare the current monolithic system with the new dual-store architecture, measuring latency, precision, memory footprint, and user feedback.


MemoryManager Class

A central orchestrator that manages two MemoryDB instances (episodic and semantic) and routes memory entries to the appropriate database based on category. It also enriches entries with database-specific fields.


MemoryDB Class

A generic table handler that interacts with LanceDB to store and retrieve memories. It is instantiated twice, once for episodic memory and once for semantic memory.


Dual MemoryDB Design

The system will use separate MemoryDB instances for episodic and semantic memory to enable more efficient storage and retrieval based on memory type.


Episodic and Semantic Memory Consolidation

The system consolidates episodic and semantic memories separately, using differentiated decay parameters and consolidation logic to manage memory footprint and recall.


Differentiated Decay Policies for Memory Categories

Different memory categories (event, decision, fact, entity, etc.) are assigned different decay functions (exponential, static, reinforcement) and parameters (decayRate, timeToLiveSeconds, reinforcementBonus) to optimize retention based on memory type.


MemoryManager

A new dual-store architecture that provides a unified search interface by intelligently combining episodic and semantic memory retrieval, implemented with LanceDB. It includes hybrid search and advanced semantic search capabilities.


Hybrid Search Fusion Algorithm

Uses weighted re-ranking and MMR to combine episodic and semantic search results. Weights are dynamically adjusted based on query type and store preference. Aims for diverse and relevant results by considering the source store (episodic/semantic).


Advanced Semantic Search with Entity Graph Expansion

Expands semantic search results by identifying related entities from initial results and then searching for memories linked to those entities in a second pass. This enhances the retrieval of relevant knowledge.


A/B Testing for Memory Upgrade Validation

Validate the new `MemoryManager` architecture with A/B testing, comparing it against the existing monolithic `memory-lancedb` plugin. The test will involve a golden dataset of 500 diverse queries annotated with query type and ground truth memory IDs.


Memory Importance Decay Curve Engineering

Focuses on engineering memory importance decay curves, likely to optimize memory usage and performance.


Cognitive Forgetting & Reinforcement Engine in MemoryDB

Proposes a more sophisticated, configurable, and adaptive decay mechanism for memory importance within MemoryDB. This includes refining decay parameters, enhancing decay functions (power law, event-based, spaced repetition), and introducing adaptive spaced repetition in reinforceMemory.


Cognitive Forgetting & Reinforcement Engine

A system designed to enhance memory realism and performance by implementing per-category decay, true power law decay, adaptive spaced repetition (SM-2), and event-based decay refinement.


Adaptive Spaced Repetition with SM-2

Overhauling reinforceMemory to use a simplified SM-2 algorithm for optimized learning intervals, tracking 'easiness factor' and 'review interval'.


Memory Decay Configuration

Defines decay parameters for different memory categories (skill, fact, entity, event, conversation, core_identity) using various decay functions (powerLaw, linear, exponential, eventBased, static). Configured within CATEGORY_DECAY_PARAMS in `enhanced_config.ts`.


MemoryDB Class Implementation

Implementation of the MemoryDB class in `enhanced_index.ts` to manage memory entries with decay functions, spaced repetition using SM-2 algorithm, and importance updates. Includes `addMemory`, `reinforceMemory`, and `_calculateDynamicScores` methods.


Spaced Repetition with SM-2 Algorithm

Utilizes the SM-2 algorithm within `reinforceMemory` to adjust memory recall intervals and easiness factors based on recall quality, optimizing long-term memory retention.


Dynamic Score Calculation with Decay Functions

Calculates recency and importance scores using category-specific decay functions (power law, event-based). These scores determine how quickly a memory fades and enable event-triggered importance boosting.


The new memory system implements granular parameters for decay, Ebbinghaus Power Law for long-term decay, adaptive spaced repetition using SM-2 algorithm, and nuanced event-based decay. It is designed to be more robust, adaptive, and aligned with human cognitive patterns, resulting in more relevant and timely retrievals in hybridSearch.


Category-Specific Memory Decay Parameters

Memory decay parameters are now defined per-category (CATEGORY_DECAY_PARAMS), allowing different decay rates for different types of memories (e.g., skills vs. conversations).


Adaptive Spaced Repetition Engine

The `reinforceMemory` function is now a true adaptive engine that infers recall quality, updates memory's `easinessFactor` using the SM-2 algorithm, and dynamically calculates the next optimal `reviewIntervalDays`.


Context-Aware Memory Injection Layer

A system designed to dynamically re-rank, filter, and place memories based on the evolving conversation context to improve relevance and reduce token waste.


Dynamic Topic Modeling & Intent Detection for Re-ranking

Infer conversation topic and user intent from recent messages to re-rank HybridSearchResult[] by adjusting finalScore based on topic alignment. Improves P90 injection precision and reduces irrelevant memory injection.


Contextual Suppression & Relevance Filtering

Identify and suppress contextually irrelevant memories, despite high finalScore, using ephemeral embeddings or negative prompts to actively penalize memories semantically distant from the current focus. Reduces token waste.


Adaptive Injection Placement

Algorithm to determine the optimal injection placement (beginning, middle, or end of context) for each memory based on its estimated urgency and cognitive load.


Context-Aware Memory Injection Layer for AIVA

A dynamic system to inject memories based on conversational context, user needs, topic modeling, contextual filtering, and adaptive placement within the AIVA plugin.


Dynamic Memory Re-Ranking by Topic Alignment

Re-rank hybrid search results by comparing topic embedding to memory embeddings, using a tunable topicAlignmentBonus to adjust original scores.


Contextual Filtering by Emphasis Comparison

Filter irrelevant memories by comparing memory embeddings to an 'emphasis string' derived from the user's immediate focus.


Adaptive Injection Placement Based on Memory Category & Intent

Dynamically determine whether to prepend or append memories based on memory category (e.g., decision, fact) and user intent (e.g., problem-solving, information-seeking).


A system to adapt to conversational drift by understanding the current topic and user intent, using a lightweight LLM call and re-ranking memory candidates based on semantic similarity between the topic and memory embeddings.


Re-ranking Logic based on Topic Similarity

Adjusts the final score of each memory candidate based on the cosine similarity between the topic's embedding and the memory's embedding, using a configurable weight `RE_RANK_TOPIC_WEIGHT`. `newScore = originalScore * (1 + (RE_RANK_TOPIC_WEIGHT * cosineSimilarity(topicVector, memoryVector)))`


A fine-grained filter to remove memories semantically related to the initial prompt but irrelevant to the immediate turn by using a current turn emphasis embedding to represent the narrow focus of the user's latest message. It uses a cosine similarity threshold (`RELEVANCE_EMPHASIS_THRESHOLD`) to discard irrelevant memories.


A rule-based system that sorts memories into `prepend` and `append` buckets based on the memory's category and the inferred user intent, optimizing the location within the context for presenting recalled information to the agent.


Context-Aware Memory Injection System

A system to inject memories into the agent's context based on conversation context, intent, and emphasis, using re-ranking, filtering, and adaptive placement strategies.


Context Inference

Infer the conversation topic and intent from the conversation history.


Dynamic Re-ranking by Context

Re-rank memory candidates based on their relevance to the inferred conversation topic using `reRankByContext` function.


Contextual Filtering

Filter memory candidates based on relevance to current turn emphasis using `filterContextualIrrelevance` function.


Adaptive Injection Placement

Determine optimal placement of injected memories (prepend/append) based on inferred intent using `optimizeInjectionPlacement` function.


Memory Compression and Tiered Storage Strategy

Design a comprehensive memory compression and tiered storage strategy for AIVA's memory-lancedb plugin to address scalability and cost challenges associated with storing a large number of memories, including progressive summarization, hierarchical memory clusters (Meta-Memories), and vector quantization.


Progressive Summarization

Implement a mechanism where memories transition from full detail to compressed summaries based on age and importance, defining MemoryCompressionTier enum (FULL, SUMMARY_SHORT, SUMMARY_LONG, COLD).  Includes creating a compressMemory method within MemoryDB and adapting db.hybridSearch and db.vectorSearch to retrieve and re-hydrate summaries.


Hierarchical Memory Clusters (Meta-Memories)

Design a system to group related MemoryEntry items into a new MetaMemory type, including creating a background process to identify clusters of MemoryEntry objects and generate summaryText and summaryVector. Memory_recall prioritizes MetaMemory objects and drills down when more detail is required.


Vector Quantization

Implement a strategy for compressing MemoryEntry.vector embeddings, considering Product Quantization (PQ) or Scalar Quantization (SQ) for LanceDB. Configuration parameters are added to MemoryDB initialization, impacting db.store, db.update, and db.vectorSearch.


Tiered Memory Management with Compression and Hierarchy

Implement a multi-layered memory system with active, meta-memory, and cold storage tiers. Use progressive summarization, hierarchical clustering, vector quantization, and cold storage to optimize memory usage and retrieval performance.


Progressive Summarization for Memory Compression

Employ a tiered approach using MemoryCompressionTier to generate short and long summaries of MemoryEntry objects. Re-embed summaries as vectors. Introduce expandMemory tool to retrieve original or fleshed-out summaries.


Create MetaMemory to define the structure of memory clusters. Use a background process to cluster related memories via vectorSearch and generate summaryText for each cluster, embedding it as summaryVector. Store in a separate LanceDB table meta_memories.


Vector Quantization with Product Quantization (PQ)

Use Product Quantization (PQ) within LanceDB to compress memory vectors by configuring parameters like num_partitions, num_sub_vectors, and num_bits in the MemoryDB.init method.


Cold Storage Tier for Stale Memories

Move stale, less important memories to a cold storage tier (e.g., local JSON files transitioning to cloud storage). Add a recall_from_archive tool to retrieve from archive and re-integrate into main LanceDB.


Reduces storage cost of stale memories by progressively lowering text fidelity through LLM-generated summaries (short, long) and moving original text to cold storage. Includes data model enhancements, compression/rehydration logic, and hashing.


Memory Compression Tiering System

Defines memory compression tiers: FULL, SUMMARY_SHORT, SUMMARY_LONG, COLD. Manages memory fidelity and storage location based on access frequency. Facilitates text compression using LLMs and storage of the original text in a cold storage system.


Memory Re-hydration via expand_memory tool

Enables agent to request the full text of a compressed memory from cold storage when a higher level of detail is needed. Utilizes a `expand_memory` tool to restore the full memory to the active database.


Introduces a new data type (MetaMemory) and process to create conceptual summaries of existing memories, addressing the flat memory structure. This includes data model enhancements in `enhanced_config.ts`, implementation in `enhanced_index.ts` (new LanceDB table, clustering method), and integration with `memory_recall` to search both regular and meta-memories.


Vector Quantization using Product Quantization (PQ)

Reduces the memory and storage footprint of vectors by using Product Quantization (PQ) in LanceDB. Configured during table creation in `MemoryDB.init` with specific parameters like `num_bits` (bits per codebook) and `num_sub_vectors`.


Use LanceDB for Meta-Memory Storage and Search

Leverages LanceDB's vector search capabilities for both regular memories and meta-memories. Includes creating and managing a new table called 'meta_memories' with a defined schema and using LanceDB's indexing capabilities, including IVF-PQ for optimized vector search.


Cold Storage Tier for Memories

Implements a cold storage tier to move old, unimportant memories out of the active database into cheap, long-term storage. Uses a file-based approach initially, with potential to upgrade to an S3-compatible object store.


Archive Stale Memories

Identifies and archives memories based on age, importance, and last recall date. Memories older than 365 days, with importance less than 0.15, and not recalled in 180 days are moved to cold storage.


Consolidate Memories

Orchestrates memory management: pruning duplicates, archiving stale memories, compressing remaining memories, and clustering.