Memory infrastructure for LLMs and AI agents

MemU is an agentic memory framework for LLM and AI agent backends. It receives multimodal inputs (conversations, documents, images), extracts them into structured memory, and organizes them into a hierarchical file system that supports both embedding-based (RAG) and non-embedding (LLM) retrieval.

MemU is collaborating with four open-source projects to launch the 2026 New Year Challenge. 🎉Between January 8–18, contributors can submit PRs to memU and earn cash rewards, community recognition, and platform credits. 🎁Learn more & get involved

MemU organizes memory using a three-layer architecture inspired by hierarchical storage systems:

Key Benefits:

• Full Traceability: Track from raw data → items → categories and back
• Progressive Summarization: Each layer provides increasingly abstracted views
• Flexible Organization: Categories evolve based on content patterns

MemU processes diverse content types into unified memory:

All modalities are unified into the same three-layer hierarchy, enabling cross-modal retrieval.

Try MemU instantly without any setup:

👉 memu.so - Hosted cloud service with full API access

For enterprise deployment and custom solutions, contact info@nevamind.ai

📚 Full API Documentation

pip install -e .

Requirements: Python 3.13+ and an OpenAI API key

Test with In-Memory Storage (no database required):

export OPENAI_API_KEY=your_api_key
cd tests
python test_inmemory.py

Test with PostgreSQL Storage (requires pgvector):

# Start PostgreSQL with pgvector
docker run -d \
  --name memu-postgres \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=memu \
  -p 5432:5432 \
  pgvector/pgvector:pg16

# Run the test
export OPENAI_API_KEY=your_api_key
cd tests
python test_postgres.py

Both examples demonstrate the complete workflow:

1. Memorize: Process a conversation file and extract structured memory
2. Retrieve (RAG): Fast embedding-based search
3. Retrieve (LLM): Deep semantic understanding search

See tests/test_inmemory.py and tests/test_postgres.py for the full source code.

MemU supports custom LLM and embedding providers beyond OpenAI. Configure them via llm_profiles:

from memu import MemUService

service = MemUService(
    llm_profiles={
        # Default profile for LLM operations
        "default": {
            "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
            "api_key": "your_api_key",
            "chat_model": "qwen3-max",
            "client_backend": "sdk"  # "sdk" or "http"
        },
        # Separate profile for embeddings
        "embedding": {
            "base_url": "https://api.voyageai.com/v1",
            "api_key": "your_voyage_api_key",
            "embed_model": "voyage-3.5-lite"
        }
    },
    # ... other configuration
)

Processes input resources and extracts structured memory:

result = await service.memorize(
    resource_url="path/to/file.json",  # File path or URL
    modality="conversation",            # conversation | document | image | video | audio
    user={"user_id": "123"}             # Optional: scope to a user
)

# Returns:
{
    "resource": {...},      # Stored resource metadata
    "items": [...],         # Extracted memory items
    "categories": [...]     # Updated category summaries
}

Retrieves relevant memory based on queries. MemU supports two retrieval strategies:

Fast embedding vector search using cosine similarity:

• ✅ Fast: Pure vector computation
• ✅ Scalable: Efficient for large memory stores
• ✅ Returns scores: Each result includes similarity score

Deep semantic understanding through direct LLM reasoning:

• ✅ Deep understanding: LLM comprehends context and nuance
• ✅ Query rewriting: Automatically refines query at each tier
• ✅ Adaptive: Stops early when sufficient information is found

Both methods support:

• Context-aware rewriting: Resolves pronouns using conversation history
• Progressive search: Categories → Items → Resources
• Sufficiency checking: Stops when enough information is retrieved

result = await service.retrieve(
    queries=[
        {"role": "user", "content": {"text": "What are their preferences?"}},
        {"role": "user", "content": {"text": "Tell me about work habits"}}
    ],
    where={"user_id": "123"}  # Optional: scope filter
)

# Returns:
{
    "categories": [...],     # Relevant categories (with scores for RAG)
    "items": [...],          # Relevant memory items
    "resources": [...],      # Related raw resources
    "next_step_query": "..." # Rewritten query for follow-up (if applicable)
}

Scope Filtering: Use where to filter by user model fields:

• where={"user_id": "123"} - exact match
• where={"agent_id__in": ["1", "2"]} - match any in list
• Omit where to retrieve across all scopes

📚 For complete API documentation, see SERVICE_API.md - includes all methods, CRUD operations, pipeline configuration, and configuration types.

Extract and organize memory from multi-turn conversations:

export OPENAI_API_KEY=your_api_key
python examples/example_1_conversation_memory.py

What it does:

• Processes multiple conversation JSON files
• Extracts memory items (preferences, habits, opinions, relationships)
• Generates category markdown files (preferences.md, work_life.md, etc.)

Best for: Personal AI assistants, customer support bots, social chatbots

Extract skills and lessons learned from agent execution logs:

export OPENAI_API_KEY=your_api_key
python examples/example_2_skill_extraction.py

What it does:

• Processes agent logs sequentially
• Extracts actions, outcomes, and lessons learned
• Demonstrates incremental learning - memory evolves with each file
• Generates evolving skill guides (log_1.md → log_2.md → skill.md)

Best for: DevOps teams, agent self-improvement, knowledge management

Process diverse content types into unified memory:

export OPENAI_API_KEY=your_api_key
python examples/example_3_multimodal_memory.py

What it does:

• Processes documents and images together
• Extracts memory from different content types
• Unifies into cross-modal categories (technical_documentation, visual_diagrams, etc.)

Best for: Documentation systems, learning platforms, research tools

MemU achieves 92.09% average accuracy on the Locomo benchmark across all reasoning tasks.

View detailed experimental data: memU-experiment

Quick Links:

• 🚀 Try MemU Cloud
• 📚 API Documentation
• 💬 Discord Community

Apache License 2.0

• GitHub Issues: Report bugs & request features
• Discord: Join the community
• X (Twitter): Follow @memU_ai
• Contact: info@nevamind.ai