Vector-Augmented Context Provider (MCP Server)

This repository presents a proof-of-concept implementation of an MCP endpoint serving as a sophisticated context augmentation service. It allows intelligent agents to query an embedded knowledge base (vector database) to gather prerequisite material for Retrieval-Augmented Generation (RAG) workflows.

Core Capabilities

• FastAPI backend exposing standardized MCP endpoints.
• Integration with FAISS for efficient nearest-neighbor vector lookups.
• Automated document segmentation and embedding generation.
• Dedicated module for fetching and parsing Move language files from GitHub.
• End-to-end RAG pipeline orchestration, including LLM interface.
• Example client utilities for immediate testing.
• Provision of initial sample datasets.

Deployment Guide

Preferred Installation Method: pipx

pipx simplifies the deployment of Python command-line tools into isolated environments.

1. Install pipx prerequisite:

# macOS (using Homebrew)
brew install pipx
pipx ensurepath

# Debian/Ubuntu
sudo apt update
sudo apt install python3-pip python3-venv
python3 -m pip install --user pipx
python3 -m pipx ensurepath

# Windows (using pip)
pip install pipx
pipx ensurepath

1. Install the package:

Navigate to the root directory of the source code and install the package in editable mode:

# Change directory to the project root
cd /path/to/project-source

pipx install -e .

1. Configuration Adjustments:
   • Duplicate .env.example to create a working .env file.
   • Populate GITHUB_TOKEN for elevated access to GitHub APIs.
   • Supply your proprietary LLM credentials (e.g., OPENAI_API_KEY) to enable the generative component of RAG.

Alternative: Direct Installation

If isolation via pipx is not desired:

1. Clone the repository contents.
2. Install required dependencies:

cd mcp_server
pip install -r requirements.txt

Command Line Interface (CLI) Operations (Post-pipx)

Fetching Sui Move Source Files from GitHub

# Basic retrieval using default search parameters
mcp-download --query "use sui" --output-dir docs/move_sources

# Detailed fetch targeting specific modules with pagination control
mcp-download --query "module sui::coin" --max-results 50 --new-index --verbose

Advanced GitHub Indexing and Vectorization (Recommended)

This command searches GitHub, extracts relevant Move code, and updates the knowledge base.

# Initiate indexing based on primary keywords
mcp-search-index --keywords "sui move"

# Search across multiple scopes, limiting repository counts, and verbosely outputting results
mcp-search-index --keywords "sui move,move framework" --max-repos 30 --output-results --verbose

# Specify an alternate location for the generated vector index file
mcp-search-index --keywords "sui coin,sui::transfer" --index-file custom/vector_store.bin --output-results

The mcp-search-index utility operates as follows:

• Prioritizes repository discovery before recursive traversal for Move files.
• Accepts a comma-separated list for simultaneous multi-keyword searches.
• Employs heuristic filtering, favoring files that contain "use sui".
• Guarantees a complete rebuild of the vector index post-ingestion.

Populating the Vector Store

# Index files located in the default document directory
mcp-index

# Customize input and output paths for the vector index
mcp-index --docs-dir path/to/source_code --index-file path/to/custom_index.bin --verbose

Vector Database Querying

# Simple semantic lookup
mcp-query "What defines a module construct in Sui Move?"

# Query with parameter tuning (k=neighbors, f=filter flag)
mcp-query "How do I instantiate a struct in Sui Move?" -k 3 -f

Executing the Full RAG Cycle

# Execute RAG (will default to a mock LLM if API key is missing)
mcp-rag "What is the purpose of a Sui module?"

# Specify LLM parameters
mcp-rag "Describe the structure of a Sui struct definition" --api-key your_key --top-k 3

# Direct output format to JSON for machine readability
mcp-rag "Contrast sui::coin behavior" --output-json > structured_answer.json

Launching the MCP Service Endpoint

# Start the API server using default configurations
mcp-server

# Specify custom network binding and point to a pre-existing index file
mcp-server --host 127.0.0.1 --port 8080 --index-file custom/vector_store.bin

Non-pipx Execution Methods

Initiating the Backend Service

cd mcp_server
python main.py

The API will typically be accessible at http://localhost:8000

GitHub Source Acquisition Workflow

Use the wrapper script to orchestrate downloads:

# Trigger download with default parameters
./run.sh --download-move

# Override the GitHub search predicate and result count
./run.sh --download-move --github-query "module sui::coin" --max-results 50

# Fetch, index the retrieved data, and launch the server
./run.sh --download-move --index

You can also invoke the dedicated Python script:

python download_move_files.py --query "use sui" --output-dir docs/source_code

Data Ingestion (Indexing)

Prior to querying, the acquired textual assets (.txt, .md, .move) located in the docs folder must be vectorized.

1. Via the helper script:

./run.sh --index

1. Via the direct indexing utility:

python index_move_files.py --docs-dir docs/source_code --index-file data/faiss_index.bin

Localized Retrieval Testing

python local_query.py "What constitutes RAG methodology?"

# Using query tuning parameters
python local_query.py -k 3 -f "Explain struct initialization syntax in Sui Move"

Direct RAG Execution (Scripted)

# Run RAG locally against the indexed data
python rag_integration.py "What is the structure of a Sui module?" --index-file data/faiss_index.bin

# Utilize an external LLM via environment variable
OPENAI_API_KEY=your_key python rag_integration.py "What are the transactional primitives for Sui coins?"

MCP HTTP Interface

The primary interface for external systems is the /mcp/action POST endpoint.

• retrieve_documents: Fetches contextually relevant vectors.
• index_documents: Triggers an indexing operation on a specified corpus location.

Request Example:

curl -X POST "http://localhost:8000/mcp/action" \
     -H "Content-Type: application/json" \
     -d '{"action_type": "retrieve_documents", "payload": {"query": "What is RAG?", "top_k": 3}}'

Full Retrieval-Augmented Generation Flow

The implemented RAG workflow proceeds through these stages:

1. Input Prompt: Agent submits a natural language inquiry.
2. Context Fetch: The query is transformed into a vector and used to search the knowledge base for closest matches.
3. Prompt Engineering: The retrieved document segments are assembled into a coherent context block.
4. LLM Synthesis: This augmented prompt is submitted to the configured Large Language Model.
5. Grounded Answer: The LLM returns a response directly substantiated by the retrieved evidence.

This complete chain is encapsulated within rag_integration.py, accessible via CLI or as an internal module.

GitHub Move Source Acquisition Details

Source code extraction from GitHub repositories is facilitated by two mechanisms:

1. GitHub API (Preferred): Requires a valid token to bypass strict rate limiting.
2. Web Scraping Fallback: Used only if the API mechanism fails or if credentials are absent.

Configure your access token within the .env file:

GITHUB_TOKEN=your_github_token_here

Project Directory Layout

mcp_server/
├── __init__.py             # Python package marker
├── main.py                # Entry point for the FastAPI listener
├── mcp_api.py             # MCP endpoint definitions and logic
├── index_move_files.py    # Utility for vector store population
├── local_query.py         # Script for offline querying the index
├── download_move_files.py # Module managing GitHub repository data extraction
├── rag_integration.py     # Orchestration layer connecting retrieval to LLM generation
├── pyproject.toml         # Project metadata and build settings
├── requirements.txt       # Dependency list
├── .env.example           # Template for environment configuration
├── README.md              # This documentation file
├── data/                  # Persistence directory for FAISS indices
│   └── faiss_index.bin    # Default vector database file
├── docs/
│   └── move_files/        # Location for ingested source code artifacts
├── models/
│   └── vector_store.py    # Specific implementation details for the FAISS backend
└── utils/
    ├── document_processor.py  # Logic for text preparation and segmentation
    └── github_extractor.py    # Low-level GitHub interaction utilities

Opportunities for Enhancement

1. Implement robust access control and authentication mechanisms.
2. Develop more granular and context-aware document preprocessing pipelines.
3. Expand support to ingest and vectorize diverse data formats.
4. Integrate supplementary or alternative commercial/open-source LLM endpoints.
5. Establish comprehensive operational logging and performance monitoring.
6. Introduce advanced static analysis features for Move code structure.

Licensing

Distributed under the MIT License.