Unstructured Management Control Point (UNS-MCP) Server Implementation

This component serves as an MCP server facilitating robust interaction with the Unstructured API for managing ingestion pipelines, processing definitions, and tracking execution status.

Supported Toolset

The following functionalities are exposed via dedicated tools:

Operation ID	Purpose
list_sources	Enumerate available input connection points from the Unstructured API.
get_source_info	Retrieve comprehensive metadata for a designated source connector instance.
create_source_connector	Provision a new source connection point.
update_source_connector	Modify parameters of an extant source connection point.
delete_source_connector	Decommission a source connection point, referencing its unique identifier.
list_destinations	List all configured output sinks within the Unstructured API environment.
get_destination_info	Fetch detailed specifications for a particular output sink connector.
create_destination_connector	Establish a new output sink connection point.
update_destination_connector	Apply parameter updates to an active output sink.
delete_destination_connector	Retire an output sink connector based on its ID.
list_workflows	Fetch the catalog of defined processing workflows.
get_workflow_info	Obtain granular configuration details for a specified workflow definition.
create_workflow	Instantiate a novel processing workflow, defining its sequence of source/sink linkages.
run_workflow	Initiate execution for a selected workflow instance.
update_workflow	Overhaul the configuration schema of an existing workflow.
delete_workflow	Erase a workflow definition from the system.
list_jobs	Present execution records associated with a particular workflow.
get_job_info	Acquire detailed telemetry for an individual job execution instance.
cancel_job	Terminate an active job execution prematurely.
list_workflows_with_finished_jobs	Provide an overview of workflows possessing completed executions, including associated connector details.

Supported Connector Types

This server implementation natively supports the following source and destination types. Consult the official Unstructured documentation here for sources and here for sinks.

Source Connectors	Destination Connectors
S3 Storage	S3 Storage
Azure Blob Storage	Weaviate Vector Store
Google Drive	Pinecone Vector DB
OneDrive	AstraDB
Salesforce	MongoDB
Sharepoint	Neo4j Graph DB
	Databricks Volumes
	Databricks Delta Table

Required Environmental Configuration

To successfully instantiate or modify connectors, corresponding authentication secrets must be present in the .env configuration file. Below details the required credential names:

Credential Key(s)	Context/Use Case
ANTHROPIC_API_KEY	Essential for the minimal_client to communicate with the gateway server.
AWS_KEY, AWS_SECRET	Required for provisioning S3 source/sink connectors. Refer to S3 documentation for setup details.
WEAVIATE_CLOUD_API_KEY	Mandatory for configuring the Weaviate destination connector.
FIRECRAWL_API_KEY	Activates Firecrawl utilities found in external/firecrawl.py. Obtain key from Firecrawl portal.
ASTRA_DB_APPLICATION_TOKEN, ASTRA_DB_API_ENDPOINT	Necessary for establishing the AstraDB destination linkage.
AZURE_CONNECTION_STRING	Authentication option 1 for Azure Blob Storage source provisioning.
AZURE_ACCOUNT_NAME+AZURE_ACCOUNT_KEY	Authentication option 2 for Azure Blob Storage source provisioning.
AZURE_ACCOUNT_NAME+AZURE_SAS_TOKEN	Authentication option 3 for Azure Blob Storage source provisioning.
NEO4J_PASSWORD	Required credential for setting up the Neo4j graph database sink.
MONGO_DB_CONNECTION_STRING	Prerequisite for provisioning the MongoDB destination connector.
GOOGLEDRIVE_SERVICE_ACCOUNT_KEY	A base64-encoded string representation of the Google Service Account JSON key file.
DATABRICKS_CLIENT_ID,DATABRICKS_CLIENT_SECRET	Needed for Databricks volume and delta table destination provisioning.
ONEDRIVE_CLIENT_ID, ONEDRIVE_CLIENT_CRED,ONEDRIVE_TENANT_ID	Required for configuring the OneDrive destination system.
PINECONE_API_KEY	Mandatory for setting up the Pinecone vector database output connector.
SALESFORCE_CONSUMER_KEY,SALESFORCE_PRIVATE_KEY	Credentials needed to connect to the Salesforce source system.
SHAREPOINT_CLIENT_ID, SHAREPOINT_CLIENT_CRED,SHAREPOINT_TENANT_ID	Required for establishing the Sharepoint source integration.
LOG_LEVEL	Controls verbosity of the minimal_client, e.g., ERROR for minimal output.
CONFIRM_TOOL_USE	Setting this to true mandates explicit client confirmation prior to any tool invocation.
DEBUG_API_REQUESTS	Setting to true enables verbose logging of request parameters within uns_mcp/server.py for diagnostic purposes.

Firecrawl Integration Details

Firecrawl, accessible via the FIRECRAWL_API_KEY, extends the MCP's capabilities with two primary web interaction methods:

1. Raw HTML Fetching: Utilize invoke_firecrawl_crawlhtml to initiate crawl tasks and check_crawlhtml_status for tracking progress.
2. LLM-Prepared Text Extraction: Employ invoke_firecrawl_llmtxt to generate text optimally structured for Language Models, tracked via check_llmtxt_status.

Crawling Workflow: - Initiation targets a base URL, dynamically identifying internal links (prioritizing sitemaps if present). - The process recursively traverses discovered links, ensuring content capture across the site structure, handling JS rendering and adhering to throttling limits. - Jobs can be halted using cancel_crawlhtml_job. - This method is ideal when the raw HTML artifact is preferred for subsequent Unstructured pipeline cleansing.

LLM Text Generation Workflow: - Post-crawling, the system refines the gathered material into clean, meaningful text. - Output is formatted specifically for efficient LLM consumption. - Results are automatically persisted to a pre-defined S3 bucket. - Important Note: LLM text generation jobs are non-cancellable once initiated. cancel_llmtxt_job is present for API consistency but is unimplemented on the Firecrawl side.

Setup and Environment Initialization

This guide assumes a development environment utilizing Python 3.12 and the uv package manager.

Prerequisites

• Python version 3.12 or newer.
• The uv package manager installed.
• An active Unstructured API key obtained from the platform portal.

Recommended: Utilizing uv Package Management

If using uvx for execution, no direct installation is necessary. For direct package installation: bash uv pip install uns_mcp

Claude Desktop Configuration (claude_desktop_config.json in ~/Library/Application Support/Claude/)

Option A: Using uvx Execution Wrapper

{ "mcpServers": { "UNS_MCP": { "command": "uvx", "args": ["uns_mcp"], "env": { "UNSTRUCTURED_API_KEY": "" } } } }

Option B: Using Standard Python Package Execution

{ "mcpServers": { "UNS_MCP": { "command": "python", "args": ["-m", "uns_mcp"], "env": { "UNSTRUCTURED_API_KEY": "" } } } }

Source Code Checkout Method

1. Clone the repository source.

2. Sync dependencies: bash uv sync

3. Configure your Unstructured API key via a root .env file: bash UNSTRUCTURED_API_KEY="YOUR_KEY" (Consult .env.template for all configurable environmental variables.)

Server execution can commence via:

Editable Package Install Method

Install in editable mode: bash uvx pip install -e . Update the configuration file: { "mcpServers": { "UNS_MCP": { "command": "uvx", "args": ["uns_mcp"] } } } **Crucial:** Ensure your environment points to the correct `uvx` executable path.

Server-Sent Events (SSE) Protocol Debugging

**Note**: SSE protocol is incompatible with Claude Desktop's default integration. Decouple server and client for enhanced logging: 1. Start the server process in terminal one: bash uv run python uns_mcp/server.py --host 127.0.0.1 --port 8080 # OR make sse-server 2. Run the local communication client in terminal two: bash uv run python minimal_client/client.py "http://127.0.0.1:8080/sse" # OR make sse-client To halt both processes, terminate the client first (`Ctrl+C`), then the server.

Standard Input/Output (Stdio) Protocol

Configure Claude Desktop for Stdio communication: { "mcpServers": { "UNS_MCP": { "command": "ABSOLUTE/PATH/TO/.local/bin/uv", "args": [ "--directory", "ABSOLUTE/PATH/TO/YOUR-UNS-MCP-REPO/uns_mcp", "run", "server.py" ] } } } Alternatively, use the local client with the server script directly: bash uv run python minimal_client/client.py uns_mcp/server.py

Minimal Client Environmental Settings

Fine-tune the behavior of the minimal client via environment variables: - LOG_LEVEL="ERROR": Filters out verbose output, presenting only critical user messages. - CONFIRM_TOOL_USE='false': Bypasses the pre-execution confirmation step. Caution: Disabling confirmation may lead to unintended execution of resource-intensive or destructive workflow operations.

Diagnostic Utilities

Anthropic provides the MCP Inspector tool for interactive debugging and testing of your locally hosted server. Execute the following command to launch the UI:

mcp dev uns_mcp/server.py

Within the Inspector, you can configure environment variables (pointing to your local setup) in the left panel and test the implemented tool capabilities under the 'tools' section.

To log the exact parameters passed to UnstructuredClient methods, set the environment variable DEBUG_API_REQUESTS=false (Note: Setting to true disables detailed request logging). Logs detailing function call parameters are written to files named unstructured-client-{date}.log.

Enhancing Client Access via Terminal Integration

Integration with @wonderwhy-er/desktop-commander grants the minimal client direct terminal interaction. WARNING: This elevates the client's access level, granting potential access to private file systems.

Install the prerequisite package: bash npx @wonderwhy-er/desktop-commander setup

Launch the client with the necessary extra argument:

bash uvx run python minimal_client/client.py "http://127.0.0.1:8080/sse" "@wonderwhy-er/desktop-commander@^0.2.11"

OR

make sse-client-terminal

Tool Context Management Considerations

• If the client must utilize the update_workflow tool, the create_workflow tool must also be present in the context, as the update mechanism requires the complete, prior configuration schema (it does not function as a JSON patch utility).

Known Operational Limitations

• Workflow Updates: The update_workflow tool mandates that the entirety of the workflow configuration be supplied. It performs a full replacement, not a partial update (patch). The client must first retrieve the current state via get_workflow_info if partial modification is intended.

Release Notes

Feature additions, fixes, and enhancements will be logged chronologically in CHANGELOG.md. Pre-release versions should adhere to the 0.x.x-dev format preceding a stable version increment.

Debugging Pitfalls

• Error: spawn ENOENT: This signals that the specified <command> executable is either not installed or not resolvable within the current system PATH environment variable.
• Action: Verify installation or supply the absolute file system path to the executable within your configuration file (e.g., replace python with /usr/bin/python3).

WIKIPEDIA_NOTE: XMLHttpRequest (XHR) is a browser API structure leveraging JavaScript objects to facilitate HTTP requests to a server subsequent to page rendering. XHR is foundational to Ajax. Historically, page navigation relied on standard form submissions/links; XHR enabled dynamic updates. Microsoft Outlook developers conceived the underlying principle in 2000, first implemented in IE 5 (1999) using ActiveXObject. By IE 7 (2006), the standardized XMLHttpRequest identifier became universally adopted across major browser engines (Gecko 2002, Safari 1.2 2004, Opera 8.0 2005).

The W3C released the first Working Draft specification for XHR on April 5, 2006. Level 2, introducing event progress monitoring, cross-origin capabilities, and byte stream handling, was published in 2008 but later merged back into the primary specification by late 2011. Development transitioned to WHATWG in 2012, maintaining a living document using Web IDL.

Standard Usage Sequence: 1. Instantiate the XHR object via its constructor. 2. Invoke the open() method to define request type, target resource URI, and operational mode (synchronous/asynchronous). 3. For asynchronous operations, attach an event listener to monitor state transitions. 4. Trigger the request payload dispatch using send(). 5. Process responses within the listener. State 4 signifies 'done', with data typically residing in responseText.

Beyond basic steps, XHR allows custom header injection, data uploading via send(), response parsing (e.g., JSON to JS object), stream processing, request abortion, and timeout specification.