Databricks MCP Service Connector

A Model Context Protocol (MCP) implementation designed to bridge LLMs with the Databricks data platform via its programmatic interfaces. It permits natural language instructions to drive data manipulation and workflow management.

Core Capabilities

• Execution of arbitrary SQL expressions against designated Databricks SQL endpoints.
• Inventory retrieval for all defined Databricks workflows/jobs.
• Status retrieval for specific, identified Databricks jobs.
• Detailed metadata fetching for any Databricks job definition.

Prerequisites for Operation

• A runtime environment utilizing Python version 3.7 or newer.
• Access to a functional Databricks account, requiring:
  • A valid Personal Access Token (PAT).
  • The endpoint configuration details for a SQL warehouse.
  • Sufficient IAM permissions to trigger queries and inspect job resources.

Setup Procedure

1. Clone the source code repository.

2. Establish and activate a dedicated virtual environment (strongly advised): bash python -m venv .venv source .venv/bin/activate # Use .venv\Scripts\activate on Windows

3. Install necessary Python packages: bash pip install -r requirements.txt

4. Populate a configuration file named .env in the root directory with required secrets and endpoints:

   DATABRICKS_HOST=your-databricks-instance.cloud.databricks.com DATABRICKS_TOKEN=your-personal-access-token DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/your-warehouse-id

5. Verify connectivity using the provided verification script (optional): bash python test_connection.py

Acquiring Databricks Credentials

1. Instance Host: The fully qualified domain name of your Databricks instance (e.g., acme-corp.cloud.databricks.com).
2. Authentication Token: Generate a PAT from the user settings interface:
   • Navigate to User Profile (top-right corner).
   • Select the "Developer" section.
   • Locate and manage "Access tokens".
   • Create a new token and ensure it is saved securely upon generation.
3. SQL Endpoint Path: Obtain this from the Databricks SQL Warehouses management panel. It is part of the connection details for the specific cluster you intend to query.

Launching the Service

Initialize the MCP gateway server: bash python main.py

Testing the running MCP service can be performed using the official inspector utility:

bash npx @modelcontextprotocol/inspector python3 main.py

Exposable MCP Function Signatures

The service exposes the following callable interfaces:

1. run_sql_query(sql: str) - Submits a query string for execution on the active Databricks SQL endpoint.
2. list_jobs() - Fetches an index of all orchestrated workflows present in the current Databricks tenant.
3. get_job_status(job_id: int) - Returns the current runtime state for a job identified by its integer ID.
4. get_job_details(job_id: int) - Retrieves comprehensive configuration and metadata for a specified job ID.

Interaction Examples (LLM Context)

When integrated with an MCP-aware language model, usage patterns include:

• "Provide a catalog of schemas available for querying."
• "Execute DML to update the quantity field in the sales ledger."
• "Enumerate the scheduled batch processing workflows."
• "What is the current execution state of workflow identifier 123?"
• "Display the full configuration for job 456."

Troubleshooting Common Issues

Connectivity Failures

• Confirm the Databricks host URL is precise and lacks any leading protocol specifiers (e.g., http://).
• Verify that the targeted SQL warehouse resource is actively running.
• Validate the PAT scope and validity against required resource permissions.
• Re-run the diagnostic script: python test_connection.py

Security Posture

• The Databricks PAT grants elevated access to workspace resources; treat it as a sensitive secret.
• The .env file must be excluded from all source code repositories (e.g., via .gitignore).
• It is best practice to scope the PAT to the minimum necessary privileges.
• Deploy and operate this service within a hardened, secure execution environment.