Databricks MCP Access Gateway

A specialized Model Completion Protocol (MCP) service layer engineered to interface with the Databricks platform. This gateway exposes core Databricks operations—including cluster lifecycle management, job execution, and notebook interaction—through the standardized MCP communication schema, allowing AI agents to manage data workloads programmatically.

Core Capabilities

• MCP Compliance: Fully adheres to the Model Completion Protocol specification for standardized LLM interaction.
• Databricks Service Bridging: Acts as an abstraction layer over the underlying Databricks RESTful API.
• Tool Exposure: Registers Databricks functionalities as callable MCP agents/tools.
• Asynchronous Processing: Utilizes asyncio to ensure high-throughput, non-blocking operations.

Exposed Tools Inventory

The following Databricks primitives are accessible via this gateway:

• cluster_inventory: Retrieves a listing of all provisioned Databricks clusters.
• provision_cluster: Initiates the creation of a new Databricks compute resource.
• decommission_cluster: Shuts down (terminates) a specified Databricks cluster.
• fetch_cluster_details: Obtains comprehensive configuration and status for a target cluster.
• reactivate_cluster: Resumes operations on a previously halted cluster instance.
• job_manifest: Returns a catalog of defined Databricks automated workflows (jobs).
• trigger_workflow: Executes a specified Databricks job run.
• workspace_artifact_listing: Enumerates files and directories within a defined workspace path.
• download_notebook: Exports the source code of a Databricks notebook artifact.
• dbfs_path_enumeration: Lists contents (files/folders) found at a specified Databricks File System (DBFS) location.
• execute_databricks_sql: Submits and runs a SQL query against an active endpoint.

Initial Setup & Deployment

Dependencies

• Requires a runtime environment of Python version 3.10 or superior.
• Strongly recommends utilizing the uv package manager for dependency resolution.

Installation Procedure

1. Install uv (If necessary):

bash # For Unix-like systems curl -LsSf https://astral.sh/uv/install.sh | sh

# For Windows (PowerShell environment) irm https://astral.sh/uv/install.ps1 | iex

Ensure your shell session is reloaded post-installation.

1. Acquire Source Code: bash git clone https://github.com/JustTryAI/databricks-mcp-server.git cd databricks-mcp-server

2. Environment Configuration via uv: bash # Establish an isolated virtual environment uv venv

# Activation (Windows) ..venv\Scripts\activate

# Activation (Linux/Mac) source .venv/bin/activate

# Install primary dependencies in editable mode uv pip install -e .

# Install supplementary development requirements uv pip install -e ".[dev]"

1. Configure Authentication Credentials: Set the necessary environmental parameters for API access:

bash # Windows Command Prompt/Batch set DATABRICKS_HOST=https://your-databricks-instance.azuredatabricks.net set DATABRICKS_TOKEN=your-personal-access-token

# Linux/Mac Shell export DATABRICKS_HOST=https://your-databricks-instance.azuredatabricks.net export DATABRICKS_TOKEN=your-personal-access-token

Alternatively, populate the provided .env.example template and use that file for configuration loading.

Launching the MCP Service

To initiate the gateway and begin listening for MCP connections:

bash

Launch via Windows convenience script

.\start_mcp_server.ps1

Launch via Linux/Mac convenience script

./start_mcp_server.sh

These scripts invoke the core server executable located within the scripts subdirectory. Upon success, the service will be ready for external MCP client communication.

Direct execution path:

bash

Execute directly on Windows

.\scripts\start_mcp_server.ps1

Execute directly on Linux/Mac

./scripts/start_mcp_server.sh

Utility Scripts for Resource Inspection

Companion scripts are bundled for quick introspection of your Databricks environment:

bash

Query and display current cluster fleet status

uv run scripts/show_clusters.py

Display the hierarchical layout of notebooks

uv run scripts/show_notebooks.py

Directory Layout Overview

databricks-mcp-server/ ├── src/ # Core application logic │ ├── init.py # Package initialization marker │ ├── main.py # Primary package execution point │ ├── main.py # Server bootstrap module │ ├── api/ # Databricks communication clients │ ├── core/ # Centralized processing components │ ├── server/ # MCP layer implementation │ │ ├── databricks_mcp_server.py # The main server instance handler │ │ └── app.py # FastAPI instance used primarily for diagnostics/testing │ └── cli/ # Command-line interface utilities ├── tests/ # Unit and integration test suites ├── scripts/ # Shell wrappers and utility executables │ ├── start_mcp_server.ps1 # Windows server start sequence │ ├── run_tests.ps1 # Test execution harness (Windows) │ ├── show_clusters.py # Cluster introspection utility │ └── show_notebooks.py # Notebook listing utility ├── examples/ # Demonstrative usage scripts ├── docs/ # Project documentation source └── pyproject.toml # Project metadata and dependency specification

Refer to project_structure.md for an exhaustive breakdown of components.

Development Standards

Code Style Adherence

• Formatting: Python code strictly adheres to PEP 8 guidelines; maximum line width enforced at 100 characters.
• Indentation: Standard four (4) space indentation; tabs are strictly prohibited.
• String Literals: Double quotes (") must be used universally.
• Documentation: All classes, methods, and functions are documented using Google-style docstrings.
• Typing: Mandatory type annotations for all source code, excluding test modules.

Quality Assurance (Linting)

Verification of code quality is managed by the following tools:

bash

Execute the full suite of static analysis checks

uv run pylint src/ tests/ uv run flake8 src/ tests/ uv run mypy src/

Testing Framework

The test suite leverages pytest. Execution is streamlined via wrapper scripts:

bash

Comprehensive test execution via convenience script

.\scripts\run_tests.ps1

Execute tests and generate a coverage report

.\scripts\run_tests.ps1 -Coverage

Targeted test execution with verbosity and coverage measurement

.\scripts\run_tests.ps1 -Verbose -Coverage tests/test_clusters.py

Alternatively, direct pytest invocation:

bash

Execute all tests

uv run pytest tests/

Execute tests with detailed source coverage reporting

uv run pytest --cov=src tests/ --cov-report=term-missing

The project mandates a minimum source code coverage threshold of 80%.

Documentation Strategy

• Sphinx is utilized for generating formal API documentation, residing in docs/api.
• Comprehensive Google-style docstrings are embedded throughout the codebase.
• Practical implementation patterns are detailed in the examples/ subdirectory.

Usage Samples

To observe the gateway in action, execute the provided examples:

bash

Direct interaction script execution

uv run examples/direct_usage.py

Example demonstrating MCP client interaction

uv run examples/mcp_client_usage.py

Contribution Guidelines

We welcome external contributions. Kindly adhere to the following steps:

1. Ensure modifications conform to the established coding style.
2. Introduce appropriate tests for any novel functionality.
3. Update relevant documentation artifacts.
4. Confirm all existing tests pass successfully prior to submission.

Licensing

This software is distributed under the terms of the MIT License; see the LICENSE file for specifics.

WIKIPEDIA: Cloud computing defines the delivery of on-demand IT resources—ranging from applications to data centers—over the internet with pay-as-you-go pricing. The concept is based on a scalable pool of shared, configurable computing assets available via a network interface, often encapsulated by the term 'the cloud'.

== Key Attributes (NIST Definition) == In 2011, the U.S. National Institute of Standards and Technology (NIST) established five fundamental properties characterizing cloud systems. These attributes, verbatim from NIST guidance, define the service model:

Self-Service Provisioning: Consumers gain the ability to procure computational capabilities (e.g., processing time, network storage) unilaterally and automatically, bypassing manual intervention from the provider. Ubiquitous Network Access: Resources are accessible across a network using standardized protocols, supporting a diverse array of client devices (mobiles, desktops, tablets). Resource Aggregation: Provider resources form a shared pool supporting numerous clients via a multi-tenant architecture, with capacity dynamically allocated based on fluctuating demand. Elastic Scalability: Capabilities can be rapidly expanded or contracted, often automatically, to match fluctuating workload requirements. From the user's perspective, capacity appears virtually limitless and instantaneously available. Measured Utility: System usage is automatically tracked, controlled, and optimized at a suitable level of abstraction (e.g., storage volume, computation cycles, data throughput). This metering provides essential transparency regarding consumption for both supplier and customer. By 2023, ISO had further elaborated upon these core tenets.

== Historical Context ==

The conceptual foundation for modern cloud services dates back to the 1960s, emerging from the development of time-sharing systems and remote job entry (RJE). This era focused on centralizing massive computing mainframes managed by operators, optimizing hardware utilization, and expanding access to computational cycles for a broader user base. The specific 'cloud' visual metaphor for networked services was introduced around 1994 by General Magic in the context of mobile agents operating within the Telescript framework. David Hoffman, a communications specialist at General Magic, is credited with adopting the metaphor, which had a long history in telecommunications diagrams. The term 'cloud computing' gained broader industry recognition following a 1996 business strategy document circulated within Compaq Computer Corporation, signaling ambitions toward next-generation internet services.