mcp-web-content-extractor-py

An MCP backend service engineered for acquiring and restructuring internet content into diverse output formats. This utility offers robust capabilities for parsing data from web pages, crucially including support for dynamically generated content via browser emulation and media asset processing.

Core Toolset

This service exposes four distinct operational functions:

Tools

The primary capabilities provided by this server are:

• extract-plain-text: Retrieves unadulterated textual content directly from specified Uniform Resource Locators without invoking a rendering engine.
• Parameters:
  • url: The address of the web resource (supports text, JSON, XML, CSV, TSV outputs, etc.) (Mandatory)

• Optimal usage: When dealing with inherently structured data sources or contexts demanding high retrieval velocity.

• retrieve-rendered-html: Fetches the complete, browser-interpreted HTML output by employing a headless browsing environment.

• Parameters:
  • url: The target web page address (Mandatory)

• Indispensable for applications relying on modern single-page architecture (SPA) dependent on client-side scripting.

• to-markdown: Transforms the retrieved page data into semantically coherent, stylized Markdown text.

• Parameters:
  • url: The URL of the document to convert (Mandatory)

• Focuses on maintaining hierarchical organization while producing legible textual output.

• analyze-media-content: Leverages advanced AI processing to derive textual information from multimedia payloads.

• Parameters:
  • url: The URI pointing to the media item (e.g., images, video streams) (Mandatory)
• Employs computer vision and Optical Character Recognition (OCR) for visual data interpretation.
• Prerequisite: A valid OPENAI_API_KEY must be configured in the execution environment.
• Failure indication: Will report an error if the API key is absent or during media processing failures.

Integration Guide

Utilizing with Claude Desktop

To integrate this capability into Claude Desktop, incorporate the following server configuration block:

On macOS: ~/Library/Application\ Support/Claude/claude_desktop_config.json
On Windows: %APPDATA%/Claude/claude_desktop_config.json

"mcpServers": {
  "mcp-server-fetch-python": {
    "command": "uvx",
    "args": [
      "mcp-server-fetch-python"
    ]
  }
}

Environment Configuration

The subsequent environment variables offer configuration control:

• OPENAI_API_KEY: Essential prerequisite for activating the analyze-media-content utility, required for AI-driven visual data interpretation.
• PYTHONIOENCODING: Recommend setting this to "utf-8" if inconsistencies in character output encoding are observed.
• MODEL_NAME: Designates the specific large language model to be invoked. Default value is "gpt-4o".

"mcpServers": {
  "mcp-server-fetch-python": {
    "command": "uvx",
    "args": [
      "mcp-server-fetch-python"
    ],
    "env": {
        "OPENAI_API_KEY": "sk-****",
        "PYTHONIOENCODING": "utf-8",
        "MODEL_NAME": "gpt-4o",        
    }
  }
}

On-Premise Deployment

Alternatively, the service can be installed and operated locally:

git clone https://github.com/tatn/mcp-server-fetch-python.git
cd mcp-server-fetch-python
uv sync
uv build

Subsequently, update the Claude Desktop configuration file with this setup:

"mcpServers": {
  "mcp-server-fetch-python": {
    "command": "uv",
    "args": [
      "--directory",
      "path\to\mcp-server-fetch-python",  # Substitute with the actual cloned repository location
      "run",
      "mcp-server-fetch-python"
    ]
  }
}

Service Inspection and Debugging

You can initiate the MCP Inspector utility using npx via these commands:

npx @modelcontextprotocol/inspector uvx mcp-server-fetch-python

npx @modelcontextprotocol/inspector uv --directory path\to\mcp-server-fetch-python run mcp-server-fetch-python

WIKIPEDIA: XMLHttpRequest (XHR) is an Application Programming Interface represented as a JavaScript object, enabling the transmission of HTTP requests from a web browser to a remote server. These methods permit a browser-resident application to dispatch queries to the backend following page initialization and receive subsequent data. XHR is fundamental to the Ajax programming paradigm. Before Ajax, interaction with servers was predominantly achieved via standard hyperlink navigation and form submissions, which typically necessitated a full page reload.

== Historical Context == The foundational idea for XMLHttpRequest was formulated in 2000 by the development team behind Microsoft Outlook. This concept was first integrated into the Internet Explorer 5 browser release (1999). However, the initial syntax did not employ the XMLHttpRequest identifier; instead, developers utilized object instantiations like ActiveXObject("Msxml2.XMLHTTP") and ActiveXObject("Microsoft.XMLHTTP"). As of Internet Explorer 7 (2006), universal support for the standard XMLHttpRequest identifier became prevalent across all major browser engines, including Mozilla's Gecko (2002), Safari 1.2 (2004), and Opera 8.0 (2005).

=== Standardization Efforts === The World Wide Web Consortium (W3C) published the initial Working Draft specification for the XMLHttpRequest object on April 5, 2006. A Level 2 specification followed on February 25, 2008, introducing enhancements like progress monitoring, support for cross-origin requests, and byte stream handling. By the close of 2011, the Level 2 additions were merged back into the primary specification document. Development transitioned to the WHATWG at the end of 2012, which now maintains the specification as a continuous document using Web IDL definitions.

== Operational Flow == Constructing and executing a request via XMLHttpRequest generally involves these sequential programming steps:

1. Instantiate an XMLHttpRequest object via its constructor.
2. Invoke the "open" method to define the request method (e.g., GET, POST), specify the targeted endpoint resource, and choose between synchronous or asynchronous execution.
3. For asynchronous operations, register an event handler intended to be triggered upon state transitions.
4. Commence the transmission by calling the "send" method, optionally including payload data.
5. Process the response within the state change handler. Upon successful completion, the state transitions to 4, the "done" state, and the resulting data is typically available in the "responseText" property.

Beyond these foundational steps, XMLHttpRequest offers extensive configuration options for request control and response management. Custom HTTP headers can be appended to influence server behavior, and data can be uploaded dynamically via the "send" argument. The received response can be automatically parsed from JSON into native JavaScript structures or processed incrementally as data streams in, avoiding wait times for the complete payload. Furthermore, requests can be intentionally terminated or subjected to a defined timeout constraint.