🐉 Orchestrate Web Interactions via Generative Models and Visual Perception 🐉

WebWeaver facilitates the automation of complex, multi-step routines across web interfaces by leveraging Large Language Models in conjunction with advanced computer vision capabilities. It furnishes a standardized programmatic interface (API) designed to execute manual procedures across a vast array of digital destinations, effectively superseding fragile, selector-dependent automation frameworks.

Conventional digital automation methodologies necessitated the creation of bespoke scripts tethered to specific website structures, typically relying on Document Object Model (DOM) traversal or fragile XPath expressions. These scripts inevitably failed upon the slightest alteration to the target site's presentation layer.

In contrast to these code-centric locator strategies, WebWeaver prioritizes visual comprehension, empowering Vision LLMs to interpret and act upon visual elements dynamically.

Operational Paradigm

WebWeaver's architecture draws inspiration from the goal-directed autonomous agent frameworks such as BabyAGI and AutoGPT. The key differentiator is the seamless integration of high-fidelity browser manipulation capabilities provided by libraries like Playwright.

WebWeaver deploys a coordinated federation of specialized agents tasked with analyzing the web context, formulating execution plans, and enacting the required steps:

This methodology confers several substantial benefits:

1. Zero-Shot Adaptability: The system can successfully operate on previously unencountered web properties by visually mapping on-screen components to necessary workflow actions, entirely obviating the need for pre-coded site-specific logic.
2. Resilience to UI Drift: Since the system eschews fixed XPaths or rigid selectors for interaction, it maintains functionality even when website layouts undergo structural revisions.
3. Cross-Platform Task Transfer: A singular defined procedure can be reliably applied across numerous dissimilar websites, as the system dynamically reasons through the necessary interaction sequences.
4. Advanced Reasoning: The integration of LLMs enables sophisticated situational comprehension. For instance, it can deduce that a user query about an 'Arnold Palmer 22 oz can at 7/11' corresponds to a '23 oz can at Gopuff' during competitor analysis, recognizing the slight size variance as negligible in context.

A comprehensive technical whitepaper detailing the engineering behind this system is accessible here.

Demonstration

https://github.com/user-attachments/assets/5cab4668-e8e2-4982-8551-aab05ff73a7f

Efficacy & Benchmarking

WebWeaver achieves state-of-the-art results on the WebBench benchmark, registering a 64.4% success rate. The validation methodology and findings are detailed in the technical report available here

Performance in WRITE Operations (e.g., data entry, credential submission, file acquisition)

WebWeaver leads in executing WRITE tasks (such as form completion, secure login, and asset downloading), which are crucial for applications adjacent to Robotic Process Automation (RPA).

Initial Setup Guide

WebWeaver Cloud Service

WebWeaver Cloud offers a fully managed deployment, abstracting away all infrastructure concerns. This service allows for concurrent execution of multiple automation instances and includes integrated defenses against bot detection, a managed proxy network, and CAPTCHA resolution utilities.

To commence usage, visit app.skyvern.com to register for an account.

Local Installation & Execution

Prerequisites: - Python 3.11.x (Compatible with 3.12; 3.13 support pending) - NodeJS & NPM

Windows Specific Dependencies: - Rust Toolchain - Visual Studio Code with necessary C++ development components and Windows SDK

1. Install WebWeaver

bash pip install skyvern

2. Initialize System

This command handles crucial first-time setup, including database schema provisioning and migrations.

bash skyvern quickstart

3. Execute Automation Job

User Interface (Recommended)

Start the core service and the graphical interface (ensure the database is active):

bash skyvern run all

Access the UI at http://localhost:8080 to initiate tasks.

Programmatic Execution

python from skyvern import Skyvern

agent = Skyvern() task_result = await agent.run_task(prompt="Identify the leading news story on Hacker News today") print(task_result)

WebWeaver launches a temporary browser instance to execute the task, closing it upon completion. The outcome is logged and viewable at http://localhost:8080/history

You can direct tasks to different endpoints: python from skyvern import Skyvern

Utilizing WebWeaver Cloud infrastructure

agent = Skyvern(api_key="YOUR_SKYVERN_CLOUD_API_KEY")

Connecting to a self-hosted local service

agent = Skyvern(base_url="http://localhost:8000", api_key="LOCAL_API_KEY")

task_result = await agent.run_task(prompt="Identify the leading news story on Hacker News today") print(task_result)

Advanced Configuration

Controlling a Local Chrome Instance

⚠️ CAUTION: Due to security changes in Chrome version 136, connections to the default user profile directory are blocked. WebWeaver mitigates this by creating a copy of your default profile at ./tmp/user_data_dir upon the initial connection to your local browser instance. ⚠️

1. Via Python Scripting python from skyvern import Skyvern

Example path for macOS browser location

browser_exec_path = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" agent = Skyvern( base_url="http://localhost:8000", api_key="YOUR_AUTH_TOKEN", browser_path=browser_exec_path, ) task_result = await agent.run_task( prompt="Identify the leading news story on Hacker News today", )

1. Via WebWeaver Service Configuration

Set the following environment variables within your .env file: bash

Specify Chrome executable location (Mac example)

CHROME_EXECUTABLE_PATH="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" BROWSER_TYPE=cdp-connect

Then, restart the main service (skyvern run all) and execute the task via the UI or code.

Remote Browser Connection via CDP

If you possess a Chrome DevTools Protocol (CDP) connection URL, pass it directly to the client:

python from skyvern import Skyvern

agent = Skyvern(cdp_url="your remote cdp connection url") task_result = await agent.run_task( prompt="Identify the leading news story on Hacker News today", )

Enforcing Structured Output Schema

Obtain predictable results by utilizing the data_extraction_schema parameter: python from skyvern import Skyvern

agent = Skyvern() task_result = await agent.run_task( prompt="Extract the key details from the Hacker News front page", data_extraction_schema={ "type": "object", "properties": { "title": { "type": "string", "description": "The headline of the most prominent article" }, "url": { "type": "string", "description": "The hyperlink for the extracted article" }, "score": { "type": "integer", "description": "The current upvote count associated with the post" } } } )

Essential Debugging Utilities

bash

Initiate the WebWeaver Backend API only

skyvern run server

Launch the User Interface component

skyvern run ui

Query the service status

skyvern status

Halt all running components

skyvern stop all

Shut down the User Interface process

skyvern stop ui

Halt the Backend API only

skyvern stop server

Docker Compose Deployment

1. Ensure Docker Desktop is operational.
2. Verify that no other PostgreSQL instance is active on port 5432 (use docker ps to check).
3. Clone the source repository and move into the root directory.
4. Execute skyvern init llm to generate the necessary .env configuration file, which will be incorporated into the Docker build context.
5. Populate the LLM credentials within the docker-compose.yml file. (If targeting a remote host, adjust the UI container's server IP definition in the file).

6. Deploy using the command: bash docker compose up -d

7. Access the platform at http://localhost:8080.

Critical Note: Port 5432 can only host one PostgreSQL instance. If you transition from the CLI-managed database to Docker Compose, you must forcibly remove the existing container first: bash docker rm -f postgresql-container

If database errors manifest during Docker execution, use docker ps to identify and inspect any active PostgreSQL containers.

WebWeaver Capabilities

Automation Assignments (Tasks)

Assignments represent the core atomic unit within WebWeaver. Each assignment initiates a single, directed web navigation sequence to achieve a specified objective.

Assignments mandate the provision of a target url and a descriptive prompt. Optionally, you can define an output data schema for structured return values or specify error codes to dictate precise termination conditions.

Operational Sequences (Workflows)

Workflows enable the composition of multiple sequential assignments into a unified, complex unit of work.

For instance, to batch-download all financial statements generated after the first day of January, a workflow could be constructed to: navigate to the statements portal, apply a date filter, extract the resultant list of documents, and then iterate through this list, invoking a download action for each item.

Another application involves e-commerce orchestration: a sequence might involve navigating to a product page, adding the item to a digital shopping cart, verifying the cart's contents, and finally proceeding through the checkout procedure.

Supported workflow construction elements include: 1. Browser Execution Block 1. Agentic Action Definition 1. Data Structuring 1. Integrity Verification 1. Iterative Looping 1. Document Analysis 1. Electronic Mail Dispatch 1. Textual Instruction Blocks 1. Direct HTTP Communication 1. Custom Script Insertion 1. Asset Upload to Remote Storage 1. (Upcoming) Conditional Branching Logic

Real-Time Viewport Streaming

WebWeaver includes functionality to broadcast the browser's active screen rendering directly to your local machine. This feature is invaluable for diagnostic purposes, allowing users to observe the AI's navigation in real-time and permitting manual intervention if needed.

Input Field Population

WebWeaver possesses intrinsic competence in inputting data into HTML forms. Supplying information via the navigation_goal parameter allows the agent to semantically understand the required inputs and populate the form fields accurately.

Information Retrieval (Data Extraction)

The system is equally adept at extracting specific datasets from web pages.

You can mandate the precise JSON structure of the required output by embedding a data_extraction_schema directly within the primary request prompt, leveraging jsonc format. WebWeaver's final response will strictly adhere to the schema provided.

Asset Acquisition

WebWeaver manages the downloading of files initiated from web pages. All retrieved assets are automatically routed to configured block storage solutions, with access provided via the operational interface.

Credential Handling

WebWeaver supports various authentication strategies to streamline automation behind restricted access points. For integration methods not immediately apparent, please contact us via email or our Discord channel.

🔐 Two-Factor Authentication (TOTP) Capabilities

Support for multiple 2FA mechanisms is integrated, facilitating automated workflows requiring secondary verification:

Examples include: 1. Time-based One-Time Password (TOTP) via QR code scanning (e.g., Google Authenticator, Authy) 1. Email-mediated verification codes 1. SMS-delivered verification codes

🔐 Further details on 2FA integration are available here.

Password Manager Synchronization

WebWeaver currently offers compatibility with the following credential management tools: - [x] Bitwarden - [ ] 1Password - [ ] LastPass

Model Context Protocol (MCP) Compliance

WebWeaver adheres to the Model Context Protocol (MCP), ensuring compatibility with any LLM framework that supports this standard interface. Refer to the MCP specification documentation here.

Integration with Workflow Orchestrators (Zapier / Make.com / N8N)

WebWeaver provides connectors for major automation platforms: * Zapier * Make.com * N8N

🔐 Detailed information regarding 2FA integration protocols can be found here.

Practical Deployment Scenarios

We encourage the community to share instances of WebWeaver being deployed in production environments. We welcome contributions documenting your use cases via PRs!

High-Volume Invoice Retrieval Across Diverse Portals

Schedule a demonstration to witness this capability live.

Automating the Digital Job Application Funnel

See a live demonstration

Streamlining Material Acquisition for Manufacturing Sectors

💡 Observe the process

Navigating and Interacting with State/Federal Web Portals for Registration or Data Submission

💡 Observe the process

Automated Submission to Unstructured 'Contact Us' Forms

💡 Observe the process

Generating Insurance Quotes from Various Providers, Multilingually

💡 Observe the process

💡 Observe the process

Developer Setup Instructions

Ensure you have the uv package manager installed. 1. Execute this command to bootstrap the virtual environment (.venv): bash uv sync --group dev

1. Run the initial service configuration: bash uv run skyvern quickstart

2. Access the platform interface at http://localhost:8080. The WebWeaver Command Line Interface supports environments running Windows, WSL, macOS, and Linux.

Comprehensive Manuals

More detailed technical documentation is hosted on our 📕 documentation portal. Should you find any section unclear or lacking, please raise an issue or contact our team via email or Discord.

Supported Language Models

Provider	Accessible Models	Description
OpenAI	gpt4-turbo, gpt-4o, gpt-4o-mini	Leading proprietary models.
Anthropic	Claude 3 (Haiku, Sonnet, Opus), Claude 3.5 (Sonnet)	High-context window models.
Azure OpenAI	All GPT variants. Superior performance with multimodal LLMs (e.g., azure/gpt4-o)	Cloud-hosted deployment.
AWS Bedrock	Anthropic Claude 3 series, Claude 3.5 Sonnet	AWS native access.
Gemini	Gemini 2.5 Pro/Flash, Gemini 2.0	Google's multimodal offerings.
Ollama	Any model executable via Ollama	Local, self-hosted inference.
OpenRouter	Models accessible via OpenRouter gateway	Unified access to various open/closed models.
OpenAI-compatible	Endpoints matching OpenAI's API structure (via liteLLM)	Flexible API mapping.

Environment Configuration Variables

OpenAI Integration

Variable	Purpose	Type	Example Value
ENABLE_OPENAI	Activate OpenAI model registration	Boolean	true, false
OPENAI_API_KEY	Required OpenAI authentication credential	String	sk-1234567890
OPENAI_API_BASE	Alternate API endpoint URL, optional	String	https://openai.api.base
OPENAI_ORGANIZATION	OpenAI account affiliation identifier, optional	String	your-org-id

Recommended Aliases: OPENAI_GPT4O, OPENAI_GPT4O_MINI, OPENAI_GPT4_1, OPENAI_O4_MINI, OPENAI_O3

Anthropic Integration

Variable	Purpose	Type	Example Value
ENABLE_ANTHROPIC	Activate Anthropic model registration	Boolean	true, false
ANTHROPIC_API_KEY	Required Anthropic API token	String	sk-1234567890

Recommended Aliases: ANTHROPIC_CLAUDE3.5_SONNET, ANTHROPIC_CLAUDE3.7_SONNET, ANTHROPIC_CLAUDE4_OPUS, ANTHROPIC_CLAUDE4_SONNET

Azure OpenAI Integration

Variable	Purpose	Type	Example Value
ENABLE_AZURE	Activate Azure OpenAI registration	Boolean	true, false
AZURE_API_KEY	Azure deployment access key	String	sk-1234567890
AZURE_DEPLOYMENT	Name of the deployed model instance	String	skyvern-deployment
AZURE_API_BASE	Base URL for the Azure endpoint	String	https://skyvern-deployment.openai.azure.com/
AZURE_API_VERSION	API service version specification	String	2024-02-01

Recommended Alias: AZURE_OPENAI

AWS Bedrock Integration

Variable	Purpose	Type	Example Value
ENABLE_BEDROCK	Activate AWS Bedrock model registration. AWS credentials must be configured externally.	Boolean	true, false

Recommended Alias: BEDROCK_ANTHROPIC_CLAUDE3.7_SONNET_INFERENCE_PROFILE, BEDROCK_ANTHROPIC_CLAUDE4_OPUS_INFERENCE_PROFILE, BEDROCK_ANTHROPIC_CLAUDE4_SONNET_INFERENCE_PROFILE

Gemini Integration

Variable	Purpose	Type	Example Value
ENABLE_GEMINI	Activate Gemini model registration	Boolean	true, false
GEMINI_API_KEY	Google Gemini authentication key	String	your_google_gemini_api_key

Recommended Aliases: GEMINI_2.5_PRO_PREVIEW, GEMINI_2.5_FLASH_PREVIEW

Ollama Integration

Variable	Purpose	Type	Example Value
ENABLE_OLLAMA	Register locally hosted models via Ollama	Boolean	true, false
OLLAMA_SERVER_URL	Network address for the local Ollama service	String	http://host.docker.internal:11434
OLLAMA_MODEL	Identifier for the specific Ollama model to utilize	String	qwen2.5:7b-instruct

Recommended Alias: OLLAMA

Note: Current Ollama integration lacks native visual processing support.

OpenRouter Integration

Variable	Purpose	Type	Example Value
ENABLE_OPENROUTER	Register models accessible via OpenRouter	Boolean	true, false
OPENROUTER_API_KEY	OpenRouter access credential	String	sk-1234567890
OPENROUTER_MODEL	The specific OpenRouter model name to employ	String	mistralai/mistral-small-3.1-24b-instruct
OPENROUTER_API_BASE	OpenRouter endpoint base URL	String	https://api.openrouter.ai/v1

Recommended Alias: OPENROUTER

OpenAI-Compatible Endpoints

Variable	Purpose	Type	Example Value
ENABLE_OPENAI_COMPATIBLE	Register a custom endpoint conforming to OpenAI API specs	Boolean	true, false
OPENAI_COMPATIBLE_MODEL_NAME	Model identifier for the compatible service	String	yi-34b, gpt-3.5-turbo, mistral-large, etc.
OPENAI_COMPATIBLE_API_KEY	API key for the compatible service	String	sk-1234567890
OPENAI_COMPATIBLE_API_BASE	Base URL for the custom API endpoint	String	https://api.together.xyz/v1, http://localhost:8000/v1, etc.
OPENAI_COMPATIBLE_API_VERSION	API protocol version, optional	String	2023-05-15
OPENAI_COMPATIBLE_MAX_TOKENS	Limit for output token count, optional	Integer	4096, 8192, etc.
OPENAI_COMPATIBLE_TEMPERATURE	Sampling temperature control, optional	Float	0.0, 0.5, 0.7, etc.
OPENAI_COMPATIBLE_SUPPORTS_VISION	Indicates native vision modality support, optional	Boolean	true, false

Supported LLM Alias: OPENAI_COMPATIBLE

Global LLM Parameters

Variable	Purpose	Type	Example Value
LLM_KEY	The primary model identifier to utilize	String	Refer to supported aliases above
SECONDARY_LLM_KEY	The model identifier for subsidiary agents	String	Refer to supported aliases above
LLM_CONFIG_MAX_TOKENS	Overrides the default maximum context size for the LLM	Integer	128000

Development Trajectory

This outlines our planned feature development schedule for the coming months. Feedback and feature requests are actively solicited via email or our Discord server.

• [x] Open Source Release - Public availability of the core system source code.
• [x] Sequential Task Chaining - Enabling the definition of dependent, multi-step operations.
• [x] Enriched Context Injection - Improving semantic understanding by supplying contextually relevant labels alongside raw text input.
• [x] Operational Cost Reduction - Enhancing system stability and minimizing LLM invocation expenses through optimization of the context tree transmission.
• [x] Modernized UI - Transitioning from the Streamlit interface to a React-based application for job initiation.
• [x] Visual Workflow Designer - Introducing a graphical interface for constructing and analyzing execution sequences.
• [x] Live Browser Feed - Implementation of real-time Chrome viewport streaming integrated into the modernized UI.
• [x] Historical Run Visualization - React UI for viewing and analyzing outcomes of past assignments.
• [X] Autonomous Workflow Generation ('Observer' Mode) - Allowing WebWeaver to dynamically map interactions into formal workflows during navigation.
• [x] Prompt Result Caching - Implementing a mechanism to store and reuse LLM outputs for common steps, significantly cutting operational expenditure.
• [x] Standardized Benchmarking - Integrating the system with public evaluation suites to track quality evolution.
• [ ] Guided Debugging Mode - Implementing a 'Plan & Approve' mechanism where the agent requests confirmation before executing steps, aiding iterative refinement.
• [ ] Browser Extension Interface - Developing a Chrome extension for convenient task initiation, voice commands, and session saving.
• [ ] Action Recording Feature - Enabling WebWeaver to observe a human user completing a task and automatically synthesizing the corresponding workflow.
• [ ] Interactive Live Feed Control - Allowing users to dynamically influence the running browser session via the live stream (e.g., manual form entry for sensitive data).
• [ ] LLM Observability Integration - Incorporating tools for back-testing prompt variations against fixed datasets and visualizing performance metrics over time.
• [x] Langchain Tool Adapter - Providing a native integration within langchain_community to expose WebWeaver as a callable tool.

Collaboration

We welcome contributions in the form of pull requests and feature suggestions! Please review our contribution guidelines and check the "Help Wanted" issues to find starting points.

For architectural discussions, structure inquiries, or complex usage questions, consult Code Sage to gain a comprehensive understanding of the codebase organization.

Usage Telemetry

By default, WebWeaver gathers minimal telemetry data to inform development priorities. To globally disable this data transmission, set the environment variable SKYVERN_TELEMETRY to the value false.

Licensing Details

The core automation engine of WebWeaver is provided as open source under the AGPL-3.0 License. Note that specialized countermeasures against sophisticated bot detection mechanisms remain proprietary to the managed cloud offering.

For any licensing inquiries or clarifications, please get in touch.

Star Trajectory

WIKIPEDIA: Cloud computing is "a paradigm for enabling network access to a scalable and elastic pool of shareable physical or virtual resources with self-service provisioning and administration on-demand," according to ISO. It is commonly referred to as "the cloud".

== Characteristics == In 2011, the National Institute of Standards and Technology (NIST) identified five "essential characteristics" for cloud systems. Below are the exact definitions according to NIST:

On-demand self-service: "A consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with each service provider." Broad network access: "Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations)." Resource pooling: " The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand." Rapid elasticity: "Capabilities can be elastically provisioned and released, in some cases automatically, to scale rapidly outward and inward commensurate with demand. To the consumer, the capabilities available for provisioning often appear unlimited and can be appropriated in any quantity at any time." Measured service: "Cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the utilized service. By 2023, the International Organization for Standardization (ISO) had expanded and refined the list.

== History ==

The history of cloud computing extends to the 1960s, with the initial concepts of time-sharing becoming popularized via remote job entry (RJE). The "data center" model, where users submitted jobs to operators to run on mainframes, was predominantly used during this era. This was a time of exploration and experimentation with ways to make large-scale computing power available to more users through time-sharing, optimizing the infrastructure, platform, and applications, and increasing efficiency for end users. The "cloud" metaphor for virtualized services dates to 1994, when it was used by General Magic for the universe of "places" that mobile agents in the Telescript environment could "go". The metaphor is credited to David Hoffman, a General Magic communications specialist, based on its long-standing use in networking and telecom. The expression cloud computing became more widely known in 1996 when Compaq Computer Corporation drew up a business plan for future computing and the Internet. The company's ambition was to superch