🐉 Drive Automated Browser Operations with Visionary LLMs and Computer Vision 🐉

Skyvern empowers the automation of web-based operational sequences utilizing advanced Large Language Models (LLMs) coupled with sophisticated computer vision capabilities. It furnishes a streamlined Application Programming Interface (API) endpoint designed to completely automate manual business routines across a vast spectrum of web properties, effectively superseding brittle or error-prone conventional automation methods.

Previous methodologies for web automation necessitated the development of bespoke scripts tailored for specific sites, frequently dependent on Document Object Model (DOM) analysis and XPath queries, which invariably failed upon any modification to the site's structure.

In contrast to solely relying on code-prescribed XPath interactions, Skyvern leverages Vision-enabled LLMs to learn and execute interactions within the digital environment.

Operational Principle

Skyvern draws foundational inspiration from the Task-Driven autonomous agent paradigm pioneered by projects such as BabyAGI and AutoGPT, adding a significant enhancement: the capacity to manipulate websites via contemporary browser automation frameworks like Playwright.

Skyvern employs a distributed collection of specialized agents to interpret the webpage structure, subsequently formulating and deploying navigational and interactive actions:

This architectural choice confers several distinct advantages:

1. Zero-Shot Adaptability: Skyvern can effectively function on previously unencountered websites by mapping visual cues to the required operational steps, entirely eliminating the need for pre-coded instructions.
2. Structural Resilience: The system exhibits robustness against website presentation shifts, as its execution logic is independent of static selectors like XPaths.
3. Workflow Generalization: A single defined operational sequence can be applied successfully across a diverse set of web destinations because the agent reasons dynamically about the necessary interactions.
4. Advanced Reasoning: Skyvern utilizes LLMs to deduce complex situational context for interactions. Examples include:
   1. Determining eligibility for an auto insurance quote based on inferred data (e.g., reasoning that receiving a license at age 16 implies eligibility to drive at 18).
   2. Performing competitive analysis by equating product listings with slight variations in size across different retailers (e.g., recognizing a 22 oz Arnold Palmer at 7/11 as identical to a 23 oz listing at Gopuff, discounting minor size discrepancies).

A comprehensive technical exposition detailing the architecture is accessible here.

Demonstration

https://github.com/user-attachments/assets/5cab4668-e8e2-4982-8551-aab05ff73a7f

Performance Metrics & Assessment

Skyvern currently achieves state-of-the-art (SOTA) outcomes on the WebBench benchmark, registering an accuracy of 64.4%. The full technical report and evaluation findings are published here

Efficacy in WRITE Operations (e.g., data entry, credential submission, file retrieval)

Skyvern demonstrates superior performance in tasks categorized as WRITE operations (such as automated form population, secure login procedures, and downloading digital assets), which are highly relevant for applications adjacent to Robotic Process Automation (RPA).

Initialization Guide

Skyvern Managed Cloud Service

Skyvern Cloud offers a fully provisioned, managed environment, abstracting away infrastructure concerns. This service facilitates concurrent execution of numerous Skyvern instances and includes integrated protections against bot detection, a robust proxy network, and CAPTCHA resolution capabilities.

To commence usage, kindly visit app.skyvern.com and establish an account.

Local Installation and Execution

Prerequisites: - Python 3.11.x (Compatibility confirmed for 3.12; 3.13 support pending) - NodeJS & NPM

For Windows Users, additional dependencies: - Rust - VS Code configured with C++ development tools and the Windows SDK

1. Installation

pip install skyvern

2. Initial Service Launch

This command is essential for first-time setup, including database schema creation and migrations.

skyvern quickstart

3. Task Execution

User Interface (Recommended Path)

Initiate the Skyvern backend service and its accompanying User Interface (assuming the database is operational):

skyvern run all

Access the web interface at http://localhost:8080 to manage and initiate tasks.

Programmatic Execution

from skyvern import Skyvern

skyvern = Skyvern()
task = await skyvern.run_task(prompt="Find the top post on hackernews today")
print(task)

Skyvern will launch a browser instance to execute the task, closing it upon completion. The task history is viewable at http://localhost:8080/history

You possess the ability to direct task execution toward different endpoints:

from skyvern import Skyvern

# Targeting Skyvern Cloud
skyvern = Skyvern(api_key="SKYVERN API KEY")

# Targeting a locally hosted Skyvern instance
skyvern = Skyvern(base_url="http://localhost:8000", api_key="LOCAL SKYVERN API KEY")

task = await skyvern.run_task(prompt="Find the top post on hackernews today")
print(task)

Advanced Configuration Options

Utilizing Your Own Chrome Instance

⚠️ IMPORTANT NOTE: Due to security changes in Chrome 136, direct connection to the default user_data_dir is blocked. For initial connection, Skyvern will create a mirrored copy of your default user data directory located at ./tmp/user_data_dir. ⚠️

1. Via Python Scripting

from skyvern import Skyvern

# Example path for macOS. Adjust for your OS.
browser_path = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
skyvern = Skyvern(
    base_url="http://localhost:8000",
    api_key="YOUR_API_KEY",
    browser_path=browser_path,
)
task = await skyvern.run_task(
    prompt="Find the top post on hackernews today",
)

1. Via Skyvern Service Environment Variables Set the following variables in your .env file:

# Specify path to your Chrome executable. Example for Mac.
CHROME_EXECUTABLE_PATH="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
BROWSER_TYPE=cdp-connect

Restart the Skyvern service (skyvern run all) and execute tasks via the UI or code.

Connecting to a Remote Browser Session

Acquire the required Chrome DevTools Protocol (CDP) connection URL and supply it to Skyvern:

from skyvern import Skyvern

skyvern = Skyvern(cdp_url="your cdp connection url")
task = await skyvern.run_task(
    prompt="Find the top post on hackernews today",
)

Enforcing Structured Output

Achieve consistent response formats by supplying the data_extraction_schema argument:

from skyvern import Skyvern

skyvern = Skyvern()
task = await skyvern.run_task(
    prompt="Find the top post on hackernews today",
    data_extraction_schema={
        "type": "object",
        "properties": {
            "title": {
                "type": "string",
                "description": "The title of the top post"
            },
            "url": {
                "type": "string",
                "description": "The URL of the top post"
            },
            "points": {
                "type": "integer",
                "description": "Number of points the post has received"
            }
        }
    }
)

Utility Commands for Troubleshooting

# Initiate the Skyvern Backend Server Independently
skyvern run server

# Initiate the Skyvern User Interface
skyvern run ui

# Query the current operational status of the Skyvern service
skyvern status

# Halt all running Skyvern components
skyvern stop all

# Halt the Skyvern User Interface
skyvern stop ui

# Halt the Skyvern Backend Server Independently
skyvern stop server

Docker Compose Deployment

1. Ensure Docker Desktop is installed and active.
2. Verify no other PostgreSQL instance is active locally (Use docker ps to check).
3. Clone the repository and navigate into the root directory.
4. Execute skyvern init llm to generate the necessary .env configuration file, which will be injected into the Docker image.
5. Populate the required LLM provider credentials within the docker-compose.yml file. (If targeting a remote host, configure the UI container's server IP address appropriately in the file).
6. Deploy the stack using the following command: bash docker compose up -d
7. Access the interactive dashboard at http://localhost:8080.

Crucial Note: Only one PostgreSQL container can occupy port 5432. If migrating from a CLI-managed PostgreSQL instance to Docker Compose, you must decommission the prior container first: bash docker rm -f postgresql-container

If database errors arise during Docker operation, use docker ps to identify and inspect any running PostgreSQL containers.

Core Skyvern Capabilities

Task Definition (Skyvern Tasks)

Tasks represent the atomic unit of execution within Skyvern. Each task constitutes a single instruction set directing Skyvern to navigate a web environment and achieve a specific objective. Tasks mandate the specification of a url and a primary prompt, with optional parameters including a data schema for output structuring and error codes to define specific termination conditions.

Sequential Operations (Skyvern Workflows)

Workflows provide the mechanism for linking multiple discrete tasks into a unified, coherent operational sequence.

For instance, automating the bulk download of invoices created after a specific date could involve a workflow that first navigates to the invoice portal, applies a date filter, extracts the list of relevant invoices, and then iterates through each item to trigger individual downloads.

Another use case involves automating e-commerce purchases: a sequence might first locate the desired product, add it to the shopping cart, validate the cart's contents, and finally proceed through the checkout sequence.

Supported workflow constructs include: 1. Browser Task Execution 1. Sequential Browser Actions 1. Data Extraction Steps 1. State Validation Blocks 1. Iterative For Loops 1. Local File Processing 1. Email Dispatch 1. Natural Language Prompts 1. Direct HTTP Request Blocks 1. Custom Code Injection 1. Uploading Assets to Remote Storage 1. (Upcoming) Conditional Branching

Real-time Browser Mirroring (Livestreaming)

Skyvern allows for the streaming of the active browser viewport directly to the user's local machine, providing full transparency into the agent's current activity. This feature is invaluable for diagnostic purposes, understanding navigation logic, and enabling on-the-fly intervention.

Form Population Capabilities

Skyvern possesses inherent proficiency in interacting with and completing input fields on web forms. Supplying data via the navigation_goal parameter enables the agent to semantically interpret the required information and populate the form fields accurately.

Information Retrieval (Data Extraction)

Skyvern is adept at extracting structured data from web pages. Users can define a data_extraction_schema within the initial instruction prompt to dictate the precise JSON structure required for the extracted output, ensuring adherence to the specified schema.

Asset Acquisition (File Downloading)

The system reliably handles file downloads from web resources. All retrieved files are automatically forwarded to configured block storage solutions, and access to these assets is provided through the management UI.

Access Control Management (Authentication)

Skyvern supports numerous authentication protocols to facilitate automation behind protected access layers. For specialized authentication flows, please engage with the team via email or Discord.

🔐 Two-Factor Authentication (2FA) Support (TOTP)

Skyvern incorporates methods to handle workflows requiring two-factor verification, supporting: 1. Time-based One-Time Password (TOTP) via QR codes (e.g., Google Authenticator, Authy) 1. Email-transmitted 2FA codes 1. SMS-delivered 2FA codes

🔐 Further details on 2FA integration are available here.

Credential Manager Synchronization

Skyvern currently integrates with the following password management solutions: - [x] Bitwarden - [ ] 1Password - [ ] LastPass

Model Context Protocol (MCP) Adherence

Skyvern is compliant with the Model Context Protocol (MCP), enabling interoperability with any LLM that supports this specification. The relevant MCP documentation can be found here

Integration with Workflow Orchestration Tools

Skyvern seamlessly integrates with popular automation platforms such as Zapier, Make.com, and N8N to extend automated sequences: * Zapier * Make.com * N8N

🔐 Refer to this link for detailed information on 2FA handling.

Real-World Application Scenarios

We actively track and showcase practical implementations of Skyvern in production environments. We encourage contributors to submit new use cases via Pull Requests!

Multi-Site Invoice Retrieval Automation

Schedule a live demonstration

Job Application Process Automation

Witness the capability

Manufacturing Procurement Material Automation

Witness the capability

Governmental Website Registration and Data Submission

Witness the capability

Automated Contact Form Completion

Witness the capability

Insurance Quotation Generation Across Providers (Multilingual Support)

Witness the capability

Witness the capability

Developer Setup Guide

Ensure you have uv installed prior to proceeding. 1. Execute this command to establish the virtual environment (.venv): bash uv sync --group dev 2. Perform initial service configuration: bash uv run skyvern quickstart 3. Launch your browser and navigate to http://localhost:8080 to begin using the platform. The Skyvern Command Line Interface (CLI) is compatible across Windows, WSL, macOS, and Linux operating systems.

Comprehensive Documentation

More exhaustive documentation is accessible via our 📕 docs page. Should any aspect remain unclear or incomplete, please report an issue or contact us directly via email or Discord.

Supported Language Models (LLMs)

Provider	Supported Models
OpenAI	gpt4-turbo, gpt-4o, gpt-4o-mini
Anthropic	Claude 3 (Haiku, Sonnet, Opus), Claude 3.5 (Sonnet)
Azure OpenAI	All GPT models. Enhanced performance with multimodal models (azure/gpt4-o)
AWS Bedrock	Anthropic Claude 3 (Haiku, Sonnet, Opus), Claude 3.5 (Sonnet)
Gemini	Gemini 2.5 Pro and flash, Gemini 2.0
Ollama	Any model hosted locally via Ollama
OpenRouter	Access models through OpenRouter
OpenAI-compatible	Any custom API endpoint conforming to OpenAI's specification (via liteLLM)

Configuration Environment Variables

OpenAI

Variable	Purpose	Type	Example Value
ENABLE_OPENAI	Activates registration of OpenAI models	Boolean	true, false
OPENAI_API_KEY	Your secret OpenAI key	String	sk-1234567890
OPENAI_API_BASE	Optional custom API endpoint URL	String	https://openai.api.base
OPENAI_ORGANIZATION	Optional OpenAI organization identifier	String	your-org-id

Recommended Model Identifiers (LLM_KEY): OPENAI_GPT4O, OPENAI_GPT4O_MINI, OPENAI_GPT4_1, OPENAI_O4_MINI, OPENAI_O3

Anthropic

Variable	Purpose	Type	Example Value
ENABLE_ANTHROPIC	Activates registration of Anthropic models	Boolean	true, false
ANTHROPIC_API_KEY	Your secret Anthropic key	String	sk-1234567890

Recommended Model Identifiers (LLM_KEY): ANTHROPIC_CLAUDE3.5_SONNET, ANTHROPIC_CLAUDE3.7_SONNET, ANTHROPIC_CLAUDE4_OPUS, ANTHROPIC_CLAUDE4_SONNET

Azure OpenAI

Variable	Purpose	Type	Example Value
ENABLE_AZURE	Activates registration of Azure OpenAI models	Boolean	true, false
AZURE_API_KEY	Azure deployment API key	String	sk-1234567890
AZURE_DEPLOYMENT	Name of the Azure OpenAI Deployment	String	skyvern-deployment
AZURE_API_BASE	Base URL for the Azure endpoint	String	https://skyvern-deployment.openai.azure.com/
AZURE_API_VERSION	Specific Azure API version	String	2024-02-01

Recommended Model Identifier (LLM_KEY): AZURE_OPENAI

AWS Bedrock

Variable	Purpose	Type	Example Value
ENABLE_BEDROCK	Activates AWS Bedrock models. Requires correct AWS configuration.	Boolean	true, false

Recommended Model Identifiers (LLM_KEY): BEDROCK_ANTHROPIC_CLAUDE3.7_SONNET_INFERENCE_PROFILE, BEDROCK_ANTHROPIC_CLAUDE4_OPUS_INFERENCE_PROFILE, BEDROCK_ANTHROPIC_CLAUDE4_SONNET_INFERENCE_PROFILE

Gemini

Variable	Purpose	Type	Example Value
ENABLE_GEMINI	Activates registration of Gemini models	Boolean	true, false
GEMINI_API_KEY	Your secret Gemini API key	String	your_google_gemini_api_key

Recommended Model Identifiers (LLM_KEY): GEMINI_2.5_PRO_PREVIEW, GEMINI_2.5_FLASH_PREVIEW

Ollama

Variable	Purpose	Type	Example Value
ENABLE_OLLAMA	Registers models hosted via Ollama	Boolean	true, false
OLLAMA_SERVER_URL	Address of your local Ollama instance	String	http://host.docker.internal:11434
OLLAMA_MODEL	Specific model name to load from Ollama	String	qwen2.5:7b-instruct

Note: Vision processing is not yet supported by the Ollama integration.

Recommended Model Identifier (LLM_KEY): OLLAMA

OpenRouter

Variable	Purpose	Type	Example Value
ENABLE_OPENROUTER	Registers models available via OpenRouter	Boolean	true, false
OPENROUTER_API_KEY	Your secret OpenRouter API key	String	sk-1234567890
OPENROUTER_MODEL	Designated OpenRouter model name	String	mistralai/mistral-small-3.1-24b-instruct
OPENROUTER_API_BASE	Optional custom API base URL	String	https://api.openrouter.ai/v1

Recommended Model Identifier (LLM_KEY): OPENROUTER

OpenAI-Compatible Endpoints

Variable	Purpose	Type	Example Value
ENABLE_OPENAI_COMPATIBLE	Registers a custom endpoint following the OpenAI structure	Boolean	true, false
OPENAI_COMPATIBLE_MODEL_NAME	Model name for the custom endpoint	String	yi-34b, gpt-3.5-turbo, mistral-large, etc.
OPENAI_COMPATIBLE_API_KEY	API credential for the custom endpoint	String	sk-1234567890
OPENAI_COMPATIBLE_API_BASE	Base URL for the custom endpoint	String	https://api.together.xyz/v1, http://localhost:8000/v1, etc.
OPENAI_COMPATIBLE_API_VERSION	Optional API version for compatibility	String	2023-05-15
OPENAI_COMPATIBLE_MAX_TOKENS	Optional limit on output tokens	Integer	4096, 8192, etc.
OPENAI_COMPATIBLE_TEMPERATURE	Optional sampling temperature setting	Float	0.0, 0.5, 0.7, etc.
OPENAI_COMPATIBLE_SUPPORTS_VISION	Optional flag indicating vision capability	Boolean	true, false

Supported LLM Key for this configuration: OPENAI_COMPATIBLE

General LLM Parameters

Variable	Purpose	Type	Example Value
LLM_KEY	Primary model identifier to utilize	String	Refer to keys listed above
SECONDARY_LLM_KEY	Model identifier for auxiliary/mini agents	String	Refer to keys listed above
LLM_CONFIG_MAX_TOKENS	Global override for maximum context window size	Integer	128000

Future Development Plan (Roadmap)

This outlines our planned development trajectory for the coming months. Suggestions for features are highly welcomed via email or Discord.

• [x] Source Code Release - Open sourcing the core component logic.
• [x] Task Chaining - Enabling sequential execution of multiple Skyvern invocations.
• [x] Enriched Contextual Awareness - Enhancing environmental comprehension by injecting relevant surrounding element labels into the text prompt context.
• [x] Operational Cost Reduction - Stability improvements and cost minimization via optimization of the context tree structure.
• [x] Modernized Interface - Transitioning from the Streamlit UI to a robust React-based frontend for job initiation.
• [x] Visual Workflow Editor - Introduction of a graphical interface for constructing and analyzing automated sequences.
• [x] Live Browser View Streaming - Implementing real-time viewport transmission to the user's browser (integrated with the new UI).
• [x] Run History Visualization - Replacing the Streamlit history view with a React component for detailed past execution review.
• [X] Autonomous Workflow Generation ("Observer" Mode) - Allowing Skyvern to build new workflows automatically during web navigation.
• [x] Prompt Caching Layer - Implementing a caching mechanism for LLM interactions to significantly reduce operational expenditure.
• [x] Standardized Evaluation Suite - Integrating Skyvern with public benchmark testing to monitor quality progression.
• [ ] Enhanced Debug Mode - Enabling agents to present action plans for approval before execution, facilitating step-by-step debugging and prompt iteration.
• [ ] Browser Extension - Developing a Chrome extension for direct user interaction (including voice control and task saving).
• [ ] User Action Recording - Capability for Skyvern to observe a user completing a task and subsequently generate the corresponding workflow blueprint.
• [ ] Interactive Stream Mirroring - Allowing users to input commands directly into the live browser feed for real-time intervention (e.g., manual handling of sensitive data entry).
• [ ] LLM Observability Integration - Incorporating tools for back-testing prompt adjustments against known datasets and visualizing long-term performance metrics.
• [x] Langchain Utility - Developing a direct integration within langchain_community to utilize Skyvern as a callable tool.

Collaboration

We encourage contributions through Pull Requests and issue reports! Reach out via email or Discord. Consult our contribution guide and review the relevant "Help Wanted" issues to get started.

To obtain a high-level structural overview of the codebase, learn about building upon it, or resolve usage queries, consult Code Sage.

Data Collection (Telemetry)

By default, Skyvern gathers minimal operational statistics to inform usage trends. To deactivate this data submission, set the environment variable SKYVERN_TELEMETRY to false.

Licensing

Skyvern is provisioned as open-source software under the AGPL-3.0 License, with the exception of proprietary anti-bot mechanisms reserved for the managed cloud service.

For any licensing inquiries or concerns, please reach out to us; we are prepared to assist.

Star Trajectory

WIKIPEDIA: Business management tools encompass the entire suite of systems, applications, controls, computational solutions, and established methodologies utilized by organizations to navigate evolving market conditions, maintain competitive parity, and drive enhanced organizational performance.

== General Perspective == Tools exist across every functional department within an enterprise, classifiable by management aspect: planning, process optimization, record-keeping, human resources management, strategic decision support, and performance monitoring. A functional taxonomy typically encompasses:

Tools for data input and verification across all operational units. Instrumentation for governing and refining core business workflows. Platforms for aggregating data and facilitating executive decision-making. Contemporary management tooling has undergone profound transformation over the last decade due to rapid technological advancements, making the selection of optimal tools for a given corporate context increasingly complex. This dynamic is fueled by the incessant pursuit of cost reduction, sales expansion, deeper customer understanding, and precise fulfillment of market demands. Consequently, leadership must adopt a strategic perspective on business management solutions rather than merely adopting the newest available technology. Over-reliance on tools without appropriate customization frequently leads to systemic instability. Business management tools demand careful selection followed by tailored adaptation to organizational requirements, not the reverse.

== Prevalent Tools (2013 Survey Highlight) == A 2013 survey by Bain & Company detailed global business tool adoption, reflecting regional needs shaped by market dynamics. The top ten instruments identified included:

Strategic planning frameworks Customer Relationship Management (CRM) suites Employee feedback mechanisms Benchmarking analysis Balanced Scorecard implementation Core competency identification Outsourcing strategy management Organizational Change Management programs Supply Chain Optimization Mission and Vision articulation Market Segmentation methodologies Total Quality Management (TQM) frameworks

== Enterprise Software Applications == Software—collections of computer programs—used by business personnel to execute diverse operational tasks are termed business software or business applications. These tools are designed to augment productivity, quantify performance metrics, and ensure accuracy in organizational functions. The evolution moved from initial Management Information Systems (MIS) to Enterprise Resource Planning (ERP) systems, subsequently incorporating CRM functionality, culminating in today's cloud-centric business management environments. While a direct correlation exists between IT investment and corporate outcomes, value maximization hinges on two factors: effective implementation and judicious selection and customization of the required tooling.

== Tools for Small and Medium Enterprises (SMEs) == SME-focused tools are vital as they offer pathways to resource conservation, particularly in areas such as...