MCP server for Firecrawl

MCP server for Firecrawl is an application that provides AI assistants with web scraping, data extraction, and search capabilities via the standard Model Context Protocol (MCP). The server is deployed on a virtual machine in Yandex Cloud.

Key Features and Benefits

• Yandex AI Studio Integration. Connect the MCP server under MCP servers and use it with your AI agents.
• Web Page Scraping. Extract page content as Markdown, HTML, or structured data.
• Site Crawling. Crawl entire websites with configurable depth and filters.
• Web Search. Search the web and retrieve content of matching pages through a unified interface.
• Structured Data Extraction. Extract data by JSON schema using LLM-based parsing.
• Self-hosted Firecrawl Support. Connect to a self-hosted Firecrawl instance with a custom certificate authority (CA) certificate.
• Bearer Token Authorization. Access to the MCP server requires a Bearer token stored in Yandex Lockbox. The Bearer token controls access but does not encrypt HTTP traffic.
• Secure Secret Storage. The Firecrawl API key is stored in Yandex Lockbox and retrieved at startup.
• Standard MCP Protocol. Uses the open Model Context Protocol for AI model interaction.
• Streamable HTTP Transport. The MCP server is available over Streamable HTTP on port 8000.
• Minimal Resources. The application runs on a virtual machine with minimal configuration (2 vCPU, 2 GB RAM).
• Docker Containerization. The server runs in Docker containers (the MCP server and an nginx proxy with Bearer token authorization).

1. Obtain a Firecrawl API key: sign up at firecrawl.dev and create a key in your account. Alternatively, deploy a self-hosted Firecrawl instance.
2. Create a secret in Yandex Lockbox to store the Firecrawl API key. The secret must contain an api_key key with the API key value.
3. Create a secret in Yandex Lockbox to store the Bearer token for authorization. The secret must contain a bearer_token key with an arbitrary token value.
4. Create a service account for deploying the application and assign it the roles compute.editor, iam.serviceAccounts.admin, vpc.admin, and resource-manager.admin on the deployment folder.
5. Assign the installer service account the lockbox.admin role separately on the Bearer token secret and the external service credentials secret. This role is required to read and modify access permissions on the selected secrets; no folder-level Lockbox role is required. This also applies to secrets in another folder. During installation, the application creates a dedicated VM service account and grants it lockbox.payloadViewer only on the selected secrets.
6. The template also grants the VM service account certificate-manager.certificates.downloader on the deployment folder. The installer needs resource-manager.admin to create this IAM binding, even when no custom certificate is selected. If using a certificate, select one from the deployment folder; for a certificate in another folder, grant the VM account access to it separately.
7. In the management console select the Cloud Apps service.
8. In the left panel, select Application Store.
9. Select MCP Firecrawl and click the Use button.
10. Specify:
    • Application name.
    • (Optional) Application description.
    • The service account created earlier.
    • The cloud subnet where the virtual machine will be deployed.
    • Lockbox secret ID containing the Bearer token for authorization.
    • Lockbox secret ID containing the Firecrawl API key.
    • (Optional) URL of a self-hosted Firecrawl instance. If the URL is not specified, the SDK uses the default Firecrawl endpoint: https://api.firecrawl.dev.
    • Certificate mode. Select Disabled if no additional CA certificate is required. Select Enabled and specify a certificate from Yandex Certificate Manager if the HTTPS connection to the self-hosted Firecrawl instance requires a custom CA certificate. This certificate is used for the outbound connection to Firecrawl and does not enable HTTPS on the MCP server.
    • External access to the MCP server (optional; disabled by default, enable for connections through the public IP address).
    • SSH access mode. Select Disabled to avoid adding an SSH user and exposing port 22 externally. Select Enabled, then specify a username and public SSH key to allow SSH connections. When SSH is enabled, the security group allows connections to port 22 from any IPv4 address; use additional network restrictions if needed.
11. Click the Install button and wait for the application to install.
12. After installation, review the values under Application data:
    • MCP server type — Streamable HTTP.
    • VPC network ID — the network ID of the selected subnet.
    • MCP server URL (internal) — http://<private-ip>:8000/mcp. Use this URL for clients that have network access to the VM’s internal IP address.
    • MCP server URL (external) — http://<nat-ip>:8000/mcp. The VM always receives a public NAT address, and the security group allows connections to port 8000 from any IPv4 address only when external access is enabled. The external URL uses plain HTTP: the Bearer token controls access but does not encrypt traffic. We recommend using the internal URL over a secure network. If you need the external URL, send traffic only through a trusted network or secure tunnel.
13. Configure your AI client to connect to the MCP server, specifying the Authorization: Bearer <your-token> header.

Connecting the MCP server to Yandex AI Studio

1. In the management console open AI Studio and go to Agent Core → MCP servers.
2. Click Create MCP server and select the required MCP server under MCP servers from Yandex Cloud Marketplace.
3. Specify the server name and select an existing application instance, or create a new one by following the deployment instructions. When deploying a new server, the Save button becomes available only after the MCP server has been fully deployed and started.
4. After you select the application, the MCP server’s Network and Service account settings are filled in automatically.
5. Click Create. After the connection has been verified successfully, select the required tools and click Save. The MCP server will be registered in AI Studio and available for use by AI agents.

• Scraping and extracting web page content using an AI assistant.
• Crawling entire sites with configurable depth to build a knowledge base.
• Searching the web and automatically extracting content from matching pages.
• Extracting structured data from web pages by JSON schema.
• Building AI agents that work with up-to-date information from the web.

Yandex Cloud technical support is available 24/7. The types of requests you can submit and the appropriate response time depend on your pricing plan. You can switch to the paid support plan in the management console. You can learn more about the technical support terms here.