MCP server for Firecrawl

MCP server for Firecrawl is an application that provides AI assistants with web scraping, data extraction, and search capabilities via the standard Model Context Protocol (MCP). The server is deployed on a virtual machine in Yandex Cloud.

Key Features and Benefits

• Yandex AI Studio Integration. The MCP server integrates with AI Studio MCP Servers — connect it to your AI agents in just a few clicks.
• Web Page Scraping. Extract page content as Markdown, HTML, or structured data.
• Site Crawling. Crawl entire websites with configurable depth and filters.
• Web Search. Search the web and retrieve content of matching pages through a unified interface.
• Structured Data Extraction. Extract data by JSON schema using LLM-based parsing.
• Self-hosted Firecrawl Support. Connect to a self-hosted Firecrawl instance with a custom CA certificate.
• Bearer Token Authorization. All requests to the MCP server are protected with a Bearer token stored in Yandex Lockbox.
• Secure Secret Storage. The Firecrawl API key is stored in Yandex Lockbox and retrieved at startup.
• Standard MCP Protocol. Uses the open Model Context Protocol for AI model interaction.
• HTTP with SSE Transport. The MCP server is accessible on port 8000 (HTTP with SSE).
• Minimal Resources. The application runs on a virtual machine with minimal configuration (2 vCPU, 2 GB RAM).
• Docker Containerization. The server runs in Docker containers (MCP + supergateway + nginx auth proxy).

1. Obtain a Firecrawl API key: sign up at firecrawl.dev and create a key in your account. Alternatively, deploy a self-hosted Firecrawl instance.
2. Create a secret in Yandex Lockbox to store the Firecrawl API key. The secret must contain an api_key key with the API key value.
3. Create a secret in Yandex Lockbox to store the Bearer token for authorization. The secret must contain a bearer_token key with an arbitrary token value.
4. Create a service account with the roles compute.editor, iam.serviceAccounts.admin, lockbox.payloadViewer, vpc.admin (the admin role includes all the specified roles).
5. In the management console select the Cloud Apps service.
6. In the left panel, select Application Store.
7. Select MCP Firecrawl and click the Use button.
8. Specify:
   • Application name.
   • (Optional) Application description.
   • The service account created earlier.
   • The cloud subnet where the virtual machine will be deployed.
   • Lockbox secret ID containing the Bearer token for authorization.
   • Lockbox secret ID containing the Firecrawl API key.
   • (Optional) URL of a self-hosted Firecrawl instance. Defaults to https://api.firecrawl.dev.
   • (Optional) Certificate from Yandex Certificate Manager for HTTPS connection to a self-hosted Firecrawl.
9. Click the Install button and wait for the application to install.
10. After installation, the MCP server will be available at http://<vm-ip>:8000.
11. Configure your AI client (Claude Desktop, Cursor, etc.) to connect to the MCP server, specifying the Authorization: Bearer <your-token> header.

• Scraping and extracting web page content using an AI assistant (Claude, ChatGPT).
• Crawling entire sites with configurable depth to build a knowledge base.
• Searching the web and automatically extracting content from matching pages.
• Extracting structured data from web pages by JSON schema.
• Building AI agents that work with up-to-date information from the web.

Yandex Cloud technical support is available 24/7. The types of requests you can submit and the appropriate response time depend on your pricing plan. You can switch to the paid support plan in the management console. You can learn more about the technical support terms here.