Spider

Spider is the fastest crawler. It converts any website into pure HTML, markdown, metadata or text while enabling you to crawl with custom actions using AI.

Overview

Spider allows you to use high performance proxies to prevent detection, caches AI actions, webhooks for crawling status, scheduled crawls etc… This guide shows how to crawl/scrape a website using Spider and loading the LLM-ready documents with SpiderLoader in LanghChain.

Setup

Get your own Spider API key on spider.cloud.

Usage

Here’s an example of how to use the SpiderLoader: Spider offers two scraping modes scrape and crawl. Scrape only gets the content of the url provided while crawl gets the content of the url provided and crawls deeper following subpages.

import { SpiderLoader } from "@langchain/community/document_loaders/web/spider";

const loader = new SpiderLoader({
  url: "https://spider.cloud", // The URL to scrape
  apiKey: process.env.SPIDER_API_KEY, // Optional, defaults to `SPIDER_API_KEY` in your env.
  mode: "scrape", // The mode to run the crawler in. Can be "scrape" for single urls or "crawl" for deeper scraping following subpages
  // params: {
  //   // optional parameters based on Spider API docs
  //   // For API documentation, visit https://spider.cloud/docs/api
  // },
});

const docs = await loader.load();

Additional Parameters

See the Spider documentation for all the available params.

Edit the source of this page on GitHub.

Connect these docs programmatically to Claude, VSCode, and more via MCP for real-time answers.

⌘I

Popular Providers

General integrations

RAG integrations

Overview

Setup

Usage

Additional Parameters

​Overview

​Setup

​Usage

​Additional Parameters

Overview

Setup

Usage

Additional Parameters