> ## Documentation Index
> Fetch the complete documentation index at: https://langchain-5e9cc07a-preview-devupd-1765394015-eccef47.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Spider

[Spider](https://spider.cloud/?ref=langchainjs) is the [fastest](https://github.com/spider-rs/spider/blob/main/benches/BENCHMARKS.md#benchmark-results) crawler. It converts any website into pure HTML, markdown, metadata or text while enabling you to crawl with custom actions using AI.

## Overview

Spider allows you to use high performance proxies to prevent detection, caches AI actions, webhooks for crawling status, scheduled crawls etc...

This guide shows how to crawl/scrape a website using [Spider](https://spider.cloud/) and loading the LLM-ready documents with `SpiderLoader` in LanghChain.

## Setup

Get your own Spider API key on [spider.cloud](https://spider.cloud/).

## Usage

Here's an example of how to use the `SpiderLoader`:

Spider offers two scraping modes `scrape` and `crawl`. Scrape only gets the content of the url provided while crawl gets the content of the url provided and crawls deeper following subpages.

```typescript theme={null}
import { SpiderLoader } from "@langchain/community/document_loaders/web/spider";

const loader = new SpiderLoader({
  url: "https://spider.cloud", // The URL to scrape
  apiKey: process.env.SPIDER_API_KEY, // Optional, defaults to `SPIDER_API_KEY` in your env.
  mode: "scrape", // The mode to run the crawler in. Can be "scrape" for single urls or "crawl" for deeper scraping following subpages
  // params: {
  //   // optional parameters based on Spider API docs
  //   // For API documentation, visit https://spider.cloud/docs/api
  // },
});

const docs = await loader.load();
```

### Additional Parameters

See the [Spider documentation](https://spider.cloud/docs/api) for all the available `params`.

***

<Callout icon="pen-to-square" iconType="regular">
  [Edit the source of this page on GitHub.](https://github.com/langchain-ai/docs/edit/main/src/oss/javascript/integrations/document_loaders/web_loaders/spider.mdx)
</Callout>

<Tip icon="terminal" iconType="regular">
  [Connect these docs programmatically](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
</Tip>