> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.rydhm.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.rydhm.ai/_mcp/server.

# Import a website for an expert

POST https://api.rydhm.ai/v1/agents/{id}/kb/crawl
Content-Type: application/json

Pages are fetched and indexed in the background. Bounded to 10 an hour per workspace.

Sessions: admins.

Needs the knowledge:write scope. A key without it gets 403 insufficient_scope. Routes not in this reference cannot be called with a key at all (403 not_available_to_api_keys).

Reference: https://docs.rydhm.ai/api-reference/knowledge/import-website

## Authentication

- `Authorization` header (bearer token, required) — Bearer authentication of the form `Bearer <token>`, where token is your auth token.

## Request

### Path parameters

- `id` (string, required) — The expert id.

### Body (application/json)

This endpoint expects a CrawlWrite.

- `url` (string, required) — The start page (https).
- `maxPages` (integer, optional) — How many pages at most.
- `includePaths` (list of string, optional)
- `excludePaths` (list of string, optional)

## Response

### 201

OK

- `id` (string, required) — A UUID.
- `url` (string, required)
- `status` (string, required)
- `maxPages` (integer, optional)
- `includePaths` (list of string, optional)
- `excludePaths` (list of string, optional)
- `createdAt` (datetime, optional)
- `updatedAt` (datetime, optional)

## Errors

### 401 Unauthorized Error

The key is missing, not valid, disabled or expired (error: invalid_api_key, api_key_disabled, api_key_expired).

- `statusCode` (integer, required)
- `message` (string, required)
- `error` (string, optional) — A stable code such as insufficient_scope, expert_not_allowed, rate_limited, plan_limit, payment_required.
- `code` (string, optional) — The stable code on the refusals that carry it here instead of in `error`: stale_edit (409 on an expert save).
- `requestId` (string, optional) — The X-Request-Id of the call, to quote to support.
- `upgradeHint` (ErrorUpgradeHint, optional) — On 402 payment_required and 403 plan_limit: what fixes it.
- `requiredScopes` (list of string, optional) — On 403 insufficient_scope: what the route needs.
- `missingScopes` (list of string, optional)
- `retryAfterSeconds` (integer, optional) — On 429: how long to wait.

### 403 Forbidden Error

The key lacks a scope (insufficient_scope), may not use this expert (expert_not_allowed), or the route is session-only (not_available_to_api_keys).

- `statusCode` (integer, required)
- `message` (string, required)
- `error` (string, optional) — A stable code such as insufficient_scope, expert_not_allowed, rate_limited, plan_limit, payment_required.
- `code` (string, optional) — The stable code on the refusals that carry it here instead of in `error`: stale_edit (409 on an expert save).
- `requestId` (string, optional) — The X-Request-Id of the call, to quote to support.
- `upgradeHint` (ErrorUpgradeHint, optional) — On 402 payment_required and 403 plan_limit: what fixes it.
- `requiredScopes` (list of string, optional) — On 403 insufficient_scope: what the route needs.
- `missingScopes` (list of string, optional)
- `retryAfterSeconds` (integer, optional) — On 429: how long to wait.

### 429 Too Many Requests Error

Rate limited; wait Retry-After seconds.

- `statusCode` (integer, required)
- `message` (string, required)
- `error` (string, optional) — A stable code such as insufficient_scope, expert_not_allowed, rate_limited, plan_limit, payment_required.
- `code` (string, optional) — The stable code on the refusals that carry it here instead of in `error`: stale_edit (409 on an expert save).
- `requestId` (string, optional) — The X-Request-Id of the call, to quote to support.
- `upgradeHint` (ErrorUpgradeHint, optional) — On 402 payment_required and 403 plan_limit: what fixes it.
- `requiredScopes` (list of string, optional) — On 403 insufficient_scope: what the route needs.
- `missingScopes` (list of string, optional)
- `retryAfterSeconds` (integer, optional) — On 429: how long to wait.

## Types

### ErrorUpgradeHint

On 402 payment_required and 403 plan_limit: what fixes it.

- `action` (enum, optional)
  - Allowed values: `top_up`, `upgrade`, `contact`
- `planCode` (string, optional) — The smallest plan that allows it, when action is upgrade.
- `href` (string, optional) — Where to go in the app.

## Examples

**Request**

```json
{
  "url": "string"
}
```

**Response**

```json
{
  "id": "string",
  "url": "string",
  "status": "string",
  "maxPages": 1,
  "includePaths": [
    "string"
  ],
  "excludePaths": [
    "string"
  ],
  "createdAt": "2024-01-15T09:30:00Z",
  "updatedAt": "2024-01-15T09:30:00Z"
}
```

**SDK Code**

```python
import requests

url = "https://api.rydhm.ai/v1/agents/id/kb/crawl"

payload = { "url": "string" }
headers = {
    "Authorization": "Bearer <apiKey>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.json())
```

```javascript
const url = 'https://api.rydhm.ai/v1/agents/id/kb/crawl';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <apiKey>', 'Content-Type': 'application/json'},
  body: '{"url":"string"}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```