Web resources

This page covers the webresources_* options, which connect CKEditor AI On-Premises to a web scraping endpoint you run. You need it only if users attach web pages to conversations or the agent fetches pages during a request.

Note

CKEditor AI On-Premises does not include a web scraping tool. You provide the tool, run it, and connect it through the endpoint below.

To connect your web scraping endpoint, set these options:

webresources_enabled – must be set to true.
webresources_endpoint – the URL of your gateway. The service sends the scrape requests to this URL.
webresources_request_timeout – request timeout in milliseconds (optional, default: 30000).

{
	"webresources_enabled": true,
	"webresources_endpoint": "[WEBRESOURCES_ENDPOINT]",
	"webresources_request_timeout": 30000
}

Request format

Your endpoint must accept POST requests with this JSON body:

{
  "url": "https://example.com/page-to-scrape"
}

Where:

url (required) – the URL of the page to scrape.

Response format

Your endpoint must return this response:

{
  "type": "text/html",
  "data": "<html>...</html>"
}

Where:

type (required) – the content type of the scraped data. Allowed values: text/html, text/markdown.
data (required) – the scraped website content.

Next steps

Web search – Connect the endpoint that runs the searches models request.
Network requirements – Check the outbound destinations a deployment needs.
Required configuration – Set the secret keys and the database, Redis, and storage options.