Skip to main content
POST
/
knowledge-base
/
discover-sitemap
Discover sitemap URLs from a website for web scraping
curl --request POST \
  --url https://api.cleartalk.ai/knowledge-base/discover-sitemap \
  --header 'Content-Type: application/json' \
  --header 'x-api-key: <api-key>' \
  --data '
{
  "url": "https://www.example.com",
  "type": "web"
}
'

Documentation Index

Fetch the complete documentation index at: https://docs.cleartalk.ai/llms.txt

Use this file to discover all available pages before exploring further.

Authorizations

​
x-api-key
string
header
required

API Key for authentication

Body

application/json
​
url
string
required

The base URL of the website to discover sitemap URLs from

Example:

"https://www.example.com"

​
type
string
default:web

Type of discovery

Example:

"web"

Response

Sitemap URLs discovered successfully