> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cleartalk.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Discover sitemap URLs from a website for web scraping



## OpenAPI

````yaml post /knowledge-base/discover-sitemap
openapi: 3.0.0
info:
  title: ClearTalk API
  description: The ClearTalk API Documentation
  version: '1.0'
  contact:
    name: Developer Support
    email: devsupport@cleartalk.ai
servers:
  - url: https://api.cleartalk.ai
    description: ClearTalk AI API Server
security: []
tags: []
paths:
  /knowledge-base/discover-sitemap:
    post:
      tags:
        - Knowledge Base
      summary: Discover sitemap URLs from a website for web scraping
      operationId: KnowledgeBaseController_discoverSitemapUrls
      parameters: []
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DiscoverSitemapDto'
      responses:
        '200':
          description: Sitemap URLs discovered successfully
        '400':
          description: Invalid URL or discovery failed
        '500':
          description: KB_ERROR - Failed to discover sitemap URLs
      security:
        - apiKey: []
components:
  schemas:
    DiscoverSitemapDto:
      type: object
      properties:
        url:
          type: string
          description: The base URL of the website to discover sitemap URLs from
          example: https://www.example.com
        type:
          type: string
          description: Type of discovery
          example: web
          default: web
      required:
        - url
  securitySchemes:
    apiKey:
      type: apiKey
      in: header
      name: x-api-key
      description: API Key for authentication

````