Use this interface to crawl a specific website using our crawlers.

This Endpoint requires an ToS;DR Issued API Key!

Authentication is done through the apikey Parameter.

Endpoints

Europe

https://api.tosdr.org/crawl/v1/eu/
CODE

United States

https://api.tosdr.org/crawl/v1/us/
CODE

Specific Regions

Europe West Cluster

https://api.tosdr.org/crawl/v1/eu-west/
CODE

Europe Central Cluster

https://api.tosdr.org/crawl/v1/eu-central/
CODE

US East Cluster

https://api.tosdr.org/crawl/v1/us-east/
CODE

US West Cluster

https://api.tosdr.org/crawl/v1/us-west/
CODE

URL Parameters

Parameter

Type

Description

apikey

String

ToS;DR Issued Api Key

url

string

The url to crawl

xpath

string

The XPath to use, defaults to //body


Repository

https://github.com/tosdr/crawler.tosdr.org

Error JSON Schema
{
    "$schema": "http://json-schema.org/draft-06/schema#",
    "$ref": "#/definitions/Welcome",
    "definitions": {
        "Welcome": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "error": {
                    "type": "boolean"
                },
                "message": {
                    "$ref": "#/definitions/Message"
                }
            },
            "required": [
                "error",
                "message"
            ],
            "title": "Welcome"
        },
        "Message": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "name": {
                    "type": "string"
                },
                "crawler": {
                    "type": "string"
                },
                "remoteStacktrace": {
                    "type": "string"
                }
            },
            "required": [
                "crawler",
                "name",
                "remoteStacktrace"
            ],
            "title": "Message"
        }
    }
}
JS
Success JSON Schema
{
    "$schema": "http://json-schema.org/draft-06/schema#",
    "$ref": "#/definitions/Welcome",
    "definitions": {
        "Welcome": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "error": {
                    "type": "boolean"
                },
                "message": {
                    "$ref": "#/definitions/Message"
                },
                "raw_html": {
                    "type": "string"
                },
                "imagedata": {
                    "type": "string"
                }
            },
            "required": [
                "error",
                "imagedata",
                "message",
                "raw_html"
            ],
            "title": "Welcome"
        },
        "Message": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "name": {
                    "type": "null"
                },
                "crawler": {
                    "type": "string"
                },
                "remoteStacktrace": {
                    "type": "null"
                }
            },
            "required": [
                "crawler",
                "name",
                "remoteStacktrace"
            ],
            "title": "Message"
        }
    }
}
JS