GET /crawl/v1/
Use this interface to crawl a specific website using our crawlers.
This Endpoint requires an ToS;DR Issued API Key!
Authentication is done through the apikey
Parameter.
Endpoints
Europe
https://api.tosdr.org/crawl/v1/eu/
CODE
United States
https://api.tosdr.org/crawl/v1/us/
CODE
Specific Regions
Europe West Cluster
https://api.tosdr.org/crawl/v1/eu-west/
CODE
Europe Central Cluster
https://api.tosdr.org/crawl/v1/eu-central/
CODE
US East Cluster
https://api.tosdr.org/crawl/v1/us-east/
CODE
US West Cluster
https://api.tosdr.org/crawl/v1/us-west/
CODE
URL Parameters
Parameter | Type | Description |
---|---|---|
apikey | String | ToS;DR Issued Api Key |
url | string | The url to crawl |
xpath | string | The XPath to use, defaults to //body |
Repository
https://github.com/tosdr/crawler.tosdr.org
Error JSON Schema
{
"$schema": "http://json-schema.org/draft-06/schema#",
"$ref": "#/definitions/Welcome",
"definitions": {
"Welcome": {
"type": "object",
"additionalProperties": false,
"properties": {
"error": {
"type": "boolean"
},
"message": {
"$ref": "#/definitions/Message"
}
},
"required": [
"error",
"message"
],
"title": "Welcome"
},
"Message": {
"type": "object",
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
},
"crawler": {
"type": "string"
},
"remoteStacktrace": {
"type": "string"
}
},
"required": [
"crawler",
"name",
"remoteStacktrace"
],
"title": "Message"
}
}
}
JS
Success JSON Schema
{
"$schema": "http://json-schema.org/draft-06/schema#",
"$ref": "#/definitions/Welcome",
"definitions": {
"Welcome": {
"type": "object",
"additionalProperties": false,
"properties": {
"error": {
"type": "boolean"
},
"message": {
"$ref": "#/definitions/Message"
},
"raw_html": {
"type": "string"
},
"imagedata": {
"type": "string"
}
},
"required": [
"error",
"imagedata",
"message",
"raw_html"
],
"title": "Welcome"
},
"Message": {
"type": "object",
"additionalProperties": false,
"properties": {
"name": {
"type": "null"
},
"crawler": {
"type": "string"
},
"remoteStacktrace": {
"type": "null"
}
},
"required": [
"crawler",
"name",
"remoteStacktrace"
],
"title": "Message"
}
}
}
JS