Webhook Notifications

How It Works

When you create a job with a webhookUrl, Scrapernode stores the URL alongside the job. Once the job finishes (successfully or with an error), Scrapernode sends a POST request to your URL with a JSON payload describing the outcome.

1. 1

   Create a job with webhookUrl

   Include a webhookUrl field in your POST /api/jobs/create request body.

2. 2

   Scrapernode processes the job

   The job runs asynchronously. You don't need to poll for status.

3. 3

   Receive the webhook

   When the job completes or fails, Scrapernode POSTs a JSON payload to your URL with the job outcome.

Quickstart

Add webhookUrl to any job creation request. That's it.

curl -X POST https://actions.scrapernode.com/api/jobs/create \
  -H "Authorization: Bearer sn_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "scraperId": "linkedin-profiles",
    "inputs": [
      { "url": "https://linkedin.com/in/satyanadella" }
    ],
    "webhookUrl": "https://your-server.com/webhooks/scrapernode"
  }'

When the job finishes, your endpoint receives:

POST /webhooks/scrapernode HTTP/1.1
Content-Type: application/json

{
  "event": "job.completed",
  "jobId": "k57a8b3c9d0e1f2g3h4",
  "scraperId": "linkedin-profiles",
  "resultCount": 1,
  "creditsUsed": 5,
  "completedAt": 1709568045000
}

Webhook Payload

Every webhook is a POST request with Content-Type: application/json. Your endpoint should return a 2xx status to acknowledge receipt.

Event Types

job.completed

Sent when the job finishes successfully. Use the jobId to fetch results via GET /api/jobs/results?jobId=...

{
  "event": "job.completed",
  "jobId": "k57a8b3c9d0e1f2g3h4",
  "scraperId": "linkedin-profiles",
  "resultCount": 2,
  "creditsUsed": 10,
  "completedAt": 1709568045000
}

job.failed

Sent when the job fails. The error field contains the failure reason. Reserved credits are refunded automatically.

{
  "event": "job.failed",
  "jobId": "k57a8b3c9d0e1f2g3h4",
  "scraperId": "linkedin-profiles",
  "resultCount": 0,
  "creditsUsed": 0,
  "error": "Data source temporarily unavailable",
  "completedAt": 1709568045000
}

Retry Behavior

Webhook delivery is best-effort with automatic retries for server errors.

Security

Webhook URLs must use https:// or http:// protocol. Other protocols are rejected at job creation time.

Verifying Webhook Origin

To confirm a webhook came from Scrapernode, use the jobId from the payload to call GET /api/jobs/get?jobId=... with your API key. If the job exists and belongs to you, the webhook is authentic.

Using with n8n

Webhooks pair perfectly with n8n's Webhook Trigger node, giving you a push-based alternative to polling.

1. 1

   Add a Webhook node in n8n

   Create a new workflow and add a "Webhook" trigger node. Set the HTTP method to POST. Copy the webhook URL (production or test).

2. 2

   Create a job with the webhook URL

   Use the Scrapernode API or a Scrapernode scraper node to create a job. Paste the n8n webhook URL as the webhookUrl parameter.

3. 3

   Process the results

   When the job completes, n8n receives the payload. Use an IF node to check the event type, then use a Scrapernode Jobs → Get Results node (or HTTP Request to GET /api/jobs/results) to fetch the scraped data.

// n8n Webhook Trigger receives this payload:
{
  "event": "job.completed",
  "jobId": "k57a8b3c9d0e1f2g3h4",
  "scraperId": "linkedin-profiles",
  "resultCount": 2,
  "creditsUsed": 10,
  "completedAt": 1709568045000
}

// Then use the jobId to fetch results:
// GET /api/jobs/results?jobId=k57a8b3c9d0e1f2g3h4

Testing Webhooks

Use a request inspection tool to test webhooks during development.

1. 1

   Get a test URL

   Use a service like webhook.site, RequestBin, or ngrok to get a public URL that logs incoming requests.

2. 2

   Create a job with the test URL

   Pass your test URL as the webhookUrl when creating a job via the API.

3. 3

   Inspect the payload

   Once the job completes, check your inspection tool to see the exact JSON payload Scrapernode sent.

curl -X POST https://actions.scrapernode.com/api/jobs/create \
  -H "Authorization: Bearer sn_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "scraperId": "linkedin-profiles",
    "inputs": [{ "url": "https://linkedin.com/in/satyanadella" }],
    "webhookUrl": "https://webhook.site/your-unique-id"
  }'

FAQ