API

API documentation

API Authorization

Add GET parameter apiKey to all your GET Requests:
– GET parameter: apiKey
– GET parameter value: your api key here…

How to make requests

Make GET Requests to appropriate Endpoints with all required parameters.
Endpoint URL sample for DataSourcesList function:
https://data.mydataprovider.com/app/WebApi/DataSourcesList?apiKey=your_api_key_here

If parameter marked with * – it is required parameters.

API Modes & Mode parameter

There are 2 modes for API exist:
1. db mode: take data from our database.
2. realtime mode: take data from site in real-time.

Case 1. If “mode field is db”
Function returns data from our database (data updated each 24-40 h).
Results will be sent immediately.

Case 2. If “mode field is realtime” – it means that data we will taken from data-source (site) in real-time.
For process this request task will be created on the service & you (your code) have to wait till task will be finished.
Function returns task_id.

DataSourcesList

Description: returns list of data sources that are supported in our service.
Parameters: there is no any input Parameters here.

Sample DataSourcesList response

{
  "results": [
    {
      "id": "1",
      "name": "site1",
      "url": "http://www.site1.com"

    },
    {
      "id": "2",
      "name": "site2",
      "url": "http://www.site2.com"
    },
    {
      "id": "3",
      "name": "site3",
      "url": "http://www.site3.com"
    }
]
}

ProductsSearch

Description: returns list of products from site.
Parameters:
– *source_id (string)
– *keyword (string)
– category_id (string) actual if mode=realtime
– mode (string, variants: db or realtime)

Sample ProductsSearch response

{
  "results": [
    {
      "id": "123dsrf3",
      "name": "SweatyRocks Women's Letter Print Crop Tops Summer Short Sleeve T-Shirt",
      "url": "https://www.amazon.com/gp/product/B071SM3D4X/ref=s9_acsd_top_hd_bw_b4NjU_c_x_3_w?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=merchandised-search-3&pf_rd_r=G0P7AB94ZVWCMTSE2XDX&pf_rd_t=101&pf_rd_p=0db635f0-6408-5802-a973-e4a61084053b&pf_rd_i=1044544"

    },
    {
      "id": "dfsd34",
      "name": "Imily Bela Womens Short Sleeve Summer Tops Cross Scoop Neck T-Shirt Knot Front USA Tees",
      "url": "https://www.amazon.com/gp/product/B07D4HRXGV/ref=s9_acsd_simh_hd_bw_b4NjU_c_x_2_w?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=merchandised-search-3&pf_rd_r=HMANQP3HY5TKJ2M61FNS&pf_rd_t=101&pf_rd_p=0db635f0-6408-5802-a973-e4a61084053b&pf_rd_i=1044544"
    }
]
}

Categories

Description: returns list of categories from site.
Parameters:
– *source_id (string)
– subcategory_id (string, default: ” empty)
– with_subcategories (bool, default: false)
– mode (string, variants: db or realtime)

Sample Categories response

{
  "results": [
    {
      "id": "abcat0020001",
      "name": "Electronics",
      "url": "http://www.site.com/site/electronics/abcat0020001.c?id=abcat0020001"

    },
    {
      "id": "abcat0020002",
      "name": "Shoes",
      "url": "http://www.site.com/site/Shoes/abcat0020002.c?id=abcat0020002"
    },
    {
      "id": "abcat0100004",
      "parent_id": "abcat0020002",
      "name": "Man",
      "url": "http://www.site.com/site/man/abcat0020004.c?id=abcat0020004"
    }
]
}

ProductsList

Description: returns list of products from site for specific category.
Parameters:
– *category_id (string)
– mode (string, variants: db or realtime)

Sample ProductsList response

{
  "results": [
    {
      "id": "123dsrf3",
      "name": "SweatyRocks Women's Letter Print Crop Tops Summer Short Sleeve T-Shirt"
    },
    {
      "id": "dfsd34",
      "name": "Imily Bela Womens Short Sleeve Summer Tops Cross Scoop Neck T-Shirt Knot Front USA Tees"
    }
]
}

Product

Description: returns product content (title, description, images, features, options, dimensions etc).
Parameters:
– *product_id (string)
– mode (string, variants: db or realtime)

Sample Product response

{
          "sku": "9036592",
          "name": "Calvin Klein Sundance",
          "category_id": "1000",
          "quantity": 10,
          "description": "<div ><ul><div ><li>SKU: #<!-- -->9036592</li><li ><div>The Calvin Klein® Sundance sandals are the perfect beach wear!</div></li></div><li><div>Synthetic upper.</div></li><li><div>Easy slip-on silhouette. </div></li><li><div>Soft man-made lining.</div></li><li><div>Lightly padded footbed.</div></li><li><div>Textured synthetic outsole.</div></li><li><div>Imported. </div></li><li><div>Weight of footwear is based on a single item, not a pair.</div></li><li><div>Measurements: <ul>  <li> Weight: 4 oz</li>  </ul> </div></li></ul></div>",
          "short_description": "",
          "manufacturer": "Calvin Klein",
          "weight": "4 oz",
          "height": "0",
          "width": "0",
          "depth": "0",
          "meta_title": "Calvin Klein Sundance at 6pm",
          "meta_keywords": "Sundance, Calvin Klein, Shoes, Women, 6pm.com, 6pm",
          "meta_description": "Sundance by Calvin Klein at 6pm. Read Calvin Klein Sundance product reviews, or select the size, width, and color of your choice.",
          "product_url": "https://www.6pm.com/p/calvin-klein-sundance-pink-pink/product/9036592/color/18585",
          "reviews_number": "",
          "rating": "",
          "price": "9.99",
          "price_old": "0",
          "price_wholesale": "0",
          "images": [
            {
              "name": "71WmKL4jcOL.jpg",
              "url": "https://m.media-amazon.com/images/I/71WmKL4jcOL.jpg.jpg"
            },
            {
              "name": "61aA0Qc8oKL.jpg",
              "url": "https://m.media-amazon.com/images/I/61aA0Qc8oKL.jpg.jpg"
            },
            {
              "name": "71WqtHoXwhL.jpg",
              "url": "https://m.media-amazon.com/images/I/71WqtHoXwhL.jpg.jpg"
            },
            {
              "name": "71h5lDB4x0L.jpg",
              "url": "https://m.media-amazon.com/images/I/71h5lDB4x0L.jpg.jpg"
            },
            {
              "name": "71oRGiyD6lL.jpg",
              "url": "https://m.media-amazon.com/images/I/71oRGiyD6lL.jpg.jpg"
            },
            {
              "name": "61ZGP4VPboL.jpg",
              "url": "https://m.media-amazon.com/images/I/61ZGP4VPboL.jpg.jpg"
            },
            {
              "name": "61YgaT3teUL.jpg",
              "url": "https://m.media-amazon.com/images/I/61YgaT3teUL.jpg.jpg"
            },
            {
              "name": "61rDokeLtLL.jpg",
              "url": "https://m.media-amazon.com/images/I/61rDokeLtLL.jpg.jpg"
            }
          ],
          "combinations": [
            {
              "sku": "B078ZKRRGP",
              "quantity": 12,
              "price": "14.99",
              "price_old": "0",
              "features": [
                {
                  "name": "Color",
                  "value": "Black/White"
                },
                {
                  "name": "Size",
                  "value": "6"
                },
                {
                  "name": "Width",
                  "value": "M"
                }
              ],
              "images": [
                {
                  "name": "71WmKL4jcOL.jpg",
                  "url": "https://m.media-amazon.com/images/I/71WmKL4jcOL.jpg.jpg"
                },
                {
                  "name": "61aA0Qc8oKL.jpg",
                  "url": "https://m.media-amazon.com/images/I/61aA0Qc8oKL.jpg.jpg"
                },
                {
                  "name": "71WqtHoXwhL.jpg",
                  "url": "https://m.media-amazon.com/images/I/71WqtHoXwhL.jpg.jpg"
                },
                {
                  "name": "71h5lDB4x0L.jpg",
                  "url": "https://m.media-amazon.com/images/I/71h5lDB4x0L.jpg.jpg"
                },
                {
                  "name": "71oRGiyD6lL.jpg",
                  "url": "https://m.media-amazon.com/images/I/71oRGiyD6lL.jpg.jpg"
                },
                {
                  "name": "61ZGP4VPboL.jpg",
                  "url": "https://m.media-amazon.com/images/I/61ZGP4VPboL.jpg.jpg"
                },
                {
                  "name": "61YgaT3teUL.jpg",
                  "url": "https://m.media-amazon.com/images/I/61YgaT3teUL.jpg.jpg"
                },
                {
                  "name": "61rDokeLtLL.jpg",
                  "url": "https://m.media-amazon.com/images/I/61rDokeLtLL.jpg.jpg"
                }
              ]
            }


ProductInventory

Description: returns product price & quantity.
Parameters:
– *product_id (string)
– mode (string, variants: db or realtime)

Sample ProductInventory response

{
          "sku": "9036592",
          "name": "Calvin Klein Sundance",
          "category_id": "1000",
          "quantity": 10,
          "price": "9.99",
          "price_old": "0",
          "price_wholesale": "0",
          "combinations": [
            {
              "sku": "B078ZKRRGP",
              "quantity": 12,
              "price": "14.99",
              "price_old": "0",
              "features": [
                {
                  "name": "Color",
                  "value": "Black/White"
                },
                {
                  "name": "Size",
                  "value": "6"
                },
                {
                  "name": "Width",
                  "value": "M"
                }
              ],
              
            }
		]
}

TaskStatus

Description: returns task status (actual if you use mode=realtime)
Parameters:
– *task_id (string)

Sample TaskStatus response

{
  "status": "running",
  "results": null
}

or

{
  "status": "finished",
  "results": -- see here sample output for related function --
}

Raw web scraping API documentation

Types of exports

We support 4 types of files (export formats) that can be downloaded via API:
csv
excel
json
xml
You can define preferred for you format in export settings for your campaign.

How to access extracted data via API

Create account at WebScraper Runner, firstly.

Step 1. Go to Profile & Turn on PRO mode

Step 2. Go to Campaigns settings.

Step 3. Go to API Edit of selected Campaign

Step 4. Generate API API Key & URL

you will get Secret URL for accessing data from your Campaign like:


https://data.mydataprovider.com/app/WebApi/CampaignDataLatestGet/1-201806261129-YNFRJIDJDJBDVPWAUFNPWWA23978SWWYKFHAQDJVUNVMXVGXWTTEYYNFRBTHSYY


Where


app/WebApi/CampaignDataLatestGet – is endpoint


1-201806261129-YNFRJIDJDJBD… – is your API key

PHP,Python,Ruby,C# API clients

Find below client samples for for CampaignDataLatestGet endpoint
Find here API clients samples:
C#
PHP
Python
Ruby

How to define categories or products urls for extraction?

By default all scrapers are designed to extract data from category or products URLs
but if it is required by business needs Support Team could implement the other type of @input data like list or SKU/keywords for extraction etc.

For using this function you have to:
1. know API key for your account / campaign from previous section.
2. make HTTP post request to https://data.mydataprovider.com/app/WebApi/CampaignTaskCreate endpoint with the next fields
– categories
– api_key

“categories” field is a special formatted text field that defines URLs for extraction, categories hierarchy etc.
Read more about “Catalog” (internal field name grab_catalog) here :
Categories – Runner page

Find here JS sample how to run this request:

$.ajax({
    type: 'POST',
    url: serviceUrl + 'https://data.mydataprovider.com/app/WebApi/CampaignTaskCreate',
    data: { catalog: catalog, secret_key:secret_key },
    dataType: 'json', 
    cache: false,
    success: AjaxSucceeded,
    error: AjaxFailed
});

Task status checking

There are 6 statuses each task registered in could have:

public enum TaskStatus
    {
        Prepare = -1, // for internal use
        Created = 0,
        Finished = 1,
        StartedToProcessed = 2,
        FinishedWithError = 3, // process.ExitCode aka %ERRORLEVEL% != 0
        Cancelled = 4, // 
    }

For using this function you have to:
1. know secret key for your account / campaign from previous section 1.
2. make HTTP post request to https://data.mydataprovider.com/app/WebApi/CampaignTaskStatus endpoint with the next fields
– task_id (get in output from CampaignTaskCreate endpoint)
– secret_key

Callback on export task finished event

Sometimes it is necessary to support scenarios when our service has to send call to client’s server with notification that task finished.
If it is actual -> contact support team and the next POST request will be sent to your server endpoint:

$.ajax({
    type: 'POST',
    url: serviceUrl + 'https://your_domain.com/your_endpoint_url',
    data: { task_id: task_id, task_status:task_status },
    dataType: 'json', 
    cache: false,
    success: AjaxSucceeded,
    error: AjaxFailed
});
Contact us for details


Related articles