API

Warning! We just inform you that our API will be implemented for each client individually. You have to understand that under each API call, Web Scraping action will be started.


Files API documentation

This section contains information how to download files with data generated by web scrapers.
Remind you that you always can download these files from our web-service manually.

Types of exports

We support 4 types of files (export formats) that can be downloaded via API:
csv
excel
json
xml
You can define preferred for you format in export settings for your campaign.

How to access extracted data via API

Open your web scraper, firstly.

Step 1. Go to Profile & Turn on PRO mode

Step 2. Go to Campaigns settings.

Step 3. Go to API Edit of selected Campaign

Step 4. Generate API API Key & URL

you will get Secret URL for accessing data from your Campaign like:


https://data.mydataprovider.com/app/WebApi/CampaignDataLatestGet/1-201806261129-YNFRJIDJDJBDVPWAUFNPWWA23978SWWYKFHAQDJVUNVMXVGXWTTEYYNFRBTHSYY


Where


app/WebApi/CampaignDataLatestGet – is endpoint


1-201806261129-YNFRJIDJDJBD… – is your API key

PHP,Python,Ruby,C# API clients

Find below client samples for for CampaignDataLatestGet endpoint
Find here API clients samples:
C#
PHP
Python
Ruby

How to define categories or products urls for extraction?

By default all scrapers are designed to extract data from category or products URLs
but if it is required by business needs Support Team could implement the other type of @input data like list or SKU/keywords for extraction etc.

For using this function you have to:
1. know API key for your account / campaign from previous section.
2. make HTTP post request to https://data.mydataprovider.com/app/WebApi/CampaignTaskCreate endpoint with the next fields
– categories
– api_key

“categories” field is a special formatted text field that defines URLs for extraction, categories hierarchy etc.

Sample :
#Women’s Clothing & Accessories[path]https://www.aliexpress.com
##Dresses[path]https://www.aliexpress.com/category/200001648/blouses-shirts.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=40
##Blouses & Shirts[path]https://www.aliexpress.com/category/200001648/blouses-shirts.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=40
##Hoodies & Sweatshirts[path]https://www.aliexpress.com/category/100003141/hoodies-sweatshirts.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=10
##Sweaters[path]https://www.aliexpress.com/category/200000783/sweaters.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=30
##Jackets & Coats[path]https://www.aliexpress.com/category/200000775/jackets-coats.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=30
##Accessories[path]https://www.aliexpress.com/category/200000724/accessories.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=20
##Bottoms[path]https://www.aliexpress.com/category/200118010/bottoms.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=20

Find here JS sample how to run this request:

$.ajax({
    type: 'POST',
    url: serviceUrl + 'https://data.mydataprovider.com/app/WebApi/CampaignTaskCreate',
    data: { catalog: catalog, secret_key:secret_key },
    dataType: 'json', 
    cache: false,
    success: AjaxSucceeded,
    error: AjaxFailed
});

Task status checking

There are 6 statuses each task registered in could have:

public enum TaskStatus
    {
        Prepare = -1, // for internal use
        Created = 0,
        Finished = 1,
        StartedToProcessed = 2,
        FinishedWithError = 3, // process.ExitCode aka %ERRORLEVEL% != 0
        Cancelled = 4, // 
    }

For using this function you have to:
1. know secret key for your account / campaign from previous section 1.
2. make HTTP post request to https://data.mydataprovider.com/app/WebApi/CampaignTaskStatus endpoint with the next fields
– task_id (get in output from CampaignTaskCreate endpoint)
– secret_key

Callback on export task finished event

Sometimes it is necessary to support scenarios when our service has to send call to client’s server with notification that task finished.
If it is actual -> contact support team and the next POST request will be sent to your server endpoint:

$.ajax({
    type: 'POST',
    url: serviceUrl + 'https://your_domain.com/your_endpoint_url',
    data: { task_id: task_id, task_status:task_status },
    dataType: 'json', 
    cache: false,
    success: AjaxSucceeded,
    error: AjaxFailed
});

Products API documentation

API Authorization

Add GET parameter apiKey to all your GET Requests:
– GET parameter: apiKey
– GET parameter value: your api key here…

How to make requests

Make GET Requests to appropriate Endpoints with all required parameters.
Endpoint URL sample for DataSourcesList function:
https://data.mydataprovider.com/app/WebApi/DataSourcesList?apiKey=your_api_key_here

If parameter marked with * – it is required parameters.

API Modes & Mode parameter

There are 2 modes for API exist:
1. db mode: take data from our database.
2. realtime mode: take data from site in real-time.

Case 1. If “mode field is db”
Function returns data from our database (data updated each 24-40 h).
Results will be sent immediately.

Case 2. If “mode field is realtime” – it means that data we will taken from data-source (site) in real-time.
For process this request task will be created on the service & you (your code) have to wait till task will be finished.
Function returns task_id.

DataSourcesList

Description: returns list of data sources that are supported in our service.
Parameters: there is no any input Parameters here.

Sample DataSourcesList response

{
  "results": [
    {
      "id": "1",
      "name": "site1",
      "url": "http://www.site1.com"

    },
    {
      "id": "2",
      "name": "site2",
      "url": "http://www.site2.com"
    },
    {
      "id": "3",
      "name": "site3",
      "url": "http://www.site3.com"
    }
]
}

ProductsSearch

Description: returns list of products from site.
Parameters:
– *source_id (string)
– *keyword (string)
– category_id (string) actual if mode=realtime
– mode (string, variants: db or realtime)

Sample ProductsSearch response

{
  "results": [
    {
      "id": "123dsrf3",
      "name": "SweatyRocks Women's Letter Print Crop Tops Summer Short Sleeve T-Shirt",
      "url": "https://www.amazon.com/gp/product/B071SM3D4X/ref=s9_acsd_top_hd_bw_b4NjU_c_x_3_w?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=merchandised-search-3&pf_rd_r=G0P7AB94ZVWCMTSE2XDX&pf_rd_t=101&pf_rd_p=0db635f0-6408-5802-a973-e4a61084053b&pf_rd_i=1044544"

    },
    {
      "id": "dfsd34",
      "name": "Imily Bela Womens Short Sleeve Summer Tops Cross Scoop Neck T-Shirt Knot Front USA Tees",
      "url": "https://www.amazon.com/gp/product/B07D4HRXGV/ref=s9_acsd_simh_hd_bw_b4NjU_c_x_2_w?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=merchandised-search-3&pf_rd_r=HMANQP3HY5TKJ2M61FNS&pf_rd_t=101&pf_rd_p=0db635f0-6408-5802-a973-e4a61084053b&pf_rd_i=1044544"
    }
]
}

Categories

Description: returns list of categories from site.
Parameters:
– *source_id (string)
– subcategory_id (string, default: ” empty)
– with_subcategories (bool, default: false)
– mode (string, variants: db or realtime)

Sample Categories response

{
  "results": [
    {
      "id": "abcat0020001",
      "name": "Electronics",
      "url": "http://www.site.com/site/electronics/abcat0020001.c?id=abcat0020001"

    },
    {
      "id": "abcat0020002",
      "name": "Shoes",
      "url": "http://www.site.com/site/Shoes/abcat0020002.c?id=abcat0020002"
    },
    {
      "id": "abcat0100004",
      "parent_id": "abcat0020002",
      "name": "Man",
      "url": "http://www.site.com/site/man/abcat0020004.c?id=abcat0020004"
    }
]
}

ProductsList

Description: returns list of products from site for specific category.
Parameters:
– *category_id (string)
– mode (string, variants: db or realtime)

Sample ProductsList response

{
  "results": [
    {
      "id": "123dsrf3",
      "name": "SweatyRocks Women's Letter Print Crop Tops Summer Short Sleeve T-Shirt"
    },
    {
      "id": "dfsd34",
      "name": "Imily Bela Womens Short Sleeve Summer Tops Cross Scoop Neck T-Shirt Knot Front USA Tees"
    }
]
}

Product

Description: returns product content (title, description, images, features, options, dimensions etc).
Parameters:
– *product_id (string)
– mode (string, variants: db or realtime)

Sample Product response

{
          "sku": "9036592",
          "name": "Calvin Klein Sundance",
          "category_id": "1000",
          "quantity": 10,
          "description": "<div ><ul><div ><li>SKU: #<!-- -->9036592</li><li ><div>The Calvin Klein® Sundance sandals are the perfect beach wear!</div></li></div><li><div>Synthetic upper.</div></li><li><div>Easy slip-on silhouette. </div></li><li><div>Soft man-made lining.</div></li><li><div>Lightly padded footbed.</div></li><li><div>Textured synthetic outsole.</div></li><li><div>Imported. </div></li><li><div>Weight of footwear is based on a single item, not a pair.</div></li><li><div>Measurements: <ul>  <li> Weight: 4 oz</li>  </ul> </div></li></ul></div>",
          "short_description": "",
          "manufacturer": "Calvin Klein",
          "weight": "4 oz",
          "height": "0",
          "width": "0",
          "depth": "0",
          "meta_title": "Calvin Klein Sundance at 6pm",
          "meta_keywords": "Sundance, Calvin Klein, Shoes, Women, 6pm.com, 6pm",
          "meta_description": "Sundance by Calvin Klein at 6pm. Read Calvin Klein Sundance product reviews, or select the size, width, and color of your choice.",
          "product_url": "https://www.6pm.com/p/calvin-klein-sundance-pink-pink/product/9036592/color/18585",
          "reviews_number": "",
          "rating": "",
          "price": "9.99",
          "price_old": "0",
          "price_wholesale": "0",
          "images": [
            {
              "name": "71WmKL4jcOL.jpg",
              "url": "https://m.media-amazon.com/images/I/71WmKL4jcOL.jpg.jpg"
            },
            {
              "name": "61aA0Qc8oKL.jpg",
              "url": "https://m.media-amazon.com/images/I/61aA0Qc8oKL.jpg.jpg"
            },
            {
              "name": "71WqtHoXwhL.jpg",
              "url": "https://m.media-amazon.com/images/I/71WqtHoXwhL.jpg.jpg"
            },
            {
              "name": "71h5lDB4x0L.jpg",
              "url": "https://m.media-amazon.com/images/I/71h5lDB4x0L.jpg.jpg"
            },
            {
              "name": "71oRGiyD6lL.jpg",
              "url": "https://m.media-amazon.com/images/I/71oRGiyD6lL.jpg.jpg"
            },
            {
              "name": "61ZGP4VPboL.jpg",
              "url": "https://m.media-amazon.com/images/I/61ZGP4VPboL.jpg.jpg"
            },
            {
              "name": "61YgaT3teUL.jpg",
              "url": "https://m.media-amazon.com/images/I/61YgaT3teUL.jpg.jpg"
            },
            {
              "name": "61rDokeLtLL.jpg",
              "url": "https://m.media-amazon.com/images/I/61rDokeLtLL.jpg.jpg"
            }
          ],
          "combinations": [
            {
              "sku": "B078ZKRRGP",
              "quantity": 12,
              "price": "14.99",
              "price_old": "0",
              "features": [
                {
                  "name": "Color",
                  "value": "Black/White"
                },
                {
                  "name": "Size",
                  "value": "6"
                },
                {
                  "name": "Width",
                  "value": "M"
                }
              ],
              "images": [
                {
                  "name": "71WmKL4jcOL.jpg",
                  "url": "https://m.media-amazon.com/images/I/71WmKL4jcOL.jpg.jpg"
                },
                {
                  "name": "61aA0Qc8oKL.jpg",
                  "url": "https://m.media-amazon.com/images/I/61aA0Qc8oKL.jpg.jpg"
                },
                {
                  "name": "71WqtHoXwhL.jpg",
                  "url": "https://m.media-amazon.com/images/I/71WqtHoXwhL.jpg.jpg"
                },
                {
                  "name": "71h5lDB4x0L.jpg",
                  "url": "https://m.media-amazon.com/images/I/71h5lDB4x0L.jpg.jpg"
                },
                {
                  "name": "71oRGiyD6lL.jpg",
                  "url": "https://m.media-amazon.com/images/I/71oRGiyD6lL.jpg.jpg"
                },
                {
                  "name": "61ZGP4VPboL.jpg",
                  "url": "https://m.media-amazon.com/images/I/61ZGP4VPboL.jpg.jpg"
                },
                {
                  "name": "61YgaT3teUL.jpg",
                  "url": "https://m.media-amazon.com/images/I/61YgaT3teUL.jpg.jpg"
                },
                {
                  "name": "61rDokeLtLL.jpg",
                  "url": "https://m.media-amazon.com/images/I/61rDokeLtLL.jpg.jpg"
                }
              ]
            }


ProductInventory

Description: returns product price & quantity.
Parameters:
– *product_id (string)
– mode (string, variants: db or realtime)

Sample ProductInventory response

{
          "sku": "9036592",
          "name": "Calvin Klein Sundance",
          "category_id": "1000",
          "quantity": 10,
          "price": "9.99",
          "price_old": "0",
          "price_wholesale": "0",
          "combinations": [
            {
              "sku": "B078ZKRRGP",
              "quantity": 12,
              "price": "14.99",
              "price_old": "0",
              "features": [
                {
                  "name": "Color",
                  "value": "Black/White"
                },
                {
                  "name": "Size",
                  "value": "6"
                },
                {
                  "name": "Width",
                  "value": "M"
                }
              ],
              
            }
		]
}

TaskStatus

Description: returns task status (actual if you use mode=realtime)
Parameters:
– *task_id (string)

Sample TaskStatus response

{
  "status": "running",
  "results": null
}

or

{
  "status": "finished",
  "results": -- see here sample output for related function --
}

Orders API Documentation

Workflow
CS – it is your online store or ERP.
AOP – automatic order processor.
1) CS registers new orders at AOP via HTTP request with JSON data inside.
2) AOP Service processes all new requests from Core System via queue (1 by 1) and saves results at database.
3) CS checks status of requests (orders) periodically by OrderID.

AOP Interfaces (API functions)

OrderRegistration

Parameters (JSON in POST)
– Order ID
– User 1st name
– User 2nd name
– Delivery Address
– Product Url

Results ( JSON output)
– Status: OK (http code 200 – registered with success
– Status: NOT (http code 500 – failed)

OrderStatusGet (JSON in POST)

Parameters
– Order ID
Results ( JSON output)
– Status: OK (http code 200 – processed with success
– Status: NOTFOUND (http code 200 – did not find order)
– Status: PROCESSING (http code 200 – at processing (in a queue)
– TrackingID : #trackId from mediamarkt.pl

Explore our API access

API access


Explore our file export formats

CSV sample files
Excel sample files
Xml sample files
Json sample files

Contact us for details