API
Warning! We just inform you that our API will be implemented for each client individually. You have to understand that under each API call, Web Scraping action will be started.
Files API documentation
This section contains information how to download files with data generated by web scrapers.
Remind you that you always can download these files from our web-service manually.
Types of exports
We support 4 types of files (export formats) that can be downloaded via API:
– csv
– excel
– json
– xml
You can define preferred for you format in export settings for your campaign.
How to access extracted data via API
Open your web scraper, firstly.
you will get Secret URL for accessing data from your Campaign like:
https://data.mydataprovider.com/app/WebApi/CampaignDataLatestGet/1-201806261129-YNFRJIDJDJBDVPWAUFNPWWA23978SWWYKFHAQDJVUNVMXVGXWTTEYYNFRBTHSYY
Where
app/WebApi/CampaignDataLatestGet – is endpoint
1-201806261129-YNFRJIDJDJBD… – is your API key
PHP,Python,Ruby,C# API clients
Find below client samples for for CampaignDataLatestGet endpoint
Find here API clients samples:
C#
PHP
Python
Ruby
How to define categories or products urls for extraction?
By default all scrapers are designed to extract data from category or products URLs
but if it is required by business needs Support Team could implement the other type of @input data like list or SKU/keywords for extraction etc.
For using this function you have to:
1. know API key for your account / campaign from previous section.
2. make HTTP post request to https://data.mydataprovider.com/app/WebApi/CampaignTaskCreate endpoint with the next fields
– categories
– api_key
“categories” field is a special formatted text field that defines URLs for extraction, categories hierarchy etc.
Sample :
#Women’s Clothing & Accessories[path]https://www.aliexpress.com
##Dresses[path]https://www.aliexpress.com/category/200001648/blouses-shirts.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=40
##Blouses & Shirts[path]https://www.aliexpress.com/category/200001648/blouses-shirts.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=40
##Hoodies & Sweatshirts[path]https://www.aliexpress.com/category/100003141/hoodies-sweatshirts.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=10
##Sweaters[path]https://www.aliexpress.com/category/200000783/sweaters.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=30
##Jackets & Coats[path]https://www.aliexpress.com/category/200000775/jackets-coats.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=30
##Accessories[path]https://www.aliexpress.com/category/200000724/accessories.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=20
##Bottoms[path]https://www.aliexpress.com/category/200118010/bottoms.html?minPrice=&maxPrice=&isBigSale=n&isFreeShip=n&isNew=n&isFavorite=y&shipFromCountry=&shipCompanies=&SearchText=&CatId=200215305&SortType=total_tranpro_desc&needQuery=y&PAGINATION_MAX_PAGE=20
Find here JS sample how to run this request:
$.ajax({
type: 'POST',
url: serviceUrl + 'https://data.mydataprovider.com/app/WebApi/CampaignTaskCreate',
data: { catalog: catalog, secret_key:secret_key },
dataType: 'json',
cache: false,
success: AjaxSucceeded,
error: AjaxFailed
});
Task status checking
There are 6 statuses each task registered in could have:
public enum TaskStatus
{
Prepare = -1, // for internal use
Created = 0,
Finished = 1,
StartedToProcessed = 2,
FinishedWithError = 3, // process.ExitCode aka %ERRORLEVEL% != 0
Cancelled = 4, //
}
For using this function you have to:
1. know secret key for your account / campaign from previous section 1.
2. make HTTP post request to https://data.mydataprovider.com/app/WebApi/CampaignTaskStatus endpoint with the next fields
– task_id (get in output from CampaignTaskCreate endpoint)
– secret_key
Callback on export task finished event
Sometimes it is necessary to support scenarios when our service has to send call to client’s server with notification that task finished.
If it is actual -> contact support team and the next POST request will be sent to your server endpoint:
$.ajax({
type: 'POST',
url: serviceUrl + 'https://your_domain.com/your_endpoint_url',
data: { task_id: task_id, task_status:task_status },
dataType: 'json',
cache: false,
success: AjaxSucceeded,
error: AjaxFailed
});
Products API documentation
API Authorization
Add GET parameter apiKey to all your GET Requests:
– GET parameter: apiKey
– GET parameter value: your api key here…
How to make requests
Make GET Requests to appropriate Endpoints with all required parameters.
Endpoint URL sample for DataSourcesList function:
https://data.mydataprovider.com/app/WebApi/DataSourcesList?apiKey=your_api_key_here
If parameter marked with * – it is required parameters.
API Modes & Mode parameter
There are 2 modes for API exist:
1. db mode: take data from our database.
2. realtime mode: take data from site in real-time.
Case 1. If “mode field is db”
Function returns data from our database (data updated each 24-40 h).
Results will be sent immediately.
Case 2. If “mode field is realtime” – it means that data we will taken from data-source (site) in real-time.
For process this request task will be created on the service & you (your code) have to wait till task will be finished.
Function returns task_id.
DataSourcesList
Description: returns list of data sources that are supported in our service.
Parameters: there is no any input Parameters here.
{
"results": [
{
"id": "1",
"name": "site1",
"url": "http://www.site1.com"
},
{
"id": "2",
"name": "site2",
"url": "http://www.site2.com"
},
{
"id": "3",
"name": "site3",
"url": "http://www.site3.com"
}
]
}
ProductsSearch
Description: returns list of products from site.
Parameters:
– *source_id (string)
– *keyword (string)
– category_id (string) actual if mode=realtime
– mode (string, variants: db or realtime)
{
"results": [
{
"id": "123dsrf3",
"name": "SweatyRocks Women's Letter Print Crop Tops Summer Short Sleeve T-Shirt",
"url": "https://www.amazon.com/gp/product/B071SM3D4X/ref=s9_acsd_top_hd_bw_b4NjU_c_x_3_w?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=merchandised-search-3&pf_rd_r=G0P7AB94ZVWCMTSE2XDX&pf_rd_t=101&pf_rd_p=0db635f0-6408-5802-a973-e4a61084053b&pf_rd_i=1044544"
},
{
"id": "dfsd34",
"name": "Imily Bela Womens Short Sleeve Summer Tops Cross Scoop Neck T-Shirt Knot Front USA Tees",
"url": "https://www.amazon.com/gp/product/B07D4HRXGV/ref=s9_acsd_simh_hd_bw_b4NjU_c_x_2_w?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=merchandised-search-3&pf_rd_r=HMANQP3HY5TKJ2M61FNS&pf_rd_t=101&pf_rd_p=0db635f0-6408-5802-a973-e4a61084053b&pf_rd_i=1044544"
}
]
}
Categories
Description: returns list of categories from site.
Parameters:
– *source_id (string)
– subcategory_id (string, default: ” empty)
– with_subcategories (bool, default: false)
– mode (string, variants: db or realtime)
{
"results": [
{
"id": "abcat0020001",
"name": "Electronics",
"url": "http://www.site.com/site/electronics/abcat0020001.c?id=abcat0020001"
},
{
"id": "abcat0020002",
"name": "Shoes",
"url": "http://www.site.com/site/Shoes/abcat0020002.c?id=abcat0020002"
},
{
"id": "abcat0100004",
"parent_id": "abcat0020002",
"name": "Man",
"url": "http://www.site.com/site/man/abcat0020004.c?id=abcat0020004"
}
]
}
ProductsList
Description: returns list of products from site for specific category.
Parameters:
– *category_id (string)
– mode (string, variants: db or realtime)
{
"results": [
{
"id": "123dsrf3",
"name": "SweatyRocks Women's Letter Print Crop Tops Summer Short Sleeve T-Shirt"
},
{
"id": "dfsd34",
"name": "Imily Bela Womens Short Sleeve Summer Tops Cross Scoop Neck T-Shirt Knot Front USA Tees"
}
]
}
Product
Description: returns product content (title, description, images, features, options, dimensions etc).
Parameters:
– *product_id (string)
– mode (string, variants: db or realtime)
{
"sku": "9036592",
"name": "Calvin Klein Sundance",
"category_id": "1000",
"quantity": 10,
"description": "<div ><ul><div ><li>SKU: #<!-- -->9036592</li><li ><div>The Calvin Klein® Sundance sandals are the perfect beach wear!</div></li></div><li><div>Synthetic upper.</div></li><li><div>Easy slip-on silhouette. </div></li><li><div>Soft man-made lining.</div></li><li><div>Lightly padded footbed.</div></li><li><div>Textured synthetic outsole.</div></li><li><div>Imported. </div></li><li><div>Weight of footwear is based on a single item, not a pair.</div></li><li><div>Measurements: <ul> <li> Weight: 4 oz</li> </ul> </div></li></ul></div>",
"short_description": "",
"manufacturer": "Calvin Klein",
"weight": "4 oz",
"height": "0",
"width": "0",
"depth": "0",
"meta_title": "Calvin Klein Sundance at 6pm",
"meta_keywords": "Sundance, Calvin Klein, Shoes, Women, 6pm.com, 6pm",
"meta_description": "Sundance by Calvin Klein at 6pm. Read Calvin Klein Sundance product reviews, or select the size, width, and color of your choice.",
"product_url": "https://www.6pm.com/p/calvin-klein-sundance-pink-pink/product/9036592/color/18585",
"reviews_number": "",
"rating": "",
"price": "9.99",
"price_old": "0",
"price_wholesale": "0",
"images": [
{
"name": "71WmKL4jcOL.jpg",
"url": "https://m.media-amazon.com/images/I/71WmKL4jcOL.jpg.jpg"
},
{
"name": "61aA0Qc8oKL.jpg",
"url": "https://m.media-amazon.com/images/I/61aA0Qc8oKL.jpg.jpg"
},
{
"name": "71WqtHoXwhL.jpg",
"url": "https://m.media-amazon.com/images/I/71WqtHoXwhL.jpg.jpg"
},
{
"name": "71h5lDB4x0L.jpg",
"url": "https://m.media-amazon.com/images/I/71h5lDB4x0L.jpg.jpg"
},
{
"name": "71oRGiyD6lL.jpg",
"url": "https://m.media-amazon.com/images/I/71oRGiyD6lL.jpg.jpg"
},
{
"name": "61ZGP4VPboL.jpg",
"url": "https://m.media-amazon.com/images/I/61ZGP4VPboL.jpg.jpg"
},
{
"name": "61YgaT3teUL.jpg",
"url": "https://m.media-amazon.com/images/I/61YgaT3teUL.jpg.jpg"
},
{
"name": "61rDokeLtLL.jpg",
"url": "https://m.media-amazon.com/images/I/61rDokeLtLL.jpg.jpg"
}
],
"combinations": [
{
"sku": "B078ZKRRGP",
"quantity": 12,
"price": "14.99",
"price_old": "0",
"features": [
{
"name": "Color",
"value": "Black/White"
},
{
"name": "Size",
"value": "6"
},
{
"name": "Width",
"value": "M"
}
],
"images": [
{
"name": "71WmKL4jcOL.jpg",
"url": "https://m.media-amazon.com/images/I/71WmKL4jcOL.jpg.jpg"
},
{
"name": "61aA0Qc8oKL.jpg",
"url": "https://m.media-amazon.com/images/I/61aA0Qc8oKL.jpg.jpg"
},
{
"name": "71WqtHoXwhL.jpg",
"url": "https://m.media-amazon.com/images/I/71WqtHoXwhL.jpg.jpg"
},
{
"name": "71h5lDB4x0L.jpg",
"url": "https://m.media-amazon.com/images/I/71h5lDB4x0L.jpg.jpg"
},
{
"name": "71oRGiyD6lL.jpg",
"url": "https://m.media-amazon.com/images/I/71oRGiyD6lL.jpg.jpg"
},
{
"name": "61ZGP4VPboL.jpg",
"url": "https://m.media-amazon.com/images/I/61ZGP4VPboL.jpg.jpg"
},
{
"name": "61YgaT3teUL.jpg",
"url": "https://m.media-amazon.com/images/I/61YgaT3teUL.jpg.jpg"
},
{
"name": "61rDokeLtLL.jpg",
"url": "https://m.media-amazon.com/images/I/61rDokeLtLL.jpg.jpg"
}
]
}
ProductInventory
Description: returns product price & quantity.
Parameters:
– *product_id (string)
– mode (string, variants: db or realtime)
{
"sku": "9036592",
"name": "Calvin Klein Sundance",
"category_id": "1000",
"quantity": 10,
"price": "9.99",
"price_old": "0",
"price_wholesale": "0",
"combinations": [
{
"sku": "B078ZKRRGP",
"quantity": 12,
"price": "14.99",
"price_old": "0",
"features": [
{
"name": "Color",
"value": "Black/White"
},
{
"name": "Size",
"value": "6"
},
{
"name": "Width",
"value": "M"
}
],
}
]
}
TaskStatus
Description: returns task status (actual if you use mode=realtime)
Parameters:
– *task_id (string)
{
"status": "running",
"results": null
}
or
{
"status": "finished",
"results": -- see here sample output for related function --
}