Web Scraping Service

How to get access

There are 2 versions of Web Scraping Service:
1) General Version – visit http://data.mydataprovider.com
2) Shopify Version – visit https://apps.shopify.com/webscraping

How to add Source (for Web Scraping)

Go to @Sources section and press @Add Button – from the list add necessary for you source.
How to add Source for web scraping service

How to select Categories for Source

Go to @Sources section and press on @Edit Button for targeted Source.
How to select Categories for Source for web scraping service
After that click on @”Categories Filter” and check necessary for you categories. Press @Save button.
How to check Categories for Source of web scraping service

How to export

Go to @Export section and click at “config Export” link.

Export to File

Go to “Export To File” section and define preferred export variant. If you want to import csv file directly to file format, for example use “CSV Shopify” or XML or JSON or Excel or CSV.

Direct export to your Store

“Export to Shop” turned off by default. If you want to Turn ON direct export to your store – check Export to Shop & Save “Export to your CMS database” section.
How to export to file or to online store from web scraping service

How to run export

Go to @Export section and click @Create button and on the next screen click @Confirm button.
How to run export from web scraping service
how to start web scraping service

How to pay

Go to @Invoices section and press on @”Get Invoice” button.
How to pay for web scraping service

Margin Setup

Go to @Sources-> @”Your source – Edit”->@”Categories change +margin” and Press @Donwload template. Template for Settings Margins for selected categories will be downloaded.
How to download template file for custom margin at web scraping service
It is Excel file – open it in Ms Excel or OpenOffice or LibreOffice editor.Sample file
How to setup custom margin for prices from web scraping service
Now you have to fill this file and upload to @“Upload new Settings” section.

Find below sample of filled file:
sample of web scraping service configuration file for margin setup

How does Margin work
Active formula for NEW Prices is the next:
“New Price” = (“Wholesale Price” * (1 + Margin %) ) + (Margin DELTA)

“Source category” column (A) – do not change it!
“Destination category” column (B) – allows you to change original category path.
“Margin % percent” column (C) – allows you to set your margin for exported products
“Margin DELTA” column (D) – allows you to add specific number to price after margin % action

Pretty prices like 4.99 or 0.95. How to do it?

Go to @Source settings and at @”Categories change + margin settings” section set @”Round delta” value: 0.05 or 0.01.
How to make pretty prices for extracted prices from web scraping service

How to apply margin based on Wholesale price range?

If you need to apply margin based on Wholesale price range you have to define your margin rules in excel config file: fine sample below, please
How to make ranges margin at web scraping service

How to do it
Go to Column (F – Margin rule) and put cell value to X
Insert new Worksheet and set it’s name to “Margin rule – X”
At “Margin rule – X” worksheet define your rules

Steps rules margins at web scraping service
Download sample here
https://drive.google.com/open?id=158BwK1xuBwRLCYtVb56q0TFYiiJ-Y6TV

Custom Categories management

If you want to manage categories itself you have to create TXT files with categories URLS at your computer.
How to setup start categories for web scraping service
Download sample here https://drive.google.com/open?id=1YBfU-qtoofcQBXyDjeoJWz_8o7cCOFHG

Keywords in this file are the next:
# – define category name and category level
[path] – splits category name and category url
For example
#cat1[path]https://www.aliexpress.com/category/100003109/women-clothing-accessories.html

When file is ready goto @sources->@Your source-Edit->@Categories filter page, goto the end of page and upload new file with categories to “Select custom grabCatalog” section and press @Save.
Example of config file with start urls for web scraping service

Custom Products management

If you want to select specific products for web scraping do the next: create similar to the next TXT file:

How to extract specific products or items at web scraping service

As it was shown at the previous image
Define category name, level & URL (URL is required). (Find how tp at “Custom Categories Management” part of this manual)
Under the “category line” (first line) define products urls (1 line is 1 product).

Download sample here

https://drive.google.com/open?id=1tHvadhNpenD3jtormWTzwjV-bO0AExc1

When file is ready upload it the same way as described in @”Custom Categories management” part of this manual.

Place for uploading config file at web scraping service

Hint
You can mix @”Custom Products management” & @”Custom Categories management” simultaneously.

How to run exports automatically

Go to @Profile->@Campaigns->@”You campaign #1” under Column Scheduler Click on @Edit

How to run export automatically at web scraping service
And define it in CRON format scheduler for export
How to setup scheduler at web scraping service
For example:
0 5
It means to start export at 5:00 daily