Web Scraping Service – User manual & FAQ

Contents

How to try service?

How to order service?

Open Web Scraping service page & send mail to sales team.

How to add Source (for Web Scraping)

Go to @Sources section and press @Add Button – from the list add necessary for you source.
How to add Source for web scraping service

How to select Categories for Source

Go to @Sources section and press on @Edit Button for targeted Source.
How to select Categories for Source for web scraping service
After that click on @”Categories Filter” and check necessary for you categories. Press @Save button.
How to check Categories for Source of web scraping service

How to export

Go to @Export section and click at “config Export” link.

Export to File

Go to “Export To File” section and define preferred export variant. If you want to import csv file directly to file format, for example use “CSV Shopify” or XML or JSON or Excel or CSV.

Direct export to your Store

“Export to Shop” turned off by default. If you want to Turn ON direct export to your store – check Export to Shop & Save “Export to your CMS database” section.
How to export to file or to online store from web scraping service

How to run export

Go to @Export section and click @Create button and on the next screen click @Confirm button.
How to run export from web scraping service
how to start web scraping service

How to pay

Go to @Invoices section and press on @”Get Invoice” button.
How to pay for web scraping service

Margin Setup

Go to @Sources-> @”Your source – Edit”->@”Categories change +margin” and Press @Donwload template. Template for Settings Margins for selected categories will be downloaded.
How to download template file for custom margin at web scraping service
It is Excel file – open it in Ms Excel or OpenOffice or LibreOffice editor.Sample file
How to setup custom margin for prices from web scraping service
Now you have to fill this file and upload to @“Upload new Settings” section.

Find below sample of filled file:
sample of web scraping service configuration file for margin setup

How does Margin work
Active formula for NEW Prices is the next:
“New Price” = (“Wholesale Price” * (1 + Margin %) ) + (Margin DELTA)

“Source category” column (A) – do not change it!
“Destination category” column (B) – allows you to change original category path.
“Margin % percent” column (C) – allows you to set your margin for exported products
“Margin DELTA” column (D) – allows you to add specific number to price after margin % action

Sample 1: calculate manually
Wholesales price is 100$
Margin Percent is 10%
Lets get results:

100$ * (1 + 10%/100) = 110$
or 100$*1,1=110$
where 1,1 is effective multiplier

Sample 2: what “Margin percent” should I set if I want just multiply “Wholesales price” at 1800
effective multiplier is 1800
Lets get results:

1 + %/100 = 1800
1800-1 = 1799
%/100 = 1799
% = 179900

So, “Margin percent” has to be 179900

Pretty prices like 4.99 or 0.95. How to do it?

Go to @Source settings and at @”Categories change + margin settings” section set @”Round delta” value: 0.05 or 0.01.
How to make pretty prices for extracted prices from web scraping service

How to apply margin based on Wholesale price range?

If you need to apply margin based on Wholesale price range you have to define your margin rules in excel config file: fine sample below, please
How to make ranges margin at web scraping service

How to do it
Go to Column (F – Margin rule) and put cell value to X
Insert new Worksheet and set it’s name to “Margin rule – X”
At “Margin rule – X” worksheet define your rules

Steps rules margins at web scraping service
Download sample here
https://drive.google.com/open?id=158BwK1xuBwRLCYtVb56q0TFYiiJ-Y6TV

Custom Categories management

If you want to manage categories itself you have to create TXT files with categories URLS at your computer.
How to setup start categories for web scraping service
Download sample here https://drive.google.com/open?id=1YBfU-qtoofcQBXyDjeoJWz_8o7cCOFHG

Keywords in this file are the next:
# – define category name and category level
[path] – splits category name and category url
For example
#cat1[path]https://www.aliexpress.com/category/100003109/women-clothing-accessories.html

When file is ready goto @sources->@Your source-Edit->@Categories filter page, goto the end of page and upload new file with categories to “Select custom grabCatalog” section and press @Save.
Example of config file with start urls for web scraping service

Custom Products management

If you want to select specific products for web scraping do the next: create similar to the next TXT file:

How to extract specific products or items at web scraping service

As it was shown at the previous image
Define category name, level & URL (URL is required). (Find how tp at “Custom Categories Management” part of this manual)
Under the “category line” (first line) define products urls (1 line is 1 product).

Download sample here

https://drive.google.com/open?id=1tHvadhNpenD3jtormWTzwjV-bO0AExc1

When file is ready upload it the same way as described in @”Custom Categories management” part of this manual.

Place for uploading config file at web scraping service

Hint
You can mix @”Custom Products management” & @”Custom Categories management” simultaneously.

How to run exports automatically

Go to @Profile->@Campaigns->@”You campaign #1” under Column Scheduler Click on @Edit

How to run export automatically at web scraping service
And define it in CRON format scheduler for export
How to setup scheduler at web scraping service
For example:
0 5
It means to start export at 5:00 daily

Demo

Video Demo : Banggood Web Scraping & prestashop

Video Demo : SammyDress Web Scraping & WooCommerce

Video Demo : Aliexpress Web Scraping & Shopify

FAQ

Do you extract ecommerce data?

We can extract e-commerce data:
– categories with subcategories & hierarchy
– products basic info: sku, price, name, description, images etc
– products features & options
– products categories

How do you export web scraping results?

We save results to csv, excel, xml, json files or to client’s database (we call it products import and support a lot of CMS databases)

I did not find in your data sources (Wholesalers List) my data source site.

Do not worry about it. We can create web scraper for any web site and we can create new web scraper for your specific data source. Data from any web site could be extracted.

Could you create web scraper for my specific site?

Yes. We could create new web scraper for your specific data source.

Is it possible to extract products from specific categories?

Yes. You could setup categories filter for web scraper.

Is it possible to filter products by SKU or Brand?

Yes. it is possible.

What do you do when source web site blocks requests?

We have private proxies (more then 20000 unique IP Addresses) so we could you it to do requests from different IP addresses.

Is it possible to run web scraping automatically?

Yes. You could setup scheduler in the App.

Is it possible to do data extraction quickly (increase speed)?

Yes. Inside the App exists special algorithm to parallel data extraction from sites.

How long does data extraction process take?

It depends on many factors: number of items what you need to process, number of requests what you need do to for 1 item, speed of Http response, time what you need to do data post processing. So the real time of web scraping for your specific web site we could know only after the real tests.

How many row fields is it possible to extract?

In theory it is possible to extract unlimited number of fields for each row. But in a real life we have to check that it is possible because sometimes it is impossible as required.

Is it possible to have access to extracted data via API?

Yes. Application has API and user can generate “SecretURL” to access fresh data from ANY programming language: php, python, java, golang, ruby, c# etc.

Is it possible to merge several data sources into 1 data export file (database)?

Yes. It is possible.

Is it possible to have more than 1 campaign under 1 account?

Yes. It is possible.

Could you export data to custom csv, xml etc format?

Yes. It is possible.

Custom Margin?

Sure. It is possible to setup custom margin for products for output export file or database & to have “your” specific prices for products.

Does your App work only with Prestashop & Banggood

No. App could work with any supplier and any CMS. For demo we showed Prestashop 1.6 & Banggood.

How does Web Scraping work

App runs tasks. Each task contains 2 steps.
Step1 – data extraction & Step2 – data import.
Data import is process of importing & updating products to your store via database or API.

I want to export only to CSV or XML or Excel etc without exporting to my database

It is possible: in export section you could define export format: CSV or XML or Excel etc.

What types of integrations do you support for suppliers

There are 3 ways how it could be integrated:
Way 1: Products info export to csv,xml,excel files.
Way 2: Products import & update via API (it works automatically & daily, without any file, by scheduler)
Way 3: Way 2 + orders processing in real time.

I want to get more info about how your App import products

Find more info about products import here

You need to try our web scraping services. Find here why

Here at MyDataProvider, we provide services that are convenient and rooted in excellence. Web scraping is supposed to make collecting information easier for you and that is what we want to deliver – a hassle free experience.

Our process of scraping information from different websites is pretty straightforward. The process is mostly automatic and bot reliant. It does not only raise the accuracy of the information that we can collect by minimizing human bias; it also makes the process easier and faster.

Try our Web Scraping Services Online

Welcome to Web Scraping software as a services “Mydataprovider”. You can scrap the web easily using “MyDataProvider Web Scraping Services” without worrying about backups, monitoring or servers. MyDataProvider manages everything for you.

We store the scraped data safely into our database where you can browse and share it from your dashboard.

MyDataProvider as a Service

MyDataProvider is the creator and maintainer of “MyDataProvider Web Scraping services”. These services are made available to users who don’t have time or tactics to scrap a website. We assure you to be in safe hands if you choose MyDataProvider web scraping service.

Benefits of using MyDataProvider as a service

MyDataProvider is proud to extract data/products from any website and deliver the information to its users in the formats of their choice. Our web scraping services are designed to scrape data easy and fast allowing you to get reliable data with minimal efforts.

1. You do not Need any Additional Software

Scraping the web sometimes can be difficult especially when scraping complicated websites. Don’t waste time finding a web scraping software, learning how it works and investing in it. With MyDataProvider web scraping service, we will help you scrape the web in the shortest time possible. Just relax, we are here for you.

2. Offers Advanced Scraping Techniques

The traditional and outdated methods have limits in their web scraping abilities. Some website requires authentication hence hard to scrap while others use JavaScript making their data unclear. With MyDataProvider service, you are sure to bypass such situations and get data that is usable.

3.Fast Data Delivery

As much as data delivery depends on the amount of data being scraped, MyDataProvider services assure its users to receive data in the shortest time possible.

4. Affordable

MyDataProvider understands that big budgets are everyones concern. This is the reason why we have made web scraping services affordable to all. Our pricing plan helps you cut unnecessary costs, and we assure our clients to get what they want.

5. Data is available in Any Format

We deliver data to customers in the formats they need. Data is available in CSV, XML, Excel formats.

Web Scraping & eCommerce. How to add value

Welcome to MyDataProvider Application where we provide excellent and reliable DropShipping Time Saver / Suppliers Integration & Web Scraping eCommerce service. Today, doing dropshipping business has become common, and that’s the reason we are here to offer the best solutions for you.

Why MyDataProvider Web Scraping Services are the Best for DropShipping

Point and Click System. You could add any site as a Data Source or supplier’s API
select web scraping source

To use MyDataProvider, you only need to point and click the data you want to be scraped. You do not need to have any programming or developer skills to do it.

Web Scraping eCommerce data

MyDataProvider can scrape data in any format that you want. You can get your data in Excel, CSV, XML or Json or direct export into your e-commerce store.

Just say what format you need your data in, we will do it for you.

Full customer support

MyDataProvider is proud to say that it provides full support to its users. There are ready tutorials on our website just for you!

In case of any troubleshooting, contact us and we will be more than ready to help you.

Product & Categories Web Scraping

web scraping categories
web scraping categories

All e-commerce websites display their products in categories. The categories can be arranged either by brand, types of products, prices, etc. With MyDataProvider, you can scrap all this information successfully.

Margin Setup for pricing rules

Improve your sales by setting up the best sales margins & setup your prices With MyDataProvider Application, you can use the data scraped from different e-commerce platforms to set up the perfect price for your products.

margin rules variant 1

margin variant 2
margin variant 2

Product Filter by Category or SKU

Ecommerce websites are the best sources of product data. If you are building your website, it is necessary to outdo your competitors with perfect pricing intelligence or doing a thorough market research. All this information is available from e-commerce sites.

If you want to scrape some SKUs from e-commerce sites, MyDataProvider are the perfect set of services for the job. Our dedicated e-commerce services can help you extract product details for your SKUs: prices, margin, discounts etc.

Different Sources Merge

You can scrap data from different websites and merge the information extracted as one. All this is possible with MyDataProvider.

Export Data in any Format

import export to csv
import export to csv

When you choose to export data using MyDataProvider, you are not restricted to any format. You can export data in CVS, XML, Excel formats.

Direct Export to your online store

import to cms
import to cms

MyDataProvider users can export products to Shopify, Woocommerce, Prestashop, Magento, Opencart stores. Users only need to copy the product URLs into MyDataProvider App. They use our API to access data from their MyDataProvider application.

Benefits for using MyDataProvider for Ecommerce Solutions

  • Users can export data in their store formats.
  • You can scrap only categories that you find relevant
  • MyDataProvider provides 100% and timely support by Email and skype/phone.

Web Scraping Software

Welcome to MyDataProvider where you can quickly turn website content into structured and usable data without any coding, programming or developer skills.

We are proud to say that the data extracted can be used for price monitoring, market research and lead generation and give positive results.

How does MyDataProvider Web Scraping Software work?

Step 1: Log in into MyDataProvider web scraping App

Setup environment.

Step 2: Identify and Collect Data

Use MyDataProvider Application to extract information from any website into a structured format.

Step3: Structure and Publish Data

You can export data in CSV, XML, Excel, SQLite, PostgreSQL formats or pull data using MyDataProvider API.

Step 4: Analyze, Report and Make Decisions

Send data extracted into your database for visualization and decision making.

What types of repricing do you support?

A1. There are three types.
Type1: You have 1 supplier and you need just add to WholesalePrice your % (simple formula)
Type2: Type1 + you have more that 2 supplier & 1 real product could be sold by more than 2 suppliers (so you need to use the best WholesalePrice for you)
Type3: Type2 + you want to use competitors prices when you do repricing.

What types of formulas do you support for repricing?

The next formula is supported: NewPrice = WholesalePrice*X + Y
Hints:
Y – you could use it as a fixed price for shipping.
X – you could use it as a margin % or Currency Converter & margin %.

Is it possible to setup different formulas for different categories?

Yes.

Do you support ‘Pretty’ Prices?

Yes. You could do any “Pretty” Prices: like 0.99, 4.95 etc.

Is it possible to setup custom price formula which will be depends on Wholesale price range?

Yes. It is possible. If for example WholesalePrice is in range 0..5 $ use X1 & Y1 for formula, If WholesalePrice is in Range 5.20$ use X2,Y2.

Could I setup custom formulas for custom data sources?

Yes. it is possible.