Step 1. Install axios via command line at your working folder
[code]
npm install axios
[/code]
Step 2. write the next code
[code]
const axios = require(‘axios’);
var soUrl = ‘here is url that you want to scrape’;
const html = await axios.get(soUrl);
[/code]
html variable will be populated with html data.
Step 3. Use proxies to prevent blocking, use https-proxy-agent module for nodejs.
Proxies will help you make requests from different ip addresses.
[code]
const HttpsProxyAgent = require("https-proxy-agent");
const axios = require(‘axios’);
const httpsAgent = new HttpsProxyAgent({host: proxyHost, port: proxyPort})//, auth: "username:password"})
const axios2 = axios.create({httpsAgent});
const html = await axios2.get(urlYouWantToScrape, {
timeout:5000,
});
[/code]
