MyDataProvider » Blog » How to create a web scraper using nodejs and axios

How to create a web scraper using nodejs and axios

  • by

Step 1. Install axios via command line at your working folder

npm install axios

Step 2. write the next code 

    const axios = require('axios');

    var soUrl = 'here is url that you want to scrape';
    const html = await axios.get(soUrl);

html variable will be populated with html data.

Step 3. Use proxies to prevent blocking, use https-proxy-agent module for nodejs.

Proxies will help you make requests from different ip addresses.


        const HttpsProxyAgent = require("https-proxy-agent");
        const axios = require('axios');
        const httpsAgent = new HttpsProxyAgent({host: proxyHost, port: proxyPort})//, auth: "username:password"})
        const axios2 = axios.create({httpsAgent});

        
        const html = await axios2.get(urlYouWantToScrape, {
            timeout:5000,
          
          });