How to create a web scraper using nodejs and axios

Step 1. Install axios via command line at your working folder


npm install axios

Step 2. write the next code


    const axios = require('axios');

    var soUrl = 'here is url that you want to scrape';
    const html = await axios.get(soUrl);

html variable will be populated with html data. Step 3. Use proxies to prevent blocking, use https-proxy-agent module for nodejs. Proxies will help you make requests from different ip addresses.



        const HttpsProxyAgent = require(&quot;https-proxy-agent&quot;);
        const axios = require('axios');
        const httpsAgent = new HttpsProxyAgent({host: proxyHost, port: proxyPort})//, auth: &quot;username:password&quot;})
        const axios2 = axios.create({httpsAgent});

        
        const html = await axios2.get(urlYouWantToScrape, {
            timeout:5000,
          
          });