AI

Have a scraping or integration task? Let our AI solve it.

Describe what data you need or which systems to connect — our AI builds and runs the scraper or integration for you. No coding required.

Free to try · already have an account?

How to create a web scraper using nodejs and axios

Step 1. Install axios via command line at your working folder 

npm install axios
Step 2. write the next code 

    const axios = require('axios');

    var soUrl = 'here is url that you want to scrape';
    const html = await axios.get(soUrl);
html variable will be populated with html data. Step 3. Use proxies to prevent blocking, use https-proxy-agent module for nodejs. Proxies will help you make requests from different ip addresses. 


        const HttpsProxyAgent = require("https-proxy-agent");
        const axios = require('axios');
        const httpsAgent = new HttpsProxyAgent({host: proxyHost, port: proxyPort})//, auth: "username:password"})
        const axios2 = axios.create({httpsAgent});

        
        const html = await axios2.get(urlYouWantToScrape, {
            timeout:5000,
          
          });

Related Articles