How to Create a Web Scraper with AI: AIScript User Guide

This guide shows how to build a web scraper in AIScript, the MyDataProvider AI web scraping tool, from the first task to a scheduled data feed. It covers how to write a task the AI understands on the first try, what happens during a build, how to read results, how to change a scraper later, and how to work with logins, files, price formulas and specific categories.

Quick Start: Your First Scraper in 5 Minutes

1. Sign up at script.mydataprovider.com with your email and confirm it. The 30-day trial needs no card.
2. Create a task. On the My tasks page press + New task, or start from one of the ready examples (website catalog parser, API export to Excel, price monitoring).
3. Describe what you need in the Describe the task for the AI agent box and press Start generating the script.
4. Confirm the plan the agent shows you, or add a correction.
5. Wait for the build. A simple feed or page takes about a minute, a catalog with detail pages about 5 to 10 minutes. When it is done, the result table and download links appear under Run script.

How to Write a Good Task

The agent is good at figuring out how to get data. What it cannot guess is what exactly you want. A good task answers four questions:

Example of a complete task:

Scrape all products from https://example-shop.com/tools/ including subcategories.
For each product open its page and collect: SKU, name, brand, price (number),
old price if there is a sale, stock status, category path, all image URLs, URL.
One row per variant (size/color), with the variant SKU and price.
Save as result.xlsx with a header row, and also result.csv (UTF-8, comma).

More tips:

• Say what "done" looks like. "The category has about 1,200 products" helps the agent notice when pagination stopped too early.
• Mention known obstacles: login, a city or ZIP code selector, "prices are only shown after choosing a size", a cookie banner.
• Describe transformations in the same task: currency conversion, markup, splitting a full name into columns, removing duplicates.
• Start smaller on huge sites. Ask for one category first, check the result, then widen the task with an edit.

What Happens During a Build

After you press Start generating the script the agent first shows how it understood the task: the steps, the fields, the output files and its assumptions. If something is off, type a correction ("the price must include VAT", "skip the outlet category") or press Looks right, go ahead.

Then the agent works in a loop: it inspects the site, writes the script, runs it in the sandbox, looks at the output and fixes problems. The status line shows the current step (Writing code, Analyzing the run result) and the live output of the test run. Details shows how the AI reasoned and what it ran. You can leave the page; the build continues on the server.

When the build is finished, the script is saved with the task and the result of its last run appears below. If the agent cannot reach a reliable result (the site blocks all requests, the data is not on the page), it says so instead of returning an empty file.

Running the Script and Reading Results

Run script starts the saved scraper again, without the AI. This is what you use for every update, and it is what the scheduler and the API do too. Runs do not use your monthly AI builds.

• Run history lists every run with its status, start and finish time, a link to its log and its files.
• Log shows the script output while it runs, so you can see progress on long jobs, and the full log afterwards. Stop cancels a run.

• Results open as a table preview with the number of rows and columns. Each file of the run has its own download link: .xlsx, .csv, .json, .xml, .txt or any other file the script wrote.

On the free tier and during the trial the preview and downloads include the first 50 rows; the paid plan unlocks the full files. See pricing.

Changing a Scraper with AI

To change a working scraper, open Change the script under the run history and write what should be different, as an instruction to the existing script. Earlier requests are listed above the box, so you see how the scraper evolved:

• "Add a column with the brand from the product page."
• "The site moved the price into a new block, fix the price column."
• "Also save a Shopify import CSV next to the Excel file."
• "Only keep products cheaper than 50 EUR."

The agent loads the current script, makes the change, tests it and saves it as a new version. Small changes are faster and more reliable than writing the whole task again.

Developers can get access to the Python code itself: write to us and we enable the code editor for your account.

Change History and Restoring Versions

Change history keeps every version of the script: who made it (AI build, AI edit or a manual change), when, and the request text that produced it. If a change made things worse, open the history and restore the previous version with one click. The restored version becomes the active one for manual runs, the schedule and the API.

Files for the AI: Docs, Samples, URL Lists

The Files button opens the task's files. Anything you upload there is available to the agent while it builds the script and to the script on every run. Typical uses:

• API documentation of a supplier or marketplace (PDF, HTML, text) so the agent uses the right endpoints and parameters.
• A sample of the output you need, for example your store's import template. Write "use exactly the columns of template.xlsx".
• Input lists: product URLs, SKUs, EANs, keywords or ZIP codes, one per line. Write "for each line of urls.txt…".
• Reference data: a category mapping table, a currency table, a list of SKUs to exclude.

The more context, the more accurate the result. Replacing a file (for example a new URL list) changes what the next run processes without any AI edit.

Secrets: Logins, Passwords and API Tokens

If the data is behind a login or an API key, add it in Script secrets instead of the task text. Give each secret a clear name, such as SUPPLIER_LOGIN, SUPPLIER_PASSWORD or SHOP_API_TOKEN, and mention in the task that the site needs a login. The agent wires the secrets into the script.

• Values are stored encrypted and passed to the script only as environment variables at run time.
• They are masked in logs and run history and are never written into the code.
• Each run is an isolated sandbox separated from other users.

When a password changes, update the secret; the script picks it up on the next run.

Scraping Only Specific Categories or Products

You do not need a special filter screen: the scope is part of the task. Common ways to define it:

• Category URLs. List the categories you need, with or without subcategories. You can also give your own category name for each: "https://shop.com/c/drills → Power tools / Drills".
• The site's own filters. Open the category on the site, apply brand, price or availability filters and paste the resulting URL. The script keeps the filters and walks all pages.
• A list of products. Upload a file with product URLs, SKUs or EANs and ask to process only those. Good for monitoring a fixed assortment.
• Rules. "Skip categories Outlet and Gift cards", "only brands Bosch and Makita", "only items added this month".

Mixing them is fine: whole categories plus a few extra product URLs from a file.

Price Rules and Markup Formulas

Scrapers for resellers usually need a retail price, not the supplier price. Describe the rule in the task and the script adds the calculated column to every export. The classic formula is:

new price = supplier price × (1 + margin % / 100) + fixed amount

For example, with a 10% margin a 100 USD product becomes 110 USD; with a 10% margin plus 2 USD it becomes 112 USD. Rules you can ask for:

Keep the original supplier price as its own column so you can always check the calculation. To push the result into a shop, see products import.

Exporting Only New or Changed Items

A script can keep files between runs. Ask for it in the task: "remember the products from the previous run and export only new products and price changes to changes.csv". The script stores its state with the task and compares each run with the previous one. This is useful for new-arrival alerts, price change reports and faster daily updates of big catalogs.

Limits and Troubleshooting

The result is empty or has fewer rows than expected. Open the log of the run. Often the site showed a captcha, a cookie wall or a region selector. Tell the agent what you see on the site ("the page asks to choose a country first") in an edit.

A scheduled run started failing. The site probably changed. Write "the last run failed, fix the script"; the agent reads the log of the failed run.

The site blocks requests. The agent slows down, uses a real browser or the site's own JSON API. Some marketplaces and heavily protected sites need our infrastructure: ask for a custom scraper.

Still stuck? Use contact and include the task name; we can look at the run log with you.