Scheduled Web Scraping: Run Your Scraper Automatically
Product data gets old fast: prices change, items go out of stock, new listings appear. The scheduler in AIScript reruns your saved scraper on its own, every few hours, every day or every week, so a fresh file is always waiting in the task and behind its export API. Scheduled runs use the saved script and do not spend AI builds.
How to Set a Schedule
- Open the task and check that a manual run gives the result you want.
- Press Schedule.
- Choose the frequency, turn the schedule on and save. The button then shows when the next run starts.
| Mode | Settings | Example |
|---|---|---|
| Every N hours | Interval in hours | Every 6 hours for competitor prices |
| Daily | Time of day | Every day at 05:00, before the store opens |
| Weekly | Day of the week and time | Monday 03:00 for a full catalog with descriptions |
No CRON syntax is needed. If you need two rhythms, for example prices every 4 hours and a full catalog weekly, create two tasks: clone the task and change its prompt and schedule.
How Often to Scrape
| Data | Recommended frequency | Why |
|---|---|---|
| Competitor prices for repricing | Every 2 to 6 hours, or daily | Depends on how often the market moves; see price monitoring |
| Supplier stock for dropshipping | 1 to 4 times a day | Do not sell items the supplier no longer has |
| Supplier price list behind a login | Daily | Suppliers usually update once a day |
| Full catalog with descriptions and images | Weekly | This content rarely changes, and big catalogs take hours |
| New listings (classifieds, jobs, real estate) | Every 1 to 3 hours | Combine with "export only new items" |
Running more often than the data changes does not make it better and puts extra load on the source site. A common setup is a fast run for prices and stock plus a weekly full export. To save time on big sites, ask the AI to export only new or changed items.
What Happens on Each Run
- The saved script starts in a fresh sandbox, with your secrets and task files available.
- A run can last up to about 5 hours, enough for catalogs with tens of thousands of products.
- The run appears in the run history with its log and files. The export API switches to the new files as soon as the run succeeds; a failed run never replaces the last good file.
- Only one run of a task is active at a time, so a slow run is not overlapped by the next one.
Automatic First Run After a Build
Many people describe a task, press generate and close the tab. So after a successful build AIScript plans one automatic run a minute later, and the full result is ready when you come back. The countdown is shown in the task; any manual, scheduled or API run cancels it.
Auto-Disable When Nobody Uses the Data
Integrations get forgotten. In the schedule settings you can set "Disable if there were no API calls for N days". If no system has read the export through the API for that long, the schedule turns itself off and you get an email. Saving the schedule again turns it back on. This keeps unused scrapers from loading source sites.
When a Scheduled Run Fails
Sites change their layout, add protection or go down. Open the failed run's log to see what happened. If the site changed, open Change the script and write "the last scheduled run failed, fix it": the agent reads the log and repairs the script. Until then, the API keeps returning the last successful file.
Plans
Scheduled runs are part of the paid plan ($20 a month); on the free tier the schedule is paused. Every account starts with a 30-day free trial. See pricing. For a managed feed with monitoring and a guaranteed delivery time, see the custom web scraping service.
Have a scraping or integration task? Let our AI solve it.
Describe what data you need or which systems to connect — our AI builds and runs the scraper or integration for you. No coding required.
Free for 30 days, no card needed · already have an account?
Frequently Asked Questions
How do I schedule a web scraper to run daily?
Open the task in AIScript, press Schedule, choose Daily, set the time, turn it on and save. The saved script will run every day at that time and the new file appears in the task and in the export API.
Can a scraper run every hour?
Yes. Choose Every N hours and set 1. For most data every 2 to 6 hours is enough and puts less load on the source site.
Do scheduled runs use AI credits?
No. A scheduled run executes the saved script. AI builds are used only when you create or change a scraper.
How do I get the scheduled results automatically?
Read them from the export API: one permanent URL always returns the file of the latest successful run. Excel, Google Sheets and Power BI can refresh from it on their own.
What if the website changes and the scheduled run breaks?
The failed run is shown in the history with its log, and the API keeps serving the last good file. Ask the AI to fix the script in one sentence and the next run uses the fixed version.