# How do you monitor scheduled AI jobs so they don't loop?

Date: 2026-03-14 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/monitor-scheduled-ai-jobs-so-they-dont/

**Answer:** Wrap every scheduled AI job in a small guard that writes one log line per run, refuses to start when a daily run count or monthly budget is used up, and alerts you on failures, unusual spending or missing runs. Add limits at the scheduler too, such as a timeout and a no-overlap rule, and keep one switch that stops everything at once.

A **scheduled AI job** is a program that calls an AI model on a timetable: a morning news digest, a nightly summary of support tickets, an hourly check of an inbox. Once it is running, nobody is watching. If a bug makes it run every minute instead of every day, or an agent inside it keeps calling tools without finishing, it can repeat for hours and spend real money before anyone looks.

Monitoring means answering three questions without having to remember to ask them: Did the job run? Did it behave? What did it cost? The answers come from a log you can read, limits the job checks before it starts, and alerts that reach you when something is off.

## The four parts of a monitoring setup

| Part | What it does | What it catches |
|---|---|---|
| Run log | One line per run: start time, status, steps, tokens, estimated cost | Everything, after the fact |
| Caps | The job refuses to start, or stops, when a count or budget is reached | Loops, runaway agents, schedule typos |
| Alerts | A message to you when a run fails, costs too much or doesn't happen | Problems while they are small |
| Kill switch | One setting that stops all runs immediately | Anything you didn't foresee |

A **token** is a small chunk of text; AI APIs bill by the number of tokens sent and received, so token counts are the most direct measure of spending.

## A worked example: a guard around any job

The script below wraps a job function, `run_job`, which stands for your own code that calls the model. It returns how many steps it took and how many tokens it used. The guard does the rest. The price and limits are illustrative; set them from your own provider's price list and your own budget.

```python
import json, os, time, urllib.request
from datetime import datetime, timezone
from pathlib import Path

LOG = Path("runs.jsonl")
STOP = Path("STOP")                    # create this file to halt every run
MAX_RUNS_PER_DAY = 3
MAX_STEPS = 10                         # passed to the job; agents must respect it
MONTHLY_BUDGET = 20.00                 # dollars, illustrative
PRICE_PER_M_TOKENS = 5.00              # illustrative blended price; use your model's real one
ALERT_URL = os.environ.get("ALERT_WEBHOOK")   # a chat or email webhook you control

def alert(text):
    print("ALERT:", text)
    if ALERT_URL:
        data = json.dumps({"text": text}).encode()
        urllib.request.urlopen(urllib.request.Request(
            ALERT_URL, data=data, headers={"Content-Type": "application/json"}), timeout=10)

def past_runs():
    if not LOG.exists():
        return []
    return [json.loads(line) for line in LOG.read_text().splitlines() if line]

def guarded_run(run_job):
    now = datetime.now(timezone.utc)
    runs = past_runs()
    today = [r for r in runs if r["start"][:10] == now.date().isoformat()]
    month_cost = sum(r.get("cost", 0) for r in runs if r["start"][:7] == now.strftime("%Y-%m"))

    if STOP.exists():
        return print("Kill switch on; not running.")
    if len(today) >= MAX_RUNS_PER_DAY:
        return alert(f"Run cap reached: {len(today)} runs today. Check the schedule.")
    if month_cost >= MONTHLY_BUDGET:
        return alert(f"Monthly budget used: ${month_cost:.2f}. Job paused.")

    entry = {"start": now.isoformat(), "status": "ok"}
    t0 = time.time()
    try:
        steps, tokens = run_job(max_steps=MAX_STEPS)
        entry.update(steps=steps, tokens=tokens, cost=tokens / 1e6 * PRICE_PER_M_TOKENS)
        if steps >= MAX_STEPS:
            entry["status"] = "hit_step_limit"
            alert(f"Job used all {MAX_STEPS} steps; it may be looping.")
    except Exception as e:
        entry.update(status="error", error=str(e)[:300])
        alert(f"Job failed: {e}")
    entry["seconds"] = round(time.time() - t0, 1)
    with LOG.open("a") as f:
        f.write(json.dumps(entry) + "\n")
    if month_cost + entry.get("cost", 0) > 0.8 * MONTHLY_BUDGET:
        alert("Over 80% of the monthly AI budget is used.")
```

Notice what the guard checks *before* calling the model: the kill switch, the run count and the budget. A loop caused by a bad schedule hits the run cap on its fourth attempt of the day, not its four-hundredth.

## Watching for the run that didn't happen

Loops are loud; silence is the quieter failure. A schedule can stop without an error. GitHub's documentation, for example, notes that scheduled workflows can be delayed when its service is busy, especially at the top of the hour, that some queued jobs may be dropped, and that in public repositories they are switched off after 60 days with no activity.

So add a second, separate check that runs once a day and alerts you if `runs.jsonl` has no line with status `ok` in the last 26 hours. That extra two hours absorbs normal delays without hiding a real stop. Engineers call this a **dead man's switch**: the alarm sounds when an expected signal is missing.

## Limits at the scheduler

If your job runs on GitHub Actions, two settings add a second layer of protection that works even when your code misbehaves. `timeout-minutes` cancels a job that runs too long; without it, GitHub allows 360 minutes. `concurrency` lets only one run in a named group happen at a time, so a slow run cannot pile up behind itself.

```yaml
on:
  schedule:
    - cron: "17 6 * * *"     # 06:17 UTC daily, away from the top of the hour
  workflow_dispatch:
concurrency:
  group: ai-digest
jobs:
  digest:
    if: vars.AI_JOB_ENABLED == 'true'   # a repository variable as a kill switch
    runs-on: ubuntu-latest
    timeout-minutes: 10
```

Because the runner starts fresh each time, store `runs.jsonl` somewhere that persists, such as a small database or a file your job commits back. To stop everything, set the variable to `false`, or disable the workflow from the Actions tab or with `gh workflow disable`.

> **Try it:** Set `MAX_RUNS_PER_DAY = 1`, run the guarded job twice by hand, and confirm the second attempt sends an alert instead of calling the model. Then create the `STOP` file and check that nothing runs at all.

## What limits should every routine have?

Every routine needs a cap on how often it may run, a cap on how much work one run may do, and a cap on what it may spend. For how often, use a daily run count slightly above the schedule you intend. For each run, use a step limit for agents, a maximum output length for single calls and a timeout. For spending, use both a monthly budget in your own code and the spending limit or budget alert most API providers offer in their account settings, so a bug in your code cannot bypass it.

Add one kill switch you can reach from your phone, such as a repository variable or a file on a shared drive, and test it before you need it. Finally, review the log weekly for the first month. Steady token counts, run times and costs are the sign a routine is healthy; a sudden jump is worth a look even when no alert fired.

## Sources

- Workflow syntax for GitHub Actions; Disabling and enabling a workflow: https://github.com/github/docs/blob/80198e3a68a3aa76edb3b402e33631dc29775540/content/actions/reference/workflows-and-actions/workflow-syntax.md
- Events that trigger workflows: https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows