This is the complete IndexNow auto-submission setup we run in production: GitHub Actions on a daily cron, a retry queue that never loses a URL, endpoint fallback, and a second authenticated Bing channel, all free. It exists because a 429 once silently cost us ten days of URLs; the root cause and the proof are inside. It is the setup we’d build first today, 429 or no 429.
Two ways to read this manual
Setting up IndexNow auto-submission? This is the setup we recommend and run ourselves. Jump straight to What you get and build from there: the queue, the fallback, and the second channel from day one, for free.
Coming from a 429? From a Cloudflare Worker, a WordPress SEO plugin on shared hosting, or any setup that submits from a shared IP: read on. The next four sections explain what the error means, why your code was never the problem, and the bug that turns a rate limit into silent data loss.
The incident: how we learned this the hard way
You run IndexNow from a Cloudflare Worker, the setup from our IndexNow manual: fetch the sitemap daily, diff it against a snapshot, push new and removed URLs to the IndexNow API. That exact Worker ran on our own site for three months without a bad day.
Then, at the start of July, every daily email started reporting the same thing:
Day after day. Big batches, small batches, even a single URL: all 429. Meanwhile 115 new glossary pages were waiting to be indexed.
What 429 means
The IndexNow API returns a small set of response codes: 200 means success, 202 means accepted with key validation still pending, 400 bad request, 403 key file not found, 422 URLs don’t belong to the host, and 429, too many requests.
The important detail, and one the IndexNow spec doesn’t document: our testing shows 429 is a per-IP verdict, not a per-site one. In every test we ran, the limit tracked the IP address the request came from, not the host, the key, or how many URLs were in the payload (the same-payload proof is two sections down). One request carrying 10,000 URLs cost the rate limiter exactly as much as one request carrying a single URL. Splitting a batch into smaller chunks makes 429 more likely, not less: more requests from the same IP.
Which raises the obvious question: our Worker made one request per day. How is one request per day "too many"?
Why Cloudflare Workers get 429’d
Cloudflare Workers don’t have their own IP address. Outbound requests leave through egress IPs shared by thousands of other people’s Workers. Running IndexNow pings on a Worker is a popular pattern (we wrote a manual about it ourselves). By the time your one daily request reaches the IndexNow API, the shared IP’s quota has already been burned by strangers.
That also explains the on-and-off pattern. Our submissions worked in April and May, then failed for ten straight days in July. Cloudflare rotates which egress IPs your Worker uses. Some are clean, some are burned, and which one your run gets is chance.
Workers aren’t the only setup with this problem. The same mechanics apply anywhere your submission leaves through an IP you share: WordPress SEO plugins (Rank Math’s own support docs describe exactly this 429 on shared hosting, where other sites on the same server burn the quota), serverless functions on any provider, and container platforms with pooled egress. If your IndexNow requests share an exit IP with strangers, their volume is your rate limit.
How to prove it’s the IP and not you
Send the exact same payload from any other machine. We did:
Same JSON body, same key, same host:
from the Cloudflare Worker → 429, 429, 429 (Bing, Seznam, api.indexnow.org)
from a random clean machine → 202 on the first try
Same JSON body, same key, same host:
from the Cloudflare Worker → 429, 429, 429 (Bing, Seznam, api.indexnow.org)
from a random clean machine → 202 on the first try
Same JSON body, same key, same host:
from the Cloudflare Worker → 429, 429, 429 (Bing, Seznam, api.indexnow.org)
from a random clean machine → 202 on the first try
If the identical request succeeds from a different IP, there is nothing wrong with your payload, your key, or your volume. No amount of retrying, batch-splitting, or key rotation fixes it from inside a shared-IP platform. The classic advice for 429 (pause submissions, slow down) doesn’t apply, because your own volume was never the problem.
The bug that makes 429 dangerous
Here is the part that cost us. The original Worker did this:
await env.SITEMAP_KV.put(KV_KEY, JSON.stringify(currentUrls));
await env.SITEMAP_KV.put(KV_KEY, JSON.stringify(currentUrls));
await env.SITEMAP_KV.put(KV_KEY, JSON.stringify(currentUrls));
It updated the sitemap snapshot regardless of whether the submission succeeded. On a 429 day, the new URLs were marked as "already seen" and never submitted again. No error, no retry, no trace. The daily email looked routine unless you read the status line.
Over ten days of 429s we silently lost 164 URLs, including an entire glossary launch, and only noticed because the status codes in the notification emails changed.
The design rule that falls out of this: "seen" and "successfully submitted" are two different facts and need two different records. Anything that failed to submit goes into a pending queue and gets retried on the next run. With that queue in place, a 429 day costs you one day of latency instead of the URLs themselves.
What you get
One GitHub repository that submits your new and removed URLs to search engines every day and survives the failures that break simpler setups. Whether you arrived here through a 429 or are building from scratch, the pipeline is the same, and its value is in three defenses most IndexNow tutorials skip.
Defense 1: a pending queue, so nothing is ever lost
Failed URLs stay queued in the state file and are retried on every following run until they get through. "Seen" and "successfully submitted" are two separate records, so a bad day (a 429, an outage, a timeout) costs you one day of latency instead of the URLs themselves. This is the defense that would have saved our 164 URLs, and it matters on every platform: most published IndexNow scripts, including our own original Worker, mark URLs as done the moment they try to submit them.
Defense 2: endpoint fallback with retry
IndexNow has several engine endpoints, and submitting to any one of them propagates to all participating engines. The script tries www.bing.com/indexnow first, then search.seznam.cz/indexnow, then api.indexnow.org, with one retry after 5 seconds on 429/5xx and no pointless retry on config errors like 403. One endpoint having a bad day stops mattering.
Defense 3: a second, authenticated channel
The Bing Webmaster URL Submission API is tied to your verified site, not your IP, so shared-IP reputation can’t touch it. New URLs go through it in parallel with IndexNow. Quotas are per-site (ours is 100 URLs/day, visible in Bing Webmaster Tools). Even on a day when every IndexNow endpoint rejects you, Bing still hears about your new pages the same afternoon.
Why GitHub Actions as the host
The submission logic doesn’t need to live next to your site. It needs a daily timer, a place to store state, and an IP address IndexNow hasn’t rate-limited. GitHub Actions has all three, for free:
Clean IPs. Workflow runs go out through GitHub’s runner pool, which behaves far better with IndexNow than shared Worker or shared-hosting egress. Our first submission from a runner: 200.
A daily timer. A cron schedule in the workflow file, plus a manual "Run workflow" button.
State storage. A JSON file committed back to the repo by the workflow itself. Every run’s diff is a visible commit in the history, which beats a KV namespace you can’t easily inspect.
Cost: zero. A run takes about 10 seconds, but GitHub rounds every job up to a whole minute, so a daily schedule bills about 30 minutes a month. Private repos get 2,000 free Actions minutes per month, so this stays free with room to spare.
Bonus discovery: Framer serves your key file natively
Our original manual routed the IndexNow key file (yoursite.com/YOUR_KEY.txt) through the Worker, with a proxied DNS record and a Worker Route. It turns out Framer can do this with zero infrastructure: from the dashboard, Domains → your domain → Files lets you upload any static file and serve it on a fixed URL at your domain root. It’s a Pro/Scale/Enterprise feature, and its home in the dashboard has already moved once (it started under Site Settings), so treat the menu path as approximate and check Framer’s current help docs if it doesn’t match what you see.
Upload a plain-text file named YOUR_KEY.txt containing just the key, and https://yoursite.com/YOUR_KEY.txt serves it directly from Framer. No Worker, no route, no proxied DNS. If the key file was the only reason you kept a Worker around, you can delete the Worker entirely.
What you need
A website with a flat sitemap at yoursite.com/sitemap.xml
A GitHub account (free plan is fine; the repo can be private)
Your IndexNow key served at yoursite.com/YOUR_KEY.txt (on Framer Pro/Scale/Enterprise: Domains → your domain → Files; on other plans, keep the key file on the Worker/DNS path from our first manual)
Optional: a Bing Webmaster Tools account with your site verified, for the second channel
Optional: a Resend API key for email summaries
No terminal is strictly required. Every file below can be created through GitHub’s web editor, but the git route is faster if you have it.
Step 1: Create the repository
Create a new private repository, e.g. yourorg/indexnow. It will hold three files:
.github/workflows/indexnow-daily.yml ← the schedule
scripts/indexnow-submit.mjs ← the logic
scripts/indexnow-state.json ← the state (written by the workflow)
.github/workflows/indexnow-daily.yml ← the schedule
scripts/indexnow-submit.mjs ← the logic
scripts/indexnow-state.json ← the state (written by the workflow)
.github/workflows/indexnow-daily.yml ← the schedule
scripts/indexnow-submit.mjs ← the logic
scripts/indexnow-state.json ← the state (written by the workflow)
Step 2: The workflow file
Create .github/workflows/indexnow-daily.yml:
name: IndexNow daily submission
on:
schedule:
- cron: "15 12 * * *" # daily, 12:15 UTC — pick your own time
workflow_dispatch: # manual "Run workflow" button
permissions:
contents: write # to commit the updated state file
concurrency:
group: indexnow
cancel-in-progress: false
jobs:
submit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5 # verified against runner v2.327.1+, July 2026
- name: Submit new/removed URLs
env:
INDEXNOW_KEY: ${{ secrets.INDEXNOW_KEY }}
BING_WMT_API_KEY: ${{ secrets.BING_WMT_API_KEY }}
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
TO_EMAIL: you@yoursite.com
FROM_EMAIL: indexnow@updates.yoursite.com
run: node scripts/indexnow-submit.mjs
- name: Commit updated state
run: |
if ! git diff --quiet scripts/indexnow-state.json; then
git config user.name "indexnow-bot"
git config user.email "actions@users.noreply.github.com"
git add scripts/indexnow-state.json
git commit -m "indexnow: update submission state"
git push
else
echo "State unchanged."
finame: IndexNow daily submission
on:
schedule:
- cron: "15 12 * * *" # daily, 12:15 UTC — pick your own time
workflow_dispatch: # manual "Run workflow" button
permissions:
contents: write # to commit the updated state file
concurrency:
group: indexnow
cancel-in-progress: false
jobs:
submit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5 # verified against runner v2.327.1+, July 2026
- name: Submit new/removed URLs
env:
INDEXNOW_KEY: ${{ secrets.INDEXNOW_KEY }}
BING_WMT_API_KEY: ${{ secrets.BING_WMT_API_KEY }}
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
TO_EMAIL: you@yoursite.com
FROM_EMAIL: indexnow@updates.yoursite.com
run: node scripts/indexnow-submit.mjs
- name: Commit updated state
run: |
if ! git diff --quiet scripts/indexnow-state.json; then
git config user.name "indexnow-bot"
git config user.email "actions@users.noreply.github.com"
git add scripts/indexnow-state.json
git commit -m "indexnow: update submission state"
git push
else
echo "State unchanged."
finame: IndexNow daily submission
on:
schedule:
- cron: "15 12 * * *" # daily, 12:15 UTC — pick your own time
workflow_dispatch: # manual "Run workflow" button
permissions:
contents: write # to commit the updated state file
concurrency:
group: indexnow
cancel-in-progress: false
jobs:
submit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5 # verified against runner v2.327.1+, July 2026
- name: Submit new/removed URLs
env:
INDEXNOW_KEY: ${{ secrets.INDEXNOW_KEY }}
BING_WMT_API_KEY: ${{ secrets.BING_WMT_API_KEY }}
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
TO_EMAIL: you@yoursite.com
FROM_EMAIL: indexnow@updates.yoursite.com
run: node scripts/indexnow-submit.mjs
- name: Commit updated state
run: |
if ! git diff --quiet scripts/indexnow-state.json; then
git config user.name "indexnow-bot"
git config user.email "actions@users.noreply.github.com"
git add scripts/indexnow-state.json
git commit -m "indexnow: update submission state"
git push
else
echo "State unchanged."
fiNote the schedule offset (:15 rather than :00): GitHub delays on-the-hour crons when everyone piles onto them.
The checkout action is pinned to v5 and verified against runner v2.327.1 and later as of July 2026. Later majors work here too. v6.0.0 raised the minimum runner to v2.329.0 and moved credentials to $RUNNER_TEMP, and v7.0.0 blocks fork-PR checkout for pull_request_target and workflow_run. None of that changes a scheduled workflow checking out its own repository on a GitHub-hosted runner.
Step 3: The script
Create scripts/indexnow-submit.mjs. This is the full production script. The pending queue, endpoint fallback, retry, chunking, Bing API channel, and email summary are all in here:
import { readFile, writeFile } from "node:fs/promises";
import { fileURLToPath } from "node:url";
const STATE_PATH = fileURLToPath(new URL("./indexnow-state.json", import.meta.url));
export const SETTINGS = {
retryDelayMs: 5000,
attemptsPerEndpoint: 2,
maxBatch: 10000,
bingApiMaxPerRun: 100
};
export const ENDPOINTS = [
"https://www.bing.com/indexnow",
"https://search.seznam.cz/indexnow",
"https://api.indexnow.org/indexnow"
];
export const HOSTS = [
{
host: "yoursite.com",
sitemap: "https://yoursite.com/sitemap.xml",
bingSiteUrls: ["http://yoursite.com", "https://yoursite.com"]
}
];
const env = process.env;
const DRY_RUN = env.DRY_RUN === "1";
async function main() {
if (!env.INDEXNOW_KEY) {
console.error("FATAL: INDEXNOW_KEY is not set");
process.exit(1);
}
let state = {};
try { state = JSON.parse(await readFile(STATE_PATH, "utf8")); }
catch { console.log("No state file yet — first run submits the full sitemap."); }
const reports = [];
for (const h of HOSTS) {
try {
reports.push(await runHost(h, state));
} catch (e) {
reports.push({ host: h.host, error: e.message });
}
}
if (!DRY_RUN) {
await writeFile(STATE_PATH, JSON.stringify(state, null, 1) + "\n");
}
const noteworthy = reports.some(r => r.error || r.submission);
if (noteworthy && !DRY_RUN) {
try { await sendSummaryNotification(reports); }
catch (err) { console.error("Email notification failed:", err.message); }
}
for (const r of reports) {
console.log(JSON.stringify(r, null, 2));
}
if (reports.every(r => r.error)) process.exit(1);
}
function fetchWithTimeout(url, opts = {}) {
return fetch(url, { ...opts, signal: AbortSignal.timeout(30_000) });
}
async function runHost(config, state) {
const sitemapRes = await fetchWithTimeout(config.sitemap);
if (!sitemapRes.ok) {
return { host: config.host, error: `Sitemap fetch failed: ${sitemapRes.status}` };
}
const xml = await sitemapRes.text();
const currentUrls = [...xml.matchAll(/<loc>(.*?)<\/loc>/g)]
.map(m => m[1].trim())
.filter(u => {
try { return new URL(u).hostname === config.host; }
catch { return false; }
});
if (currentUrls.length === 0) {
return { host: config.host, error: "No matching-host URLs found in sitemap" };
}
const hostState = state[config.host] ?? { last_urls: [], pending_urls: [] };
const previousSet = new Set(hostState.last_urls);
const currentSet = new Set(currentUrls);
const newUrls = currentUrls.filter(u => !previousSet.has(u));
const removedUrls = hostState.last_urls.filter(u => !currentSet.has(u));
const pending = [...new Set([...hostState.pending_urls, ...newUrls, ...removedUrls])];
state[config.host] = { last_urls: currentUrls, pending_urls: pending };
if (pending.length === 0) {
return { host: config.host, totalUrls: currentUrls.length, newUrls: [], removedUrls: [], retriedCount: 0, submission: null };
}
if (DRY_RUN) {
return { host: config.host, totalUrls: currentUrls.length, newUrls, removedUrls,
retriedCount: hostState.pending_urls.length, submission: { dryRun: true, wouldSubmit: pending.length } };
}
const submission = await submitToIndexNow(config.host, env.INDEXNOW_KEY, pending);
state[config.host].pending_urls = submission.failedUrls;
let bingApi = null;
if (env.BING_WMT_API_KEY && config.bingSiteUrls.length > 0 && newUrls.length > 0) {
bingApi = await submitToBingApi(config.bingSiteUrls, newUrls.slice(0, SETTINGS.bingApiMaxPerRun));
if (newUrls.length > SETTINGS.bingApiMaxPerRun) {
bingApi.note = `capped at ${SETTINGS.bingApiMaxPerRun} of ${newUrls.length} new URLs (daily quota)`;
}
}
return {
host: config.host,
totalUrls: currentUrls.length,
newUrls,
removedUrls,
retriedCount: hostState.pending_urls.length,
submission,
bingApi
};
}
export async function submitToIndexNow(host, key, urls) {
const attempts = [];
const failedUrls = [];
let acceptedCount = 0;
const chunks = [];
for (let i = 0; i < urls.length; i += SETTINGS.maxBatch) {
chunks.push(urls.slice(i, i + SETTINGS.maxBatch));
}
for (let i = 0; i < chunks.length; i++) {
const label = chunks.length > 1 ? ` [batch ${i + 1}/${chunks.length}]` : "";
const ok = await submitChunk(host, key, chunks[i], attempts, label);
if (ok) acceptedCount += chunks[i].length;
else failedUrls.push(...chunks[i]);
}
return { ok: failedUrls.length === 0, acceptedCount, failedUrls, attempts };
}
async function submitChunk(host, key, chunk, attempts, label) {
const body = JSON.stringify({
host,
key,
keyLocation: `https://${host}/${key}.txt`,
urlList: chunk
});
for (const endpoint of ENDPOINTS) {
for (let attempt = 1; attempt <= SETTINGS.attemptsPerEndpoint; attempt++) {
let status;
try {
const res = await fetchWithTimeout(endpoint, {
method: "POST",
headers: { "Content-Type": "application/json; charset=utf-8" },
body
});
status = res.status;
} catch (err) {
status = `network error: ${err.message}`;
}
attempts.push(`${endpoint}${label} (attempt ${attempt}): ${status}`);
if (status === 200 || status === 202) return true;
const retryable = status === 429 || status === 500 || status === 503 || typeof status === "string";
if (!retryable) break;
if (attempt < SETTINGS.attemptsPerEndpoint) await sleep(SETTINGS.retryDelayMs);
}
}
return false;
}
async function submitToBingApi(siteUrls, urls) {
const attempts = [];
for (const siteUrl of siteUrls) {
let status, detail = "";
try {
const res = await fetchWithTimeout(`https://ssl.bing.com/webmaster/api.svc/json/SubmitUrlBatch?apikey=${env.BING_WMT_API_KEY}`, {
method: "POST",
headers: { "Content-Type": "application/json; charset=utf-8" },
body: JSON.stringify({ siteUrl, urlList: urls })
});
status = res.status;
if (!res.ok) detail = (await res.text()).slice(0, 200);
} catch (err) {
status = `network error: ${err.message}`;
}
attempts.push(`SubmitUrlBatch ${siteUrl}: ${status} ${detail}`.trim());
if (status === 200) return { ok: true, submitted: urls.length, attempts };
}
return { ok: false, submitted: 0, attempts };
}
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
async function sendSummaryNotification(reports) {
if (!env.RESEND_API_KEY || !env.TO_EMAIL) return;
const fromEmail = env.FROM_EMAIL || "onboarding@resend.dev";
const failedHosts = reports.filter(r => r.error || (r.submission && !r.submission.ok));
const totalNew = reports.reduce((n, r) => n + (r.newUrls?.length ?? 0), 0);
const totalRemoved = reports.reduce((n, r) => n + (r.removedUrls?.length ?? 0), 0);
const subject = failedHosts.length > 0
? `IndexNow: FAILED (${failedHosts.map(r => r.host).join(", ")}), queued for retry`
: `IndexNow: OK — ${totalNew} new, ${totalRemoved} removed`;
const lines = [];
lines.push(`IndexNow daily run (GitHub Actions) — ${reports.length} host(s)`);
lines.push(`Timestamp: ${new Date().toISOString()}`);
lines.push(``);
for (const r of reports) {
lines.push(`=== ${r.host} ===`);
if (r.error) { lines.push(` ERROR: ${r.error}`, ``); continue; }
if (!r.submission) { lines.push(` Nothing changed (${r.totalUrls} URLs in sitemap).`, ``); continue; }
const s = r.submission;
lines.push(` IndexNow: ${s.ok ? "OK" : `FAILED, ${s.failedUrls.length} URLs queued for retry`}`);
lines.push(` Accepted: ${s.acceptedCount} URLs`);
if (r.retriedCount > 0) lines.push(` Retried from previous failed runs: ${r.retriedCount} URLs`);
lines.push(` Attempts:`);
s.attempts.forEach(a => lines.push(` ${a}`));
if (r.bingApi) {
lines.push(` Bing Submission API: ${r.bingApi.ok ? `OK — ${r.bingApi.submitted} URLs` : "FAILED"}${r.bingApi.note ? ` (${r.bingApi.note})` : ""}`);
r.bingApi.attempts.forEach(a => lines.push(` ${a}`));
}
if (r.newUrls.length > 0) {
lines.push(` New URLs (${r.newUrls.length}):`);
r.newUrls.forEach(u => lines.push(` ${u}`));
}
if (r.removedUrls.length > 0) {
lines.push(` Removed URLs (${r.removedUrls.length}):`);
r.removedUrls.forEach(u => lines.push(` ${u}`));
}
lines.push(``);
}
const res = await fetchWithTimeout("https://api.resend.com/emails", {
method: "POST",
headers: {
"Authorization": `Bearer ${env.RESEND_API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
from: fromEmail,
to: env.TO_EMAIL,
subject,
text: lines.join("\n")
})
});
if (!res.ok) throw new Error(`Resend API ${res.status}: ${await res.text()}`);
}
main();
import { readFile, writeFile } from "node:fs/promises";
import { fileURLToPath } from "node:url";
const STATE_PATH = fileURLToPath(new URL("./indexnow-state.json", import.meta.url));
export const SETTINGS = {
retryDelayMs: 5000,
attemptsPerEndpoint: 2,
maxBatch: 10000,
bingApiMaxPerRun: 100
};
export const ENDPOINTS = [
"https://www.bing.com/indexnow",
"https://search.seznam.cz/indexnow",
"https://api.indexnow.org/indexnow"
];
export const HOSTS = [
{
host: "yoursite.com",
sitemap: "https://yoursite.com/sitemap.xml",
bingSiteUrls: ["http://yoursite.com", "https://yoursite.com"]
}
];
const env = process.env;
const DRY_RUN = env.DRY_RUN === "1";
async function main() {
if (!env.INDEXNOW_KEY) {
console.error("FATAL: INDEXNOW_KEY is not set");
process.exit(1);
}
let state = {};
try { state = JSON.parse(await readFile(STATE_PATH, "utf8")); }
catch { console.log("No state file yet — first run submits the full sitemap."); }
const reports = [];
for (const h of HOSTS) {
try {
reports.push(await runHost(h, state));
} catch (e) {
reports.push({ host: h.host, error: e.message });
}
}
if (!DRY_RUN) {
await writeFile(STATE_PATH, JSON.stringify(state, null, 1) + "\n");
}
const noteworthy = reports.some(r => r.error || r.submission);
if (noteworthy && !DRY_RUN) {
try { await sendSummaryNotification(reports); }
catch (err) { console.error("Email notification failed:", err.message); }
}
for (const r of reports) {
console.log(JSON.stringify(r, null, 2));
}
if (reports.every(r => r.error)) process.exit(1);
}
function fetchWithTimeout(url, opts = {}) {
return fetch(url, { ...opts, signal: AbortSignal.timeout(30_000) });
}
async function runHost(config, state) {
const sitemapRes = await fetchWithTimeout(config.sitemap);
if (!sitemapRes.ok) {
return { host: config.host, error: `Sitemap fetch failed: ${sitemapRes.status}` };
}
const xml = await sitemapRes.text();
const currentUrls = [...xml.matchAll(/<loc>(.*?)<\/loc>/g)]
.map(m => m[1].trim())
.filter(u => {
try { return new URL(u).hostname === config.host; }
catch { return false; }
});
if (currentUrls.length === 0) {
return { host: config.host, error: "No matching-host URLs found in sitemap" };
}
const hostState = state[config.host] ?? { last_urls: [], pending_urls: [] };
const previousSet = new Set(hostState.last_urls);
const currentSet = new Set(currentUrls);
const newUrls = currentUrls.filter(u => !previousSet.has(u));
const removedUrls = hostState.last_urls.filter(u => !currentSet.has(u));
const pending = [...new Set([...hostState.pending_urls, ...newUrls, ...removedUrls])];
state[config.host] = { last_urls: currentUrls, pending_urls: pending };
if (pending.length === 0) {
return { host: config.host, totalUrls: currentUrls.length, newUrls: [], removedUrls: [], retriedCount: 0, submission: null };
}
if (DRY_RUN) {
return { host: config.host, totalUrls: currentUrls.length, newUrls, removedUrls,
retriedCount: hostState.pending_urls.length, submission: { dryRun: true, wouldSubmit: pending.length } };
}
const submission = await submitToIndexNow(config.host, env.INDEXNOW_KEY, pending);
state[config.host].pending_urls = submission.failedUrls;
let bingApi = null;
if (env.BING_WMT_API_KEY && config.bingSiteUrls.length > 0 && newUrls.length > 0) {
bingApi = await submitToBingApi(config.bingSiteUrls, newUrls.slice(0, SETTINGS.bingApiMaxPerRun));
if (newUrls.length > SETTINGS.bingApiMaxPerRun) {
bingApi.note = `capped at ${SETTINGS.bingApiMaxPerRun} of ${newUrls.length} new URLs (daily quota)`;
}
}
return {
host: config.host,
totalUrls: currentUrls.length,
newUrls,
removedUrls,
retriedCount: hostState.pending_urls.length,
submission,
bingApi
};
}
export async function submitToIndexNow(host, key, urls) {
const attempts = [];
const failedUrls = [];
let acceptedCount = 0;
const chunks = [];
for (let i = 0; i < urls.length; i += SETTINGS.maxBatch) {
chunks.push(urls.slice(i, i + SETTINGS.maxBatch));
}
for (let i = 0; i < chunks.length; i++) {
const label = chunks.length > 1 ? ` [batch ${i + 1}/${chunks.length}]` : "";
const ok = await submitChunk(host, key, chunks[i], attempts, label);
if (ok) acceptedCount += chunks[i].length;
else failedUrls.push(...chunks[i]);
}
return { ok: failedUrls.length === 0, acceptedCount, failedUrls, attempts };
}
async function submitChunk(host, key, chunk, attempts, label) {
const body = JSON.stringify({
host,
key,
keyLocation: `https://${host}/${key}.txt`,
urlList: chunk
});
for (const endpoint of ENDPOINTS) {
for (let attempt = 1; attempt <= SETTINGS.attemptsPerEndpoint; attempt++) {
let status;
try {
const res = await fetchWithTimeout(endpoint, {
method: "POST",
headers: { "Content-Type": "application/json; charset=utf-8" },
body
});
status = res.status;
} catch (err) {
status = `network error: ${err.message}`;
}
attempts.push(`${endpoint}${label} (attempt ${attempt}): ${status}`);
if (status === 200 || status === 202) return true;
const retryable = status === 429 || status === 500 || status === 503 || typeof status === "string";
if (!retryable) break;
if (attempt < SETTINGS.attemptsPerEndpoint) await sleep(SETTINGS.retryDelayMs);
}
}
return false;
}
async function submitToBingApi(siteUrls, urls) {
const attempts = [];
for (const siteUrl of siteUrls) {
let status, detail = "";
try {
const res = await fetchWithTimeout(`https://ssl.bing.com/webmaster/api.svc/json/SubmitUrlBatch?apikey=${env.BING_WMT_API_KEY}`, {
method: "POST",
headers: { "Content-Type": "application/json; charset=utf-8" },
body: JSON.stringify({ siteUrl, urlList: urls })
});
status = res.status;
if (!res.ok) detail = (await res.text()).slice(0, 200);
} catch (err) {
status = `network error: ${err.message}`;
}
attempts.push(`SubmitUrlBatch ${siteUrl}: ${status} ${detail}`.trim());
if (status === 200) return { ok: true, submitted: urls.length, attempts };
}
return { ok: false, submitted: 0, attempts };
}
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
async function sendSummaryNotification(reports) {
if (!env.RESEND_API_KEY || !env.TO_EMAIL) return;
const fromEmail = env.FROM_EMAIL || "onboarding@resend.dev";
const failedHosts = reports.filter(r => r.error || (r.submission && !r.submission.ok));
const totalNew = reports.reduce((n, r) => n + (r.newUrls?.length ?? 0), 0);
const totalRemoved = reports.reduce((n, r) => n + (r.removedUrls?.length ?? 0), 0);
const subject = failedHosts.length > 0
? `IndexNow: FAILED (${failedHosts.map(r => r.host).join(", ")}), queued for retry`
: `IndexNow: OK — ${totalNew} new, ${totalRemoved} removed`;
const lines = [];
lines.push(`IndexNow daily run (GitHub Actions) — ${reports.length} host(s)`);
lines.push(`Timestamp: ${new Date().toISOString()}`);
lines.push(``);
for (const r of reports) {
lines.push(`=== ${r.host} ===`);
if (r.error) { lines.push(` ERROR: ${r.error}`, ``); continue; }
if (!r.submission) { lines.push(` Nothing changed (${r.totalUrls} URLs in sitemap).`, ``); continue; }
const s = r.submission;
lines.push(` IndexNow: ${s.ok ? "OK" : `FAILED, ${s.failedUrls.length} URLs queued for retry`}`);
lines.push(` Accepted: ${s.acceptedCount} URLs`);
if (r.retriedCount > 0) lines.push(` Retried from previous failed runs: ${r.retriedCount} URLs`);
lines.push(` Attempts:`);
s.attempts.forEach(a => lines.push(` ${a}`));
if (r.bingApi) {
lines.push(` Bing Submission API: ${r.bingApi.ok ? `OK — ${r.bingApi.submitted} URLs` : "FAILED"}${r.bingApi.note ? ` (${r.bingApi.note})` : ""}`);
r.bingApi.attempts.forEach(a => lines.push(` ${a}`));
}
if (r.newUrls.length > 0) {
lines.push(` New URLs (${r.newUrls.length}):`);
r.newUrls.forEach(u => lines.push(` ${u}`));
}
if (r.removedUrls.length > 0) {
lines.push(` Removed URLs (${r.removedUrls.length}):`);
r.removedUrls.forEach(u => lines.push(` ${u}`));
}
lines.push(``);
}
const res = await fetchWithTimeout("https://api.resend.com/emails", {
method: "POST",
headers: {
"Authorization": `Bearer ${env.RESEND_API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
from: fromEmail,
to: env.TO_EMAIL,
subject,
text: lines.join("\n")
})
});
if (!res.ok) throw new Error(`Resend API ${res.status}: ${await res.text()}`);
}
main();
import { readFile, writeFile } from "node:fs/promises";
import { fileURLToPath } from "node:url";
const STATE_PATH = fileURLToPath(new URL("./indexnow-state.json", import.meta.url));
export const SETTINGS = {
retryDelayMs: 5000,
attemptsPerEndpoint: 2,
maxBatch: 10000,
bingApiMaxPerRun: 100
};
export const ENDPOINTS = [
"https://www.bing.com/indexnow",
"https://search.seznam.cz/indexnow",
"https://api.indexnow.org/indexnow"
];
export const HOSTS = [
{
host: "yoursite.com",
sitemap: "https://yoursite.com/sitemap.xml",
bingSiteUrls: ["http://yoursite.com", "https://yoursite.com"]
}
];
const env = process.env;
const DRY_RUN = env.DRY_RUN === "1";
async function main() {
if (!env.INDEXNOW_KEY) {
console.error("FATAL: INDEXNOW_KEY is not set");
process.exit(1);
}
let state = {};
try { state = JSON.parse(await readFile(STATE_PATH, "utf8")); }
catch { console.log("No state file yet — first run submits the full sitemap."); }
const reports = [];
for (const h of HOSTS) {
try {
reports.push(await runHost(h, state));
} catch (e) {
reports.push({ host: h.host, error: e.message });
}
}
if (!DRY_RUN) {
await writeFile(STATE_PATH, JSON.stringify(state, null, 1) + "\n");
}
const noteworthy = reports.some(r => r.error || r.submission);
if (noteworthy && !DRY_RUN) {
try { await sendSummaryNotification(reports); }
catch (err) { console.error("Email notification failed:", err.message); }
}
for (const r of reports) {
console.log(JSON.stringify(r, null, 2));
}
if (reports.every(r => r.error)) process.exit(1);
}
function fetchWithTimeout(url, opts = {}) {
return fetch(url, { ...opts, signal: AbortSignal.timeout(30_000) });
}
async function runHost(config, state) {
const sitemapRes = await fetchWithTimeout(config.sitemap);
if (!sitemapRes.ok) {
return { host: config.host, error: `Sitemap fetch failed: ${sitemapRes.status}` };
}
const xml = await sitemapRes.text();
const currentUrls = [...xml.matchAll(/<loc>(.*?)<\/loc>/g)]
.map(m => m[1].trim())
.filter(u => {
try { return new URL(u).hostname === config.host; }
catch { return false; }
});
if (currentUrls.length === 0) {
return { host: config.host, error: "No matching-host URLs found in sitemap" };
}
const hostState = state[config.host] ?? { last_urls: [], pending_urls: [] };
const previousSet = new Set(hostState.last_urls);
const currentSet = new Set(currentUrls);
const newUrls = currentUrls.filter(u => !previousSet.has(u));
const removedUrls = hostState.last_urls.filter(u => !currentSet.has(u));
const pending = [...new Set([...hostState.pending_urls, ...newUrls, ...removedUrls])];
state[config.host] = { last_urls: currentUrls, pending_urls: pending };
if (pending.length === 0) {
return { host: config.host, totalUrls: currentUrls.length, newUrls: [], removedUrls: [], retriedCount: 0, submission: null };
}
if (DRY_RUN) {
return { host: config.host, totalUrls: currentUrls.length, newUrls, removedUrls,
retriedCount: hostState.pending_urls.length, submission: { dryRun: true, wouldSubmit: pending.length } };
}
const submission = await submitToIndexNow(config.host, env.INDEXNOW_KEY, pending);
state[config.host].pending_urls = submission.failedUrls;
let bingApi = null;
if (env.BING_WMT_API_KEY && config.bingSiteUrls.length > 0 && newUrls.length > 0) {
bingApi = await submitToBingApi(config.bingSiteUrls, newUrls.slice(0, SETTINGS.bingApiMaxPerRun));
if (newUrls.length > SETTINGS.bingApiMaxPerRun) {
bingApi.note = `capped at ${SETTINGS.bingApiMaxPerRun} of ${newUrls.length} new URLs (daily quota)`;
}
}
return {
host: config.host,
totalUrls: currentUrls.length,
newUrls,
removedUrls,
retriedCount: hostState.pending_urls.length,
submission,
bingApi
};
}
export async function submitToIndexNow(host, key, urls) {
const attempts = [];
const failedUrls = [];
let acceptedCount = 0;
const chunks = [];
for (let i = 0; i < urls.length; i += SETTINGS.maxBatch) {
chunks.push(urls.slice(i, i + SETTINGS.maxBatch));
}
for (let i = 0; i < chunks.length; i++) {
const label = chunks.length > 1 ? ` [batch ${i + 1}/${chunks.length}]` : "";
const ok = await submitChunk(host, key, chunks[i], attempts, label);
if (ok) acceptedCount += chunks[i].length;
else failedUrls.push(...chunks[i]);
}
return { ok: failedUrls.length === 0, acceptedCount, failedUrls, attempts };
}
async function submitChunk(host, key, chunk, attempts, label) {
const body = JSON.stringify({
host,
key,
keyLocation: `https://${host}/${key}.txt`,
urlList: chunk
});
for (const endpoint of ENDPOINTS) {
for (let attempt = 1; attempt <= SETTINGS.attemptsPerEndpoint; attempt++) {
let status;
try {
const res = await fetchWithTimeout(endpoint, {
method: "POST",
headers: { "Content-Type": "application/json; charset=utf-8" },
body
});
status = res.status;
} catch (err) {
status = `network error: ${err.message}`;
}
attempts.push(`${endpoint}${label} (attempt ${attempt}): ${status}`);
if (status === 200 || status === 202) return true;
const retryable = status === 429 || status === 500 || status === 503 || typeof status === "string";
if (!retryable) break;
if (attempt < SETTINGS.attemptsPerEndpoint) await sleep(SETTINGS.retryDelayMs);
}
}
return false;
}
async function submitToBingApi(siteUrls, urls) {
const attempts = [];
for (const siteUrl of siteUrls) {
let status, detail = "";
try {
const res = await fetchWithTimeout(`https://ssl.bing.com/webmaster/api.svc/json/SubmitUrlBatch?apikey=${env.BING_WMT_API_KEY}`, {
method: "POST",
headers: { "Content-Type": "application/json; charset=utf-8" },
body: JSON.stringify({ siteUrl, urlList: urls })
});
status = res.status;
if (!res.ok) detail = (await res.text()).slice(0, 200);
} catch (err) {
status = `network error: ${err.message}`;
}
attempts.push(`SubmitUrlBatch ${siteUrl}: ${status} ${detail}`.trim());
if (status === 200) return { ok: true, submitted: urls.length, attempts };
}
return { ok: false, submitted: 0, attempts };
}
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
async function sendSummaryNotification(reports) {
if (!env.RESEND_API_KEY || !env.TO_EMAIL) return;
const fromEmail = env.FROM_EMAIL || "onboarding@resend.dev";
const failedHosts = reports.filter(r => r.error || (r.submission && !r.submission.ok));
const totalNew = reports.reduce((n, r) => n + (r.newUrls?.length ?? 0), 0);
const totalRemoved = reports.reduce((n, r) => n + (r.removedUrls?.length ?? 0), 0);
const subject = failedHosts.length > 0
? `IndexNow: FAILED (${failedHosts.map(r => r.host).join(", ")}), queued for retry`
: `IndexNow: OK — ${totalNew} new, ${totalRemoved} removed`;
const lines = [];
lines.push(`IndexNow daily run (GitHub Actions) — ${reports.length} host(s)`);
lines.push(`Timestamp: ${new Date().toISOString()}`);
lines.push(``);
for (const r of reports) {
lines.push(`=== ${r.host} ===`);
if (r.error) { lines.push(` ERROR: ${r.error}`, ``); continue; }
if (!r.submission) { lines.push(` Nothing changed (${r.totalUrls} URLs in sitemap).`, ``); continue; }
const s = r.submission;
lines.push(` IndexNow: ${s.ok ? "OK" : `FAILED, ${s.failedUrls.length} URLs queued for retry`}`);
lines.push(` Accepted: ${s.acceptedCount} URLs`);
if (r.retriedCount > 0) lines.push(` Retried from previous failed runs: ${r.retriedCount} URLs`);
lines.push(` Attempts:`);
s.attempts.forEach(a => lines.push(` ${a}`));
if (r.bingApi) {
lines.push(` Bing Submission API: ${r.bingApi.ok ? `OK — ${r.bingApi.submitted} URLs` : "FAILED"}${r.bingApi.note ? ` (${r.bingApi.note})` : ""}`);
r.bingApi.attempts.forEach(a => lines.push(` ${a}`));
}
if (r.newUrls.length > 0) {
lines.push(` New URLs (${r.newUrls.length}):`);
r.newUrls.forEach(u => lines.push(` ${u}`));
}
if (r.removedUrls.length > 0) {
lines.push(` Removed URLs (${r.removedUrls.length}):`);
r.removedUrls.forEach(u => lines.push(` ${u}`));
}
lines.push(``);
}
const res = await fetchWithTimeout("https://api.resend.com/emails", {
method: "POST",
headers: {
"Authorization": `Bearer ${env.RESEND_API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
from: fromEmail,
to: env.TO_EMAIL,
subject,
text: lines.join("\n")
})
});
if (!res.ok) throw new Error(`Resend API ${res.status}: ${await res.text()}`);
}
main();Replace yoursite.com in the HOSTS array with your domain.
Step 4: Add the secrets
In the repo: Settings → Secrets and variables → Actions → New repository secret, three times:
INDEXNOW_KEY: your IndexNow key, exactly matching the key file your site serves
BING_WMT_API_KEY: Bing Webmaster Tools → Settings → API access (optional; skip it and the script skips that channel)
RESEND_API_KEY: from resend.com (optional; skip it and no emails are sent)
Secrets never appear in workflow logs. GitHub masks them automatically.
Step 5: First run
Go to Actions → IndexNow daily submission → Run workflow.
With no state file, the first run submits your entire sitemap. That’s correct IndexNow behavior for a first submission, and it seeds the state. The workflow then commits scripts/indexnow-state.json back to the repo. Run it a second time and you should see:
{
"host": "yoursite.com",
"totalUrls": 196,
"newUrls": [],
"removedUrls": [],
"retriedCount": 0,
"submission": null
}{
"host": "yoursite.com",
"totalUrls": 196,
"newUrls": [],
"removedUrls": [],
"retriedCount": 0,
"submission": null
}{
"host": "yoursite.com",
"totalUrls": 196,
"newUrls": [],
"removedUrls": [],
"retriedCount": 0,
"submission": null
}"submission": null means nothing changed and nothing was sent. The diff is working.
To verify submissions are landing, check Bing Webmaster Tools → IndexNow. Submitted URLs appear there within minutes, labeled "Self".
How it works after setup
Every day the workflow fetches your sitemap, diffs it against the committed state, and submits only what changed. On a good day, one request to Bing’s IndexNow endpoint covers every participating engine, and new URLs also go through the authenticated Bing API. On a bad day (a 429, a timeout, an outage) the failed URLs stay in the pending queue and go out with the next run. The email subject tells you which kind of day it was: IndexNow: OK or IndexNow: FAILED, queued for retry. No email means the sitemap didn’t change.
One habit to build: the workflow commits state changes to the repo, so run git pull --rebase before pushing any local edits to it.
Which path is yours?
Setting up IndexNow for the first time. Not a migration question: start with this manual, not the Worker one. You get all three defenses from day one, the setup takes about twenty minutes, and there is nothing to move later.
Running a Worker (or plugin) that returns 200s. No urgency. The shared-IP problem is a lottery, and you might keep winning. But port Defense 1 to your setup now: the silent-loss bug is a design flaw regardless of platform, and the queue turns a future 429 from data loss into a one-day delay. Our Worker manual has the fixed snapshot code.
Getting 429s, from anywhere. No amount of in-place fixes will reliably help, because the problem is the IP pool, not your code. Move the ping to this setup. Keep the key file wherever it’s already served. If you’re on a Framer Pro/Scale/Enterprise plan, the dashboard’s Domains → Files feature means you may not need your old infrastructure in this pipeline at all. Check your plan and the current menu location first.
What we saw after switching
The first submission from a GitHub runner returned 200 on the first attempt, with the same payload and the same key that had been getting 429 from the Worker for ten days. All 196 URLs, including the lost glossary batch, appeared in Bing Webmaster Tools’ IndexNow report within the hour, labeled "Self". Total infrastructure cost of the new setup: zero. Total code we deleted: one Worker, one KV namespace, one cron trigger, one Worker route.
Read next
Email Alerts for Your IndexNow Pipeline: Cloudflare Workers and GitHub Actions: the monitoring chapter. The alerts described there are what caught our 429 incident, and Part 3 wires them into this exact setup.
How to Push New URLs to Bing and Yandex the Moment You Publish: the original Worker setup this manual supersedes for the ping itself; still the best explainer of how IndexNow works.
Inline Images in Framer CMS Render at 600px. Here’s the Fix.: companion fix in the same Framer publishing pipeline.
