Beyond Google: Bing Webmaster Tools, IndexNow and AI-era…

Video lecture

Beyond Google: Bing Webmaster Tools, IndexNow and AI-era crawling

12 chapters · about 8 min · full transcript

Coming soon

Chapter 1 of 12

Beyond Google

Bing Webmaster Tools, set up properly
IndexNow: push, don't wait
AI-era crawling, technically

The narrated lecture is in production

Every chapter is scripted and ready. Browse the chapters and read the full transcript now — the video will appear here when it’s published.

Chapters

Transcript of the narration, chapter by chapter.

0:00 Beyond Google

Most technical SEOs live inside Google Search Console. That's understandable. But in twenty twenty-six, ignoring Bing is a mistake, because Bing's index powers Microsoft Copilot's web grounding, sits behind other search products, and now reports AI citations directly. In this lecture you'll set up Bing Webmaster Tools properly, implement IndexNow to push changes instantly to participating engines, and handle the technical side of AI-era crawling.

0:28 Why Bing matters

Why Bing? Its share of classic search varies by market, but its reach goes further than bing.com. It grounds Copilot's answers, and it has fed other assistants over the years. And since February twenty twenty-six, Bing Webmaster Tools has an AI Performance report showing how often your pages are cited in Copilot and Bing AI answers, which pages, and the grounding queries used to find them. That's data Google doesn't give you. So if you care about AI visibility, Bing is suddenly one of your best sources of evidence.

1:07 Setup in 15 minutes

Setup takes about fifteen minutes. The fastest route is importing your sites from Search Console. Then submit your sitemap index and check the Sitemaps report for errors. Use Site Explorer and URL Inspection to see index status and crawl details. Review crawl settings: Bing lets you shape crawl rate by time of day, and unlike Google, Bingbot supports crawl-delay in robots.txt. Then review the SEO reports, backlinks and AI Performance monthly, and add team members with sensible roles.

1:41 IndexNow

Now IndexNow. Think of the difference between waiting for the postman to check whether you have letters, and ringing the sorting office to say, I've just posted three. IndexNow is an open protocol that lets you notify participating search engines immediately when URLs are added, updated or deleted. Participants include Bing, Yandex, Seznam, Naver and Yep, and a submission to one is shared with the others. One important caveat: Google doesn't use IndexNow. It relies on crawling and sitemaps.

2:15 How IndexNow works

Here's how it works. Generate a key, eight to one hundred and twenty-eight characters of letters, numbers and dashes. Host it as a text file at your site root, containing only the key, so engines can confirm you own the site. Then, when URLs change, POST a JSON payload with your host, key, key location and a URL list, up to ten thousand URLs per request, to an IndexNow endpoint. The lesson text has a short Python example that reads the key from an environment variable and prints the response. Two hundred or two-oh-two means accepted.

2:57 Operational rules

Operational rules matter more than the code. Submit only URLs that actually changed: new pages, meaningful updates, and deletions or redirects. Don't resubmit your whole site every day. Trigger it from your CMS or deploy pipeline, and check first whether your CMS or CDN already has built-in IndexNow support. Log every submission and its response, and alert on repeated four-hundred-level errors, which usually mean the key file, host or URL list is wrong. And keep your sitemaps accurate anyway. IndexNow complements sitemaps; it doesn't replace them.

3:34 AI-era crawling, technically

Next, AI-era crawling from a technical point of view. AI crawlers add load and new failure modes. Some are bursty, so monitor request rates by user agent and use CDN rate limits for abusive patterns rather than blanket blocks. Verify identities with published IP ranges or reverse DNS where providers document them. Keep server-rendered HTML lean, because many AI crawlers don't run JavaScript. And align your CDN bot settings with your written policy, because block AI bots toggles can silently block search-oriented AI crawlers you actually want.

4:12 Example 1: Manchester dental practice

Worked example one, simple. A Manchester dental practice's site has never been verified in Bing. Importing from Search Console takes two minutes. Bing's Sitemaps report immediately flags that the sitemap lists old HTTP URLs from before the site moved to HTTPS. Fixing the sitemap helps both Bing and Google. A good reminder that a second tool often spots problems the first one didn't highlight.

4:40 Example 2: Lahore news portal (illustrative)

Worked example two, with illustrative details. A Lahore Urdu news portal publishes around one hundred and fifty stories a day. Google finds them quickly through the news sitemap, but Bing is slower and Copilot rarely cites the portal. The team imports the property into Bing Webmaster Tools, fixes sitemap errors Bing reports, including relative URLs in one sitemap, and adds IndexNow calls to the CMS publish and update hooks. Logs then show Bingbot fetching new stories much sooner, and the AI Performance report starts listing citations for explainer articles.

5:19 Watch me do it: Bing + IndexNow in 20 minutes

Watch me do it. I'll set up Bing and IndexNow for a small publisher in twenty minutes. Step one: in Bing Webmaster Tools, I choose import from Google Search Console, sign in, and select the property. It's verified in about a minute. Step two: Sitemaps. Bing shows the sitemap index, and one child sitemap has a warning: relative URLs. I note it for the developer. Step three: Site Explorer. I check that the main sections are indexed, and open URL Inspection for a recent article. Indexed, crawled two days ago. Step four: AI Performance. There are a few citations for one explainer article, and I copy the grounding queries. Step five: IndexNow. I check the CMS plugin list first, and it already supports IndexNow, so I don't build anything. I switch it on. The plugin generates the key and hosts the key file at the root. I open that URL in the browser to confirm it shows the key. Step six: I publish a test update to an article and check the plugin's submission log. The response is two-oh-two, accepted. Step seven: I set a reminder to check the log weekly for four-hundred errors, and to compare publish-to-first-Bingbot-hit times in the logs next month.

6:48 Measures and mistakes

How do you measure success? For Bing: indexed pages versus submitted, time from publishing to the first Bingbot hit in your logs, and the AI Performance citations trend. For IndexNow: the share of submissions accepted, and errors per week. For AI crawling: request rates and status codes by bot, with no unintended blocks. Common mistakes: ignoring Bing because nobody uses it, pinging IndexNow with every URL every day, expecting IndexNow to speed up Google, and letting CDN defaults decide which AI crawlers are allowed.

7:25 Recap and try this now

Recap. Bing's index powers Copilot and more, and its Webmaster Tools now shows AI citations and grounding queries. IndexNow pushes changed URLs to Bing and other participants instantly, but not to Google. AI crawlers need monitoring, verification and deliberate CDN settings. Try this now. Import your site into Bing Webmaster Tools, check the Sitemaps and AI Performance reports, and find out whether your CMS already supports IndexNow. If it does, switch it on and watch the submission log for a week.

Why Bing deserves real attention in 2026

Bing's share of classic search varies by market, but its index matters far beyond bing.com: it powers Microsoft Copilot's web grounding, it sits behind other search products and partners, and it has been used by other assistants. Since February 2026, Bing Webmaster Tools also reports AI Performance — how often your pages are cited in Copilot and Bing AI answers, and the grounding queries that retrieved them. A technical SEO who only looks at Google is missing a growing part of the picture.

Setting up Bing Webmaster Tools properly

Import from Search Console (fastest) or verify via DNS/meta tag/XML file.
Submit your sitemap index; check Sitemaps for errors.
Review Site Explorer and URL Inspection for index status and crawl details.
Check Crawl settings — Bing lets you shape crawl rate by time of day; Bingbot also supports crawl-delay in robots.txt (Google does not).
Review SEO reports, Backlinks and the AI Performance report monthly.
Add team members with appropriate roles.

IndexNow: push, don't wait

IndexNow is an open protocol that lets you notify participating search engines immediately when URLs are added, updated or deleted. Participants include Bing, Yandex, Seznam, Naver and Yep; a submission to one is shared with the others. Google does not use IndexNow (it relies on crawling, sitemaps and, for eligible job and livestream pages only, its Indexing API).

How it works:

Generate a key (8-128 characters: letters, numbers, dashes).
Host it as a text file at your site root, e.g. https://www.example.com/3f9a1c2b7d.txt, containing only the key.
When URLs change, POST them (up to 10,000 per request) to an IndexNow endpoint.

# pip install requests
import os, requests

KEY = os.environ["INDEXNOW_KEY"]
payload = {
    "host": "www.example.com",
    "key": KEY,
    "keyLocation": f"https://www.example.com/{KEY}.txt",
    "urlList": [
        "https://www.example.com/products/brass-lantern/",
        "https://www.example.com/blog/eid-gift-guide/",
    ],
}
r = requests.post("https://api.indexnow.org/indexnow", json=payload, timeout=15)
print(r.status_code, r.text[:200])   # 200/202 = accepted; 4xx = check key file, host or URL list

Operational rules:

Submit only URLs that actually changed (new, meaningfully updated, deleted/redirected). Don't resubmit the whole site daily.
Trigger it from your CMS or deploy pipeline (many CMS plugins and CDNs offer built-in IndexNow support — check before building your own).
Log every submission and response code; alert on repeated 4xx.
Keep sitemaps accurate anyway: IndexNow complements them, it doesn't replace them.

AI-era crawling on the technical side

AI crawlers add load and new failure modes. Technically:

Treat them in capacity planning. Some AI crawlers are bursty. Monitor request rates by user agent and apply rate limits at the CDN for abusive patterns rather than blanket blocks.
Verify identities with published IP ranges or reverse DNS where providers document them.
Keep server-rendered HTML lean, because many AI crawlers don't execute JavaScript.
Align CDN bot settings with policy — "block AI bots" toggles can silently block search-oriented AI crawlers too.

Bing vs Google: technical differences worth knowing

Topic	Google	Bing
`crawl-delay` in robots.txt	Ignored	Supported; crawl rate also adjustable in Webmaster Tools
Push indexing	No IndexNow; Indexing API only for job posting and livestream pages	IndexNow; URL submission in Webmaster Tools
AI reporting	Generative AI report: impressions only	AI Performance: citations and grounding queries
Verification of crawler	Reverse DNS to googlebot.com / google.com	Reverse DNS to search.msn.com
JavaScript rendering	Renders with evergreen Chromium	Renders JavaScript too, but keep critical content in HTML for all crawlers

Worked example: a Pakistani news portal speeds up Bing discovery

A Lahore Urdu news portal (illustrative) publishes 150 stories a day. Google discovers them quickly via its news sitemap; Bing is slower and Copilot rarely cites the portal. The team imports the property into Bing Webmaster Tools, fixes sitemap errors Bing reports (relative URLs in one sitemap), and adds IndexNow calls to the CMS publish and update hooks. Logs then show Bingbot fetching new stories much sooner after publication, and the AI Performance report starts listing citations for explainer articles.

Measuring success

Bing: indexed pages vs submitted (Site Explorer / Sitemaps), time from publish to first Bingbot hit (logs), AI Performance citations trend.
IndexNow: share of submissions accepted, errors per week.
AI crawling: request rate and status codes by bot; no unintended blocks.

Common mistakes

Ignoring Bing because "nobody uses it" — its index feeds Copilot and more.
Pinging IndexNow with every URL every day.
Expecting IndexNow to speed up Google.
Letting CDN defaults decide which AI crawlers are allowed.

Bing's index powers Copilot grounding and other products; Bing Webmaster Tools now reports AI citations and grounding queries.
IndexNow pushes changed URLs to Bing, Yandex, Seznam, Naver and Yep (shared between them); Google does not use it.
Implement IndexNow with a root key file and POSTs of genuinely changed URLs from your CMS or deploy pipeline; log responses.
Bingbot supports crawl-delay and crawl-rate settings; Googlebot ignores crawl-delay.
Monitor, verify and rate-limit AI crawlers deliberately; align CDN settings with policy.

Import your site into Bing Webmaster Tools, review Sitemaps and AI Performance, then enable IndexNow (via your CMS or the script) and check a week of submission logs.

Enrol for free to save your progress

Reading is always free. Enrol to keep your place, take the final assessment and earn a verifiable certificate.

Beyond Google: Bing Webmaster Tools, IndexNow and AI-era crawling