Stop refreshing manual Search Console reports. Use Python to submit, inspect, and monitor URLs at scale. Covers real API limits, error handling, and a worked example with concrete numbers.
Manual indexing workflows break at scale. You paste a URL, wait for the inspector, click request indexing, repeat. Fine for ten pages. Impossible for ten thousand.
Python gives you direct control over the Google Search Indexing API, allowing programmatic submission, inspection, and status monitoring. The core bottleneck is not the API itself — it is your error handling, quota management, and filtering logic. A common situation we see: a team submits 200 URLs, gets 43 errors, and has no idea which failed or why. That is not automation. That is noise.
This guide focuses on the operational details: setting up OAuth, batching submissions, handling 429 and 500 errors, and building a recovery loop. We also include a worked example with exact numbers.
| Method | What It Does | Daily Quota Limit | Common Failure Mode |
|---|---|---|---|
| urlNotifications.publish Submit URL for indexing | Sends a URL to Google's crawl queue for priority indexing | 200 per property per day Shared across all users | Quota exhaustion — no error returned until second request after limit hit |
| urlNotifications.inspect Check URL status | Returns current index state, last crawl, and any sitemap references | 100 per property per day Not same limit as publish | Stale data — response may be cached up to 2 hours after a publish call |
| Batch publish via list Submit multiple URLs in one request | Sends up to 200 URLs per batch call to the same property | 1 batch = 1 call against daily publish limit | Partial failure — the API returns a single error for the batch, not per-URL details |
| sitemaps.submit Submit a sitemap file | Tells Google to fetch and process a sitemap.xml | 10 sitemaps per day per property | Slow propagation — submitted sitemap can take 24+ hours to reflect in Search Console |
In practice, when you run a batch of 200 URLs, expect 10-20% to fail for reasons unrelated to your code. Blocked URLs (robots.txt disallow, noindex tag, or 403 server response) will return a 400 error with a message like 'URL not accessible to Googlebot'. Your script must differentiate these from transient network errors — retrying a blocked URL is wasteful.
Duplicate lists also cause silent failures. Submitting the same URL twice within a short window returns a 200 OK but Google ignores the second request. Track submission timestamps in a local dictionary or Redis set. Empty results happen when the inspect method returns 'INSPECTION_RESULT_UNSPECIFIED' — this usually means Google has never crawled the URL. Do not assume the URL is indexed. Log it and re-submit after 24 hours.
For those working with PBN or guest post networks, safe indexing requires additional caution. The 2026 sandbox escape protocol describes a staged approach: pre-warm domains with non-commercial content, then submit commercial URLs slowly over a week. This reduces the risk of algorithmic penalties from rapid indexing spikes.
Scenario: You have 500 URLs from a site migration. Each URL must be submitted to the Indexing API. Your daily limit is 200 per property, so you need 3 days. Here is the concrete setup:
Settings: Batch size = 50 URLs per call. Retry = 3 attempts with backoff (2s, 4s, 8s). Filter = skip URLs with status code != 200 before submitting. Track = store submission timestamp in a CSV file.
Day 1: Submitted 200 URLs. Results: 187 success, 13 failed (8 blocked by robots.txt, 5 returned 500 errors). Retried the 5 transient errors: 3 succeeded, 2 failed again — logged as 'needs manual check'.
Day 2: Submitted next 200 URLs. Results: 192 success, 8 failed (all 429 rate limit — waited 15 minutes, retried, 6 succeeded). Remaining 2 failed — logged.
Day 3: Submitted final 100 URLs. Results: 98 success, 2 failed (both inspect calls showed 'crawl anomaly' — suggested rechecking sitemap).
Total indexed: 477 out of 500. Error recovery rate: 95.4%. Without retry and filtering, you would have lost 23 URLs permanently.
Filter by HTTP 200 status, remove duplicates, check robots.txt
Group 50 URLs per API call, send to urlNotifications.publish
Check for error codes: 200 = success, 429 = rate limit, 400 = blocked
Exponential backoff: 2s, 4s, 8s, max 3 retries on 429 or 500
Save all permanent errors (blocked, not found) to a CSV for manual review
After 24h, run urlNotifications.inspect on a sample to confirm indexing
| Error Code / Message | Root Cause | Immediate Action | Long-Term Prevention |
|---|---|---|---|
| 429 RESOURCE_EXHAUSTED Rate limit exceeded | You hit the 200 daily publish limit or sent too many requests per second | Stop all requests for 24 hours. Do not retry same day. | Implement a quota tracker that counts calls and alerts at 180 |
| 400 URL_NOT_ACCESSIBLE Googlebot cannot fetch URL | URL blocked by robots.txt, returns 403, or has noindex meta tag | Fix server access or remove noindex tag. Re-submit after fix. | Pre-check URLs with a HEAD request before submitting |
| 500 Internal Server Error | Temporary Google backend issue (rare) | Retry after 60 seconds with backoff. Usually resolves. | Add to retry queue with max 3 attempts, then escalate |
| 403 PERMISSION_DENIED OAuth scope error | Service account not added as owner in Search Console | Add service account email as owner in Search Console settings | Automate owner addition via Google Groups for multi-property setups |
| INSPECTION_RESULT_UNSPECIFIED No crawl data | URL has never been crawled by Google | Submit via publish and wait 24-48 hours before inspecting | Track submission timestamp; do not re-inspect before 24 hours |
The Indexing API is not the only tool. Sitemap submission works well for large, stable sites where you want Google to discover new pages at its own pace. However, for time-sensitive content (e.g., breaking news, product launches, or guest posts), the Indexing API is faster.
A hybrid approach often works best: submit critical URLs via the API, and include all URLs in your sitemap for ongoing coverage. If you need a step-by-step walkthrough of sitemap submission, check this guide to indexing a sitemap in Google quickly — it covers XML formatting, sitemap index files, and common pitfalls like missing lastmod tags.
OAuth credentials stored as environment variables, not hardcoded
Service account email added as owner in Google Search Console for each property
Batch size set to 50 or less to avoid timeout and partial failure issues
Retry logic with exponential backoff implemented for HTTP 429 and 500
Duplicate URL detection using a set or Redis before submission
Logging to file with timestamps, URL, status, and error message
Daily quota counter that stops submission at 190 calls to leave room for retries
Inspect function scheduled to run 24 hours after publish to verify indexing
The free tier allows 200 URL publish calls per Search Console property per day. That limit is shared across all users of the property. You can increase it by requesting a higher quota from Google Cloud, but approval is not guaranteed. For large-scale projects, consider distributing across multiple properties or combining with sitemap submission.
Create one service account per client and add it as an owner in their Search Console. Use a parent script that iterates over client IDs, each with its own OAuth token. Store tokens in a secure database. Track daily usage per property; if one client hits the 200 limit, do not fall over to another client's quota. This isolates failures.
Yes, but with extreme caution. The Indexing API does not bypass algorithmic filters. Submitting many low-authority URLs too quickly can trigger sandboxing. Follow a staged protocol: warm the domain with non-commercial content for 2 weeks, then submit backlinks at a rate of 10-15 per day. Use the inspect method to confirm each URL is indexed before submitting the next batch.
First, enable the API in Google Cloud Console. Create a service account, download the JSON key, and share it with your Search Console property as an owner. Install google-api-python-client. Authenticate using service_account.Credentials.from_service_account_file. Then call urlNotifications.publish with the URL and type URL_UPDATED. Wrap in try/except for error handling.
429 RESOURCE_EXHAUSTED means you hit the quota — stop and wait 24 hours. 400 URL_NOT_ACCESSIBLE means Googlebot cannot fetch the URL — check robots.txt and noindex tags. 403 PERMISSION_DENIED means the service account lacks owner permissions. 500 errors are temporary — retry with backoff. INSPECTION_RESULT_UNSPECIFIED means no crawl data yet — wait 24 hours.
Yes, a minimal template involves: load URLs from a CSV, filter by HTTP 200 status, batch them into groups of 50, authenticate, call urlNotifications.publish for each batch, catch exceptions, and log results. Use time.sleep(2) between batches to avoid rate limits. Store successful submissions in a separate file so you can retry only failures.
Call the urlNotifications.inspect method with the URL. The response includes a inspectionResult field with indexStatusEnum: INDEXING_STATE_NEEDS_CRAWL, INDEXING_STATE_CRAWLING, or INDEXING_STATE_INDEXED. Wait at least 24 hours after publish before inspecting, otherwise you may get stale results. If status is unspecified, the URL has never been crawled.
Google does not publicly list a paid tier for the Indexing API. The 200 URL per day limit is fixed. For higher volume, you must apply for quota increase through Google Cloud — approval depends on usage justification. Some third-party services offer reseller access but violate Google's ToS. Stick to the free quota or use sitemaps for additional coverage.
Use cron (Linux) or Task Scheduler (Windows) to run the script once every 24 hours. Add a database or file to track last submission timestamps per URL so you do not re-submit the same URL within 7 days. Log all errors to a file. Set up email or Slack alerts if the failure rate exceeds 10%.
Yes, the Indexing API works with any URL that Googlebot can access, including PDFs and images. The same 200 per day limit applies. However, for images, ensure they are not blocked by robots.txt and have proper alt text. PDFs should have text content, not scanned images, to be indexed effectively.
Quick calculator. Put in the expected monthly value of a page or link batch and the natural waiting time.