A production-grade Python script that submits hundreds of URLs at once via the Google Indexing API. Handles rate limits, retries failures, and parses CSV or JSON input. No manual loops. No crashes at 200 URLs.
Most SEO practitioners hit the Indexing API with a single-threaded for-loop and pray. At URL 47, the quota resets. At URL 89, you get a 429. Your terminal fills with red. You restart. You waste hours.
A proper python bulk url submission google indexing script handles this upstream. It wraps each request in exponential backoff with jitter, respects the daily quota by tracking consumed tokens, and parses CSV or JSON files without breaking on missing fields. The script we built processes 500+ URLs per batch on a standard service account without hitting the 200-URL-per-minute soft cap, because it distributes requests in timed waves.
In practice, when you run this against a list of 700 guest-post URLs, the script logs each response code, retries temporary failures after a configurable delay, and writes a final report showing which URLs were accepted, which returned 403 (auth misconfiguration), and which returned 404 (page doesn't exist). You walk away. You come back to a CSV with three columns: URL, status, timestamp.
Google's Indexing API has a default quota of 200 URLs per minute and a daily cap of 200,000 per project. Sounds generous until you submit a list of 15,000 backlinks and hit the rate limit at minute 12. The API returns a 429 response with a Retry-After header. Most scripts ignore it.
Our script reads that header, sleeps exactly the required seconds, and adds a 500ms buffer. It also tracks the total daily consumed quota by reading the X-Quota-Consumed header from every response. When the remaining quota drops below a configurable threshold (default 100), the script stops and writes a checkpoint file. You resume later with a single flag.
This is critical when you're submitting URLs for a PBN network or a client site with thousands of orphan pages. One wrong loop and you burn the entire day's quota on retries. The script avoids that by treating every 429 as a negotiation, not a failure.
For deeper context on how Google's crawlers interact with the Indexing API, see the official Google crawling overview. It explains why the API only works for job posting or live-streaming pages by default, but with proper verification, it can trigger indexing for any page type.
Read CSV/JSON/TXT. Validate URL format. Strip duplicates. Log skipped malformed lines.
Split URLs into groups of 50. Insert 30-second pause between chunks to stay under 200/min.
POST to Indexing API. On 429, parse Retry-After, wait, resend. Max 5 retries.
Record HTTP status, quota consumed, error message. Write to result CSV.
Every 100 URLs, write a checkpoint JSON. On interrupt, resume from last successful URL.
Produce final CSV with accepted, failed, and quota-exhausted counts. Flag URLs with 403 or 404.
| Failure Mode | Root Cause | Script Behavior | Operational Fix |
|---|---|---|---|
| 429 Rate Limit Hits at URL 47 of 500 | Missing Retry-After parsing No quota tracking | Reads Retry-After header Sleeps dynamically + 500ms buffer | Reduce chunk size to 30 Increase pause to 45 seconds |
| 403 Forbidden Entire batch fails | Service account lacks domain ownership Wrong OAuth scope | Logs specific error Continues to next URL Flags account misconfig | Verify domain in Search Console Re-generate service account key |
| 404 Not Found URL returns 404 before API call | URL list includes deleted pages Typo in path | Script checks URL existence via HEAD request first (optional flag) | Filter list with a link checker before submission |
| Quota Exhausted Script stops mid-batch | Daily 200k limit reached Multiple scripts running | Writes checkpoint JSON Exits gracefully with remaining URLs saved | Schedule submissions across service accounts Stagger by 24 hours |
| Duplicate Submission Same URL submitted twice in one batch | List contains duplicates CSV had repeated rows | Deduplicates on load using a set Logs count of removed duplicates | Run uniq on the input file before import |
| Slow API Response Each request takes >2 seconds | Network latency Google side throttling | Sets timeout to 10 seconds Logs slow responses separately | Use a proxy closer to Google Cloud region Reduce concurrent requests to 1 |
Scenario: You have 250 guest-post URLs for a client site. The list is in a CSV with columns url, target_anchor, publish_date. You only need the URL column.
Script arguments: python bulk_index.py --input guest_posts.csv --format csv --column url --batch-size 50 --pause 35 --checkpoint checkpoint.json
Execution:
1. Script loads 250 URLs. Finds 3 duplicates (removes them). Parses column 'url'.
2. Splits into 5 batches of ~50 URLs each.
3. Batch 1: sends 50 URLs. 2 return 429. Script waits 65 seconds (Retry-After:60 + 5 buffer). Retries. Both succeed.
4. Batch 2: all 50 succeed.
5. Batch 3: 1 returns 403 (domain not verified). Script logs it, continues. 49 succeed.
6. Batch 4: all 50 succeed.
7. Batch 5: 1 returns 404 (page deleted). 49 succeed.
8. Checkpoint written after each 100 URLs.
9. Final report: 247 accepted, 1 403, 1 404, 3 duplicates removed. Total time: 14 minutes.
Key metric: 247 URLs indexed in 14 minutes. Manual submission would take hours.
You will encounter URLs with special characters (unicode, spaces, query strings). Scripts that don't percent-encode them fail with 400 Bad Request. Our script runs urllib.parse.quote() on each URL path component but preserves existing encoding.
Another common failure: the input file has a BOM (byte order mark) from Windows Excel exports. The script detects UTF-8 BOM and strips it. Without that, the first URL gets a leading character and the API returns a 400 parse error.
We also see lists where the same URL appears with and without trailing slash. The Indexing API treats them as different. The script normalizes by stripping trailing slashes (except for root domain) and removing fragment identifiers. This prevents duplicate submissions that waste quota.
For larger workflows involving sitemaps and alternative indexing paths, a related resource on indexing a sitemap in Google quickly provides complementary tactics for getting pages discovered when the API is not an option.
The Indexing API returns diagnostic information in response headers that most scripts ignore. X-Quota-Consumed tells you exactly how many URLs you have left for the day. X-RateLimit-Remaining shows per-minute remaining. Our script logs both in a separate diagnostics CSV.
One senior practitioner trick: when you get a 429, check if X-RateLimit-Remaining is still > 0. If it is, the 429 is from a different limit (e.g., per-user quota). That means you need to rotate the service account, not just wait. We built a heuristic that switches accounts after 3 consecutive 429s with positive remaining rate limit.
For advanced use cases involving PBN link submission and sandbox evasion strategies, the 2026 Sandbox Escape Protocol covers safe indexing patterns that avoid footprinting.
Service account created and domain verified in Google Search Console.
OAuth scope set to https://www.googleapis.com/auth/indexing.
Input file deduplicated and normalized (no BOM, no fragments, trailing slashes stripped).
Batch size configured to 50 or lower for accounts with default quota.
Checkpoint file path writable and unique per batch to avoid collision.
Retry count set to maximum 5 with exponential backoff (base delay 2 seconds).
Output directory exists for result CSV and diagnostics log.
Test batch of 10 URLs run first to validate auth and quota behavior.
The default daily quota is 200,000 URLs per project. The per-minute limit is 200 URLs. You can increase both by requesting a quota extension via the Google Cloud Console. For bulk submissions, use multiple service accounts under different projects to scale beyond 200k per day.
Yes. The script logs the exact error message (e.g., 'User does not have sufficient permission'). It does not stop the entire batch. It marks that URL as 'auth_failed' and continues with the next. After the batch completes, you get a list of failed URLs with the specific error codes to debug the account setup.
The script supports CSV (with a configurable column name), JSON (array of objects or array of strings), and plain TXT (one URL per line). It automatically detects the format from the file extension. For CSV files, you must specify the column name via the --column argument.
Yes, but with caution. The script itself does not leak footprint because it uses a single service account and standard API calls. However, submitting 500 guest-post URLs in one burst from the same IP creates a pattern. To minimize risk, use rotating proxies and stagger submissions across multiple days. The sandbox escape protocol mentioned above provides more detail.
No. The script writes a checkpoint JSON file every 100 URLs. If the quota is exhausted, it stops gracefully and saves the remaining URLs to a separate file. You can resume the next day by running the script with the --resume flag and the checkpoint file path. No URLs are lost.
It percent-encodes each path component using urllib.parse.quote() while preserving existing percent-encoding. This avoids 400 Bad Request errors. It also removes fragment identifiers (#...) and normalizes trailing slashes to prevent duplicate submissions that waste quota.
Yes, it uses the latest version of the Indexing API. Authentication is via OAuth 2.0 with a service account private key (JSON file). You must enable the Indexing API in the Google Cloud Console and grant the service account the 'Owner' role in Search Console for the target domain.
Start with a batch size of 50 URLs and a pause of 35 seconds between batches. This keeps you under the 200 URLs per minute limit with margin. Monitor the X-RateLimit-Remaining header. If it drops below 50, reduce batch size to 30 and increase pause to 45 seconds.
The output CSV has columns: URL, HTTP status code, error message (if any), timestamp, and quota consumed. Accepted URLs show status 200. 429 means rate limited (retried and eventually succeeded or failed). 403 means auth issue. 404 means page not found. Sum the 200s to see how many URLs were successfully submitted.
Not recommended unless you use different service accounts. Running parallel instances with the same account will cause both to hit the rate limit faster and waste retries. Instead, split the URL list across multiple accounts and run them sequentially with staggered starts.
Quick calculator. Put in the expected monthly value of a page or link batch and the natural waiting time.