Stop hitting 403 errors and quota failures. This guide walks you through service account creation, precise scope scoping, and token refresh logic so your Python environment stays authenticated in production.
The bottleneck is never the code. It is the OAuth configuration. Developers copy-paste old service account JSONs, forget to enable the Indexing API in the Google Cloud Console, or assign the wrong IAM role. The result: a silent 403 that wastes hours of debugging.
A common situation we see: an agency sets up a service account, grants it Owner (overkill), then wonders why notifications stop after a week. The fix is a dedicated service account with the Indexing API Service Agent role, a single scope (https://www.googleapis.com/auth/indexing), and a 3600-second token refresh loop.
In practice, when you authenticate with the Indexing API, you need exactly three things: a service account key file, the correct scope, and a refresh mechanism. Nothing else. Skip the wizard and do it manually.
| Component | Recommended Setting | Production Behavior | Failure Mode / Risk |
|---|---|---|---|
| Service Account Role IAM assignment in GCP | Indexing API Service Agent (roles/indexing.serviceAgent) | Grants read/write access to Indexing API only | Owner or Editor role exposes full project; policy violation risk |
| OAuth Scope URL used in credentials | https://www.googleapis.com/auth/indexing | Single scope, no extra permissions | Using cloud-platform scope triggers consent screen issues and over-permissioning |
| Token Lifetime Default expiry | 3600 seconds (1 hour) | Token expires exactly 60 min after issue; must refresh | Batch jobs that run >60 min fail mid-way; silent data loss |
| Refresh Strategy How to handle expiry | Check expiry at 50 min; re-authenticate if < 600 sec remaining | Continuous auth for long-running scrapers | Relying on implicit refresh leads to 401 errors and dropped URL submissions |
| Key File Storage Where to keep JSON key | Environment variable GOOGLE_APPLICATION_CREDENTIALS; never in repo | Flexible per environment (dev/staging/prod) | Committing key to Git exposes project; immediate revocation needed |
Enable the Indexing API in the Google Cloud Console under 'APIs & Services > Library'; it is not enabled by default.
Create a dedicated service account with the role 'Indexing API Service Agent'; do not reuse an account with Editor or Owner roles.
Verify the service account email is added as an owner of the site in Google Search Console (Property level, not user level).
Download the JSON key file and set the environment variable <code>GOOGLE_APPLICATION_CREDENTIALS</code> — never hardcode the path.
Test authentication with a single URL submission before running bulk operations.
Toggle on Indexing API in GCP Library; takes 2 min, often missed
Assign role 'Indexing API Service Agent'; no Owner
Add service account email as site owner in GSC
JSON key file stored as env var; never in source code
Use service_account.Credentials with single scope
Submit; if token < 600 sec left, re-authenticate
Assume you have a service account key file at /secrets/indexing-key.json. Your batch of 5 URLs must be submitted every 30 minutes. Token expires in 3600 seconds. Here is the exact code logic:
from google.oauth2 import service_account
from googleapiclient.discovery import build
SCOPES = ['https://www.googleapis.com/auth/indexing']
creds = service_account.Credentials.from_service_account_file('/secrets/indexing-key.json', scopes=SCOPES)
service = build('indexing', 'v3', credentials=creds)
urls = ['https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3', 'https://example.com/page4', 'https://example.com/page5']
for url in urls:
body = {'url': url, 'type': 'URL_UPDATED'}
response = service.urlNotifications().publish(body=body).execute()
print(f'Submitted {url}, response: {response}')
# Token refresh check after 50 minutes
if creds.expiry and (creds.expiry - datetime.now()).seconds < 600:
creds.refresh(Request()) # requires google.auth.transport.requests
If any URL returns a 403, check that the service account email is an owner in Search Console. The most common failure: the URL is not in a verified property, or the property is a domain property but the URL uses www prefix.
Production is different from a tutorial. You will hit blocked URLs (e.g., noindex or robots.txt disallow), quota limits (200 URLs per day per service account), and silent 500 errors that the API returns as empty responses.
For blocked URLs: the Indexing API still returns a 200 success code but Google ignores the request. There is no built-in error. You must pre-validate the URL with a HEAD request checking X-Robots-Tag and robots.txt. A common situation we see: an SEO tool submits 200 URLs, gets 200 success responses, but only 50 actually get indexed because the rest are blocked. No alert.
For quota limits: you can request an increase via the GCP Quotas page, but Google rarely grants more than 500 per day for new projects. Spread submissions across multiple service accounts if you need higher volume. Each account must be an owner of the Search Console property.
For 500 errors: implement exponential backoff with a max of 5 retries. If the error persists, the endpoint may be unhealthy; wait 15 minutes before retrying the batch.
Create one service account per client GCP project, add each as an owner of the respective Search Console property. Use a config file mapping client domains to service account JSON paths. Loop over clients and authenticate fresh for each. Do not reuse a single service account across properties; Google will reject submissions.
Use exactly <code>https://www.googleapis.com/auth/indexing</code>. Do not use <code>https://www.googleapis.com/auth/cloud-platform</code> or <code>https://www.googleapis.com/auth/webmasters</code>. The narrow scope ensures the service account can only call the Indexing API and nothing else, reducing security risk and avoiding consent screen issues.
Two causes: (1) the service account email is not added as an owner in Google Search Console at the property level, or (2) the Indexing API is not enabled in the GCP project. Go to Search Console > Settings > Users & Permissions and add the email. Then verify the API is enabled under APIs & Services > Library.
Use <code>google.auth.transport.requests.Request()</code> to call <code>creds.refresh(request)</code> when <code>creds.expiry</code> is less than 600 seconds away. Wrap your submission loop in a while loop that checks expiry every 50 minutes. Alternatively, use <code>google.auth.compute_engine.Credentials</code> if running on GCP, which auto-refreshes.
Default quota is 200 URLs per day per service account. You can request up to 500 via GCP Quotas page, but approval is rare. For bulk submissions (e.g., 2000 URLs), create 10 service accounts, each as an owner of the property. Distribute URLs evenly across accounts. Monitor quota with <code>quotaUser</code> parameter in API calls.
Enable detailed logging: <code>googleapiclient.discovery.build</code> with <code>cache_discovery=False</code> and set <code>logging.basicConfig(level=logging.DEBUG)</code>. Check the full HTTP response for error details. Common errors: 'invalidPayload' means the URL format is wrong (include protocol), 'authError' means the service account is missing from Search Console.
Use <code>URL_UPDATED</code> for new or changed pages you want indexed. Use <code>URL_DELETED</code> only when a page returns a 404 or 410 and you want Google to remove it from the index. Never use <code>URL_DELETED</code> for temporary removals; Google treats it as a permanent signal and may take weeks to re-index if you change your mind.
Technically yes, but the API only works for URLs in a Search Console property you own. For backlinks on external sites, you cannot submit those URLs directly unless you own the site. Some tools use the API to index their own PBNs, but Google's documentation explicitly discourages submitting URLs that are not owned. Use the API for your own content only.
Create a separate test GCP project with a test Search Console property (e.g., a subdomain like test.example.com). Submit a single URL with <code>URL_UPDATED</code>. Check the response code and log. If it returns 200, your auth is working. Use the test project for development to avoid consuming your production quota of 200 URLs per day.
Never store keys in the codebase. Use environment variables or a secrets manager like Google Secret Manager, AWS Secrets Manager, or HashiCorp Vault. Set <code>GOOGLE_APPLICATION_CREDENTIALS</code> to the path of the JSON file mounted from a secure volume. For CI/CD, inject the key as a base64-encoded environment variable and decode it at runtime.
Quick calculator. Put in the expected monthly value of a page or link batch and the natural waiting time.