{"slug":"google-search-console-integration","meta":{"title":"Google Search Console — OAuth Client, Callback URI \u0026 Pulling Query Data","slug":"google-search-console-integration","category":"Integrations","summary":"Wire Google Search Console into a Rails app with zero gems — create the Google Cloud OAuth client ID (Web application type), set the Authorized redirect URI callback, store credentials in .env, mint a durable refresh token (in-app flow or OAuth Playground), and pull real search-query data (impressions, clicks, CTR, position) with a plain Net::HTTP client.","tags":["google-search-console","gsc","oauth","seo","api","rest","service","env"],"status":"stable","visibility":"public","source_project":"llamapress.ai","layers":["controller","view"],"related":[{"title":"Google Ads API — OAuth, REST Client \u0026 Keyword Research","url":"/cookbook/google-ads-api-integration","summary":"Sibling recipe — the same Google OAuth web-client + zero-gem Net::HTTP pattern, for Ads instead of Search Console."},{"title":"Search Console API reference (Search Analytics query)","url":"https://developers.google.com/webmaster-tools/v1/searchanalytics/query","summary":"Official docs for the searchAnalytics.query endpoint used below — dimensions, filters, row limits."},{"title":"Setting up OAuth 2.0 (Google Cloud)","url":"https://support.google.com/cloud/answer/6158849","summary":"Google's own walkthrough of consent screens, client IDs, and redirect URIs."}]},"body":"# Google Search Console — OAuth Client, Callback URI \u0026 Pulling Query Data\n\n\u003e ⚠️ **Cookbook example — not live code.** (KEEP THIS CALLOUT.) Every code block below\n\u003e is an **example snippet**, **not part of the llamapress.ai codebase**, and **not\n\u003e running on this server**. This is a reference recipe for a **Leo instance (an AI coding\n\u003e agent) to implement in its own app** — read it to understand the pattern, then recreate\n\u003e it there.\n\nYour app needs real Google Search data — the actual search terms people typed to find\nthe site, with impressions, clicks, CTR, and average position, per page and per day,\nup to 16 months back. That data lives in **Google Search Console** (GSC), and this\nrecipe is the complete integration: creating the **OAuth client ID** (Application\ntype: **Web application**) in Google Cloud, setting the **Authorized redirect URI**\n(the callback your app handles), storing the credentials in `.env`, minting a durable\n**refresh token**, and a **zero-gem** `Net::HTTP` client that queries the Search\nAnalytics API. This pattern is proven in production (it powers a weekly SEO loop).\n\n\u003e **When to use:** an SEO dashboard, a \"which queries rank just off page 1\" report,\n\u003e feeding real query data into content planning, showing a client their search\n\u003e performance inside the app you built them.\n\u003e **When not to:** you only need to *look* at the data occasionally — the Search\n\u003e Console web UI (search.google.com/search-console) is free and needs no code. Also\n\u003e not for Google Analytics (different API, different scope entirely).\n\n---\n\n## The 80/20 in one breath\n\n1. In **Google Cloud Console** (console.cloud.google.com): pick/create a project →\n   **APIs \u0026 Services → Library** → enable the **Google Search Console API** →\n   configure the **OAuth consent screen** (External, then **Publish to production** —\n   this is load-bearing, see Gotchas) → **Credentials → Create Credentials →\n   OAuth client ID** → Application type **Web application** → under **Authorized\n   redirect URIs** add your callback, e.g.\n   `https://yourapp.example.com/gsc_auth/callback`.\n2. Put the client's credentials in `.env` as `GOOGLE_CLIENT_ID` /\n   `GOOGLE_CLIENT_SECRET`, then **recreate the web container**\n   (`docker compose up -d --force-recreate web`) — a plain restart does NOT reload `.env`.\n3. Verify the site as a **property** in Search Console (Domain property via a DNS TXT\n   record is best: it covers all subdomains and both http/https, and its site URL is\n   `sc-domain:example.com`).\n4. Add the controller + routes below and visit `/gsc_auth/new`. Sign in with a Google\n   account that has access to the property. The callback exchanges the code and shows\n   you the **refresh token** — put it in `.env` as `GSC_REFRESH_TOKEN`.\n5. Copy `GscSearchAnalytics` into `app/services/gsc_search_analytics.rb` and run:\n   `GscSearchAnalytics.new.top_queries` — real query data comes back.\n\n---\n\n## Layer 0 — Google Cloud setup (the part outside your codebase)\n\nThis happens in the browser, once, by a human (or by you walking the human through it):\n\n1. **Project**: console.cloud.google.com → select or create a project.\n2. **Enable the API**: APIs \u0026 Services → **Library** → search \"Google Search Console\n   API\" → **Enable**. (Skipping this yields a 403 later even with perfect OAuth.)\n3. **Consent screen**: APIs \u0026 Services → **OAuth consent screen**. User type\n   **External** (unless everyone signing in is on your own Google Workspace). Add the\n   `.../auth/webmasters.readonly` scope. Then **Publish to production**. A consent\n   screen left in **\"Testing\" expires every refresh token after 7 days** — the #1\n   cause of \"it worked last week.\"\n4. **Create the OAuth client ID**: APIs \u0026 Services → **Credentials** → **Create\n   Credentials → OAuth client ID**:\n   - **Application type:** `Web application` (\"For use with requests from a web\n     server\" — this is the right type for a Rails app; not Desktop, not iOS/Android).\n   - **Name:** anything (e.g. `MyApp Web Client`) — only shown in the console, never\n     to end users. If your app runs on multiple platforms, each platform needs its\n     own client ID.\n   - **Authorized redirect URIs:** add the EXACT callback URL your app will handle:\n     ```\n     https://yourapp.example.com/gsc_auth/callback\n     ```\n     Add `http://localhost:3000/gsc_auth/callback` too if you'll test locally.\n     Google matches this string **exactly** — scheme, host, port, path, no trailing\n     slash difference allowed. A mismatch fails with `Error 400: redirect_uri_mismatch`.\n5. Copy the **Client ID** and **Client secret** into `.env`:\n\n```bash\n# .env\nGOOGLE_CLIENT_ID=1234567890-abc123.apps.googleusercontent.com\nGOOGLE_CLIENT_SECRET=GOCSPX-xxxxxxxxxxxxxxxx\n# Filled in after the connect flow (Layer 1):\nGSC_REFRESH_TOKEN=\n# The Search Console property (domain property syntax shown):\nGSC_SITE_URL=sc-domain:example.com\n```\n\nThen recreate the container so the new `.env` loads (`docker compose up -d\n--force-recreate web` — `restart` alone keeps the old environment).\n\n6. **Verify the property** in Search Console (search.google.com/search-console → Add\n   property). Prefer a **Domain** property: Google gives you a\n   `google-site-verification=...` TXT record to add at the domain apex in your DNS.\n   Domain properties cover every subdomain and both http/https, and their API site\n   URL is `sc-domain:example.com`. A **URL-prefix** property\n   (`https://example.com/`) also works — its API site URL is the full prefix URL.\n\n---\n\n## Layer 1 — Controller: the OAuth connect flow (mints the refresh token)\n\nOne-time flow: start at `/gsc_auth/new`, Google redirects back to the callback with a\n`code`, the callback exchanges the code for tokens and displays the **refresh token**\nfor you to put in `.env`. Only the refresh token is durable — access tokens die in\nabout an hour and the client below re-mints them on demand.\n\n```ruby\n# app/controllers/gsc_auth_controller.rb\nrequire 'net/http'\nrequire 'json'\nrequire 'securerandom'\n\nclass GscAuthController \u003c ApplicationController\n  before_action :authenticate_user!\n  before_action :ensure_admin\n\n  SCOPE     = 'https://www.googleapis.com/auth/webmasters.readonly'.freeze\n  AUTH_URL  = 'https://accounts.google.com/o/oauth2/v2/auth'.freeze\n  TOKEN_URL = 'https://oauth2.googleapis.com/token'.freeze\n\n  # GET /gsc_auth/new — start the OAuth flow\n  def new\n    state = SecureRandom.hex(24)\n    session[:gsc_oauth_state] = state\n\n    query = {\n      client_id: ENV.fetch('GOOGLE_CLIENT_ID'),\n      redirect_uri: gsc_auth_callback_url,   # must EXACTLY match an Authorized redirect URI\n      response_type: 'code',\n      scope: SCOPE,\n      state: state,\n      access_type: 'offline',   # request a refresh token\n      prompt: 'consent'         # force consent so the refresh token is actually returned\n    }\n    redirect_to \"#{AUTH_URL}?#{query.to_query}\", allow_other_host: true\n  end\n\n  # GET /gsc_auth/callback — Google redirects here with ?code=...\u0026state=...\n  def callback\n    if params[:state].blank? || params[:state] != session.delete(:gsc_oauth_state)\n      return render plain: \"Invalid state parameter — start over at /gsc_auth/new\", status: :forbidden\n    end\n    if params[:error].present?\n      return render plain: \"Google authorization failed: #{params[:error]}\", status: :bad_request\n    end\n\n    resp = Net::HTTP.post_form(URI(TOKEN_URL), {\n      'client_id' =\u003e ENV.fetch('GOOGLE_CLIENT_ID'),\n      'client_secret' =\u003e ENV.fetch('GOOGLE_CLIENT_SECRET'),\n      'code' =\u003e params[:code],\n      'grant_type' =\u003e 'authorization_code',\n      'redirect_uri' =\u003e gsc_auth_callback_url   # must match the one used in #new\n    })\n    data = JSON.parse(resp.body)\n    unless resp.is_a?(Net::HTTPSuccess)\n      return render plain: \"Token exchange failed: #{data['error_description'] || data['error']}\", status: :bad_request\n    end\n\n    # Show the refresh token ONCE for the operator to copy into .env.\n    # (Alternative: persist it encrypted on a settings/organization model instead.)\n    render plain: \u003c\u003c~MSG\n      Connected. Add this line to .env, then recreate the web container:\n\n      GSC_REFRESH_TOKEN=#{data['refresh_token'] || '(no refresh_token returned — see the guide gotchas: re-run with prompt=consent)'}\n    MSG\n  end\n\n  private\n\n  def ensure_admin\n    redirect_to root_path, alert: \"Admin access required.\" unless current_user.admin?\n  end\nend\n```\n\n```ruby\n# config/routes.rb (add inside the draw block)\nnamespace :gsc_auth do\n  get :new\n  get :callback\nend\n```\n\nIf you add a \"Connect Search Console\" button instead of typing the URL, the form/link\nMUST carry `data: { turbo: false }` — otherwise Turbo intercepts the submit and the\ncross-origin redirect to `accounts.google.com` dies on CORS with no useful error.\n\n**No-code alternative — the OAuth Playground.** If you don't want the controller at\nall (single-tenant app, one-time mint), add\n`https://developers.google.com/oauthplayground` as an extra Authorized redirect URI,\nopen the Playground, click the gear → **\"Use your own OAuth credentials\"** → paste\nyour client ID/secret, put EXACTLY `https://www.googleapis.com/auth/webmasters.readonly`\n(only the URL, nothing else) in the \"Input your own scopes\" box, **Authorize APIs**,\nsign in, then **\"Exchange authorization code for tokens\"** and copy the refresh token.\nYou can remove the Playground redirect URI afterward. (Playground-specific traps are\nin the Gotchas.)\n\n---\n\n## Layer 2 — The client (zero gems, Net::HTTP)\n\nToken refresh + the Search Analytics query endpoint + the three reports that matter.\n\n```ruby\n# app/services/gsc_search_analytics.rb\nrequire 'net/http'\nrequire 'json'\nrequire 'erb'\n\n# Read-only Google Search Console (Search Analytics API) client.\n# Zero gems: refresh-token exchange and the REST endpoint via Net::HTTP.\nclass GscSearchAnalytics\n  SCOPE     = 'https://www.googleapis.com/auth/webmasters.readonly'.freeze\n  TOKEN_URL = 'https://oauth2.googleapis.com/token'.freeze\n  API_BASE  = 'https://searchconsole.googleapis.com/webmasters/v3'.freeze\n\n  class ConfigError \u003c StandardError; end\n  class ApiError \u003c StandardError; end\n\n  def initialize(site: ENV['GSC_SITE_URL'], refresh_token: ENV['GSC_REFRESH_TOKEN'])\n    @site = site.presence or raise ConfigError, \"GSC_SITE_URL is not set (e.g. sc-domain:example.com)\"\n    @refresh_token = refresh_token.presence or raise ConfigError, \"GSC_REFRESH_TOKEN is not set — run the connect flow first\"\n  end\n\n  # Sanity check: lists the properties this token can see. Your site must appear.\n  def sites\n    get(\"/sites\").fetch('siteEntry', [])\n  end\n\n  # Raw Search Analytics query. Each row:\n  # { \"keys\" =\u003e [..per dimension..], \"clicks\", \"impressions\", \"ctr\", \"position\" }\n  def query(dimensions:, start_date:, end_date:, row_limit: 1000, dimension_filter_groups: nil)\n    body = {\n      startDate: start_date.to_s,\n      endDate:   end_date.to_s,\n      dimensions: Array(dimensions),\n      rowLimit: row_limit\n    }\n    body[:dimensionFilterGroups] = dimension_filter_groups if dimension_filter_groups\n    post(\"/sites/#{ERB::Util.url_encode(@site)}/searchAnalytics/query\", body)['rows'] || []\n  end\n\n  # Top search terms in the last N days.\n  def top_queries(days: 28, limit: 200)\n    query(dimensions: ['query'], start_date: since(days), end_date: today, row_limit: limit)\n      .map { |r| { query: r['keys'].first, **metrics(r) } }\n  end\n\n  # Top landing pages in the last N days.\n  def top_pages(days: 28, limit: 200)\n    query(dimensions: ['page'], start_date: since(days), end_date: today, row_limit: limit)\n      .map { |r| { page: r['keys'].first, **metrics(r) } }\n  end\n\n  # LOW-HANGING FRUIT: page+query pairs ranking just off page 1 (positions 8-20)\n  # with real impressions — the fastest SEO wins, sorted by opportunity size.\n  def low_hanging_fruit(days: 28, min_position: 8.0, max_position: 20.0, min_impressions: 10)\n    query(dimensions: %w[page query], start_date: since(days), end_date: today, row_limit: 5000)\n      .map { |r| { page: r['keys'][0], query: r['keys'][1], **metrics(r) } }\n      .select { |h| h[:position].between?(min_position, max_position) \u0026\u0026 h[:impressions] \u003e= min_impressions }\n      .sort_by { |h| -h[:impressions] }\n  end\n\n  # Every query one specific page ranks for.\n  def queries_for_page(page_url, days: 28, limit: 500)\n    filter = [{ filters: [{ dimension: 'page', operator: 'equals', expression: page_url }] }]\n    query(dimensions: ['query'], start_date: since(days), end_date: today,\n          row_limit: limit, dimension_filter_groups: filter)\n      .map { |r| { query: r['keys'].first, **metrics(r) } }\n      .sort_by { |h| -h[:impressions] }\n  end\n\n  private\n\n  def metrics(row)\n    {\n      clicks:      row['clicks'].to_i,\n      impressions: row['impressions'].to_i,\n      ctr:         (row['ctr'].to_f * 100).round(2),  # as a percentage\n      position:    row['position'].to_f.round(1)\n    }\n  end\n\n  def today = Date.today\n  def since(days) = Date.today - days.to_i\n\n  def access_token\n    return @access_token if @access_token \u0026\u0026 @access_token_expires_at \u003e Time.current\n\n    resp = Net::HTTP.post_form(URI(TOKEN_URL), {\n      'client_id' =\u003e ENV.fetch('GOOGLE_CLIENT_ID'),\n      'client_secret' =\u003e ENV.fetch('GOOGLE_CLIENT_SECRET'),\n      'refresh_token' =\u003e @refresh_token,\n      'grant_type' =\u003e 'refresh_token'\n    })\n    data = JSON.parse(resp.body)\n    unless resp.is_a?(Net::HTTPSuccess)\n      raise ConfigError, \"GSC token refresh failed (revoked? consent screen in Testing? \" \\\n                         \"wrong client?): #{data['error']}: #{data['error_description']}\"\n    end\n    @access_token_expires_at = Time.current + data.fetch('expires_in', 3600).to_i - 60\n    @access_token = data.fetch('access_token')\n  end\n\n  def get(path)  = request(Net::HTTP::Get, path)\n  def post(path, body) = request(Net::HTTP::Post, path, body)\n\n  def request(klass, path, body = nil)\n    uri = URI(\"#{API_BASE}#{path}\")\n    req = klass.new(uri)\n    req['Authorization'] = \"Bearer #{access_token}\"\n    req['Content-Type']  = 'application/json'\n    req.body = body.to_json if body\n    resp = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true, read_timeout: 60) { |http| http.request(req) }\n    raise ApiError, \"GSC API #{resp.code} on #{path}: #{resp.body\u0026.slice(0, 500)}\" unless resp.is_a?(Net::HTTPSuccess)\n    JSON.parse(resp.body)\n  end\nend\n```\n\n---\n\n## Layer 3 — Using it\n\n```ruby\n# Anywhere in the app (or bin/rails runner):\ngsc = GscSearchAnalytics.new\n\n# 1) Verify auth + property access — your site must be in this list:\ngsc.sites\n# =\u003e [{\"siteUrl\"=\u003e\"sc-domain:example.com\", \"permissionLevel\"=\u003e\"siteOwner\"}]\n\n# 2) The reports:\ngsc.top_queries(days: 28)\ngsc.top_pages(days: 28)\ngsc.low_hanging_fruit                        # positions 8-20 → your fastest SEO wins\ngsc.queries_for_page(\"https://example.com/pricing\")\n# Each row: { query:/page:, clicks:, impressions:, ctr:, position: }\n```\n\nWire the results into a dashboard, a weekly report job, or a CSV export — the data\nshape is a flat array of hashes, so a plain table view is enough to start.\n\n---\n\n## Gotchas (the hard-won stuff)\n\n- **Consent screen in \"Testing\" = refresh tokens die after 7 days.** The integration\n  works perfectly for a week, then every call fails with `invalid_grant`. Publish the\n  consent screen to **production** before minting the token you intend to keep. Tell:\n  if the token exchange response includes `refresh_token_expires_in`, you are in\n  Testing mode; a production-mode exchange has no expiry field.\n- **The redirect URI must match EXACTLY.** `redirect_uri_mismatch` means the\n  `redirect_uri` your app sent differs from every entry under Authorized redirect\n  URIs — check scheme (http vs https), host, port, path, and trailing slash. Behind a\n  proxy, Rails may generate `http://` URLs; force `https` in the URL helper or config.\n- **Google omits the refresh token on re-consent.** A second OAuth pass for a user\n  who already approved returns NO `refresh_token` at all. Always send\n  `access_type: 'offline'` + `prompt: 'consent'`, and never overwrite a stored\n  refresh token with a blank.\n- **`.env` changes need a container recreate, not a restart.** Docker bakes env vars\n  in at container create time. `docker compose restart` keeps the old values —\n  use `docker compose up -d --force-recreate web` (or `down` + `up -d`).\n- **Turbo breaks the OAuth redirect.** Any \"Connect\" form/button must carry\n  `data: { turbo: false }`, or the cross-origin redirect to `accounts.google.com`\n  fails silently on CORS.\n- **Scope box in the OAuth Playground: paste ONLY the URL.**\n  `Error 400: invalid_scope ... invalid=[console, search]` means words like \"Search\n  Console\" got into the scope box and were space-split into bogus scopes.\n- **Playground: \"Use your own OAuth credentials\" must actually take.** After enabling\n  it you must RE-run the Authorize step — the tell is the exchange request showing\n  YOUR `client_id`. A token minted under Google's default Playground client cannot be\n  refreshed by your app (`unauthorized_client`) and is auto-revoked within 24 hours.\n  And use **\"Exchange authorization code for tokens\"**, not \"Refresh access token\" —\n  refreshing just re-uses whatever (possibly wrong-client) token you already had.\n- **403 with working auth = API not enabled, or no property access.** Enable the\n  Search Console API in the SAME Google Cloud project as the OAuth client, and make\n  sure the Google account that approved the consent has at least \"Full\" (or\n  restricted read) access to the property in Search Console → Settings → Users.\n- **Domain property site URL is `sc-domain:example.com`** — not a normal URL. Only\n  URL-prefix properties use the `https://example.com/` form. Passing the wrong form\n  404s/403s the query call. `gsc.sites` shows the exact strings your token can see.\n- **Data starts at verification — Google does not backfill.** A property verified\n  today has no history; \"before vs after\" comparisons against pre-verification dates\n  compare against empty data. Also: the most recent 1–2 days are always incomplete\n  (finalization lag), and a brand-new property shows nothing for the first ~2–3 days.\n- **The refresh token is durable, but not immortal.** It survives restarts and\n  deploys in `.env`. It dies if: the OAuth client is rotated (this also kills every\n  OTHER token minted under that client — Gmail integrations included, if they share\n  the client), the user revokes the app in their Google Account security settings, or\n  the token sits completely unused for 6 months. `ConfigError: token refresh failed`\n  is the re-mint signal.\n\n---\n\n## Files this pattern touches\n\n```\napp/controllers/gsc_auth_controller.rb      (one-time connect flow; removable after minting)\napp/services/gsc_search_analytics.rb\nconfig/routes.rb\n.env                                        (GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET,\n                                             GSC_REFRESH_TOKEN, GSC_SITE_URL)\n```\n\n## How to adapt to your schema\n\n1. **Multi-tenant (each org connects its own Search Console)?** Move the refresh\n   token off `.env` onto your tenant model (`organizations.gsc_refresh_token`,\n   `encrypts :gsc_refresh_token`), persist it in the callback instead of rendering\n   it, and pass it to `GscSearchAnalytics.new(refresh_token: org.gsc_refresh_token,\n   site: org.gsc_site_url)`. The Google Ads cookbook guide shows this exact\n   per-organization credential pattern.\n2. **Single-tenant / no controller wanted?** Skip Layer 1 entirely and mint via the\n   OAuth Playground path (Layer 1, last paragraph) — zero code changes, just the\n   `.env` entries.\n3. **URL-prefix property instead of a domain property?** Set\n   `GSC_SITE_URL=https://example.com/` (the exact verified prefix, trailing slash\n   included).\n4. **Need write access (submit sitemaps, inspect URLs)?** Swap the scope to\n   `https://www.googleapis.com/auth/webmasters` and re-mint — the readonly token\n   cannot be upgraded in place.\n5. **Safe to drop:** the `low_hanging_fruit` / `queries_for_page` helpers if you only\n   want totals; the whole auth controller once the token is minted and stored.\n"}