Building a Receipt Tracker with Telegram and Claude

I needed to track receipts. Groceries, supplies, random purchases - everything generates a receipt. Some are thermal paper that fades in a week. Some get crumpled. All need to be tracked for taxes.

Tried the shoebox method first. Twelve months of faded receipts at tax time, half illegible.

Tried spreadsheets next. Better, but manually typing every receipt after a long day? I’d fall behind, receipts would pile up, back to the shoebox.

Looked at receipt scanning apps. Monthly subscriptions, clunky interfaces, data locked in their ecosystem. I wanted my data in a Google Sheet where I could run my own formulas. Images in Drive for backups.

The constraint: friction had to be near zero. Take a photo, forget about it.

The Solution

I built a Telegram bot. I already have Telegram on my phone. Snap a photo of the receipt at the register, send it to the bot, done. Three seconds. When I have time later, I hit /process and Claude reads everything I’ve sent that day.

The pipeline:

1. Send a receipt photo to a Telegram bot
2. Image gets uploaded to Google Drive (organized by year/month)
3. On command, Claude AI reads every unprocessed receipt and extracts the data
4. Store name, date, items, totals - all land in a Google Sheet

I split it into two phases on purpose. Ingestion is instant - the bot saves your photo and replies in seconds. Processing happens when you’re ready, via a /process command. This keeps the bot snappy even when Claude is chewing through a stack of receipts. I can send five receipts in a row and process them all at once later.

The Stack

• Python with python-telegram-bot for the bot
• Claude CLI (Claude Code) as a subprocess for vision/OCR - no API key needed, just your existing Claude subscription
• Google Drive API for image storage
• Google Sheets API (via gspread) for the spreadsheet
• Docker to package everything

How It Works

Phase 1: Snap and Save

When you send a photo, the bot downloads it, uploads to Drive in a YYYY/MM-Month/ folder structure, and appends a “pending” row to the sheet:

 1async def photo_handler(update: Update, context: ContextTypes.DEFAULT_TYPE):
 2    status_msg = await update.message.reply_text("Saving receipt...")
 3
 4    # Download the highest resolution photo
 5    photo = update.message.photo[-1]
 6    file = await context.bot.get_file(photo.file_id)
 7    image_bytes = bytes(await file.download_as_bytearray())
 8
 9    # Upload to Drive
10    today = datetime.now().strftime("%Y-%m-%d")
11    short_id = uuid.uuid4().hex[:8]
12    filename = f"{today}_receipt_{short_id}.jpg"
13    drive_link, drive_file_id = await asyncio.to_thread(
14        drive.upload_receipt, image_bytes, filename, today
15    )
16
17    # Add pending row to Sheet
18    user_label = _get_user_label(update)
19    row_num = await asyncio.to_thread(
20        sheets.append_pending, user_label, drive_link, drive_file_id
21    )
22
23    await status_msg.edit_text(
24        f"Receipt saved (row {row_num}).\n"
25        f"Use /process to extract data."
26    )

Drive folders are created automatically - year folder, then month subfolder. January receipts and February receipts are filed where I can find them, without me ever creating a folder.

Phase 2: Extract with Claude

When you send /process, the bot finds your pending receipts, downloads each image from Drive, and pipes it through Claude CLI:

 1async def extract_receipt(image_bytes: bytes) -> ReceiptData:
 2    with tempfile.NamedTemporaryFile(suffix=".jpg", delete=False) as f:
 3        f.write(image_bytes)
 4        temp_path = f.name
 5
 6    try:
 7        prompt = EXTRACTION_PROMPT.format(image_path=temp_path)
 8        cmd = ["claude", "-p", prompt, "--output-format", "text",
 9               "--allowedTools", "Read"]
10
11        proc = await asyncio.create_subprocess_exec(
12            *cmd,
13            stdout=asyncio.subprocess.PIPE,
14            stderr=asyncio.subprocess.PIPE,
15        )
16        stdout, stderr = await asyncio.wait_for(
17            proc.communicate(), timeout=120
18        )
19
20        output = stdout.decode().strip()
21        data = _extract_json(output)
22        return ReceiptData(**data)
23    finally:
24        os.unlink(temp_path)

The prompt asks Claude to return structured JSON with store name, date, items, totals, tax, and payment method. Claude is remarkably good at this. Even crumpled thermal paper receipts come back with accurate line items - quantities, unit prices, tax, payment method.

Claude doesn’t always return clean JSON - sometimes it wraps it in markdown fences or adds explanation. The parser tries multiple strategies to extract it.

Key Challenges

Google Auth

I originally planned to use a service account for Google APIs. Turns out Google removed storage quota from service accounts in late 2024 - they can’t own files anymore. The fix was switching to OAuth with the device code flow:

 1def _device_code_flow() -> Credentials:
 2    client_id, client_secret = _load_client_secret()
 3
 4    # Request device code
 5    resp = http_requests.post(DEVICE_AUTH_URL, data={
 6        "client_id": client_id,
 7        "scope": " ".join(SCOPES),
 8    })
 9    device_data = resp.json()
10
11    print("=" * 50)
12    print("Google Authentication Required")
13    print(f"1. Visit: {device_data['verification_url']}")
14    print(f"2. Enter code: {device_data['user_code']}")
15    print("=" * 50)
16
17    # Poll for authorization
18    while time.time() - start < expires_in:
19        time.sleep(interval)
20        token_data = # ... poll token endpoint
21        if "access_token" in token_data:
22            return Credentials(token=token_data["access_token"], ...)

On first run, you get a URL and a code. Visit the URL on any device, enter the code, done. Works perfectly from Docker containers and headless servers.

Claude CLI in Docker

Claude CLI auth lives in the macOS Keychain, not in ~/.claude. That directory has debug logs and session data, but no credentials. Inside Docker, you need to run claude /login and persist the volume.

Also, Claude CLI needs --allowedTools Read in non-interactive mode. Without it, Claude politely asks for file permission… to nobody. The process hangs and returns nothing useful.

What I Learned

Decouple ingestion from processing. Making the bot respond instantly and deferring the slow AI work to an explicit command was the right call. Nobody wants to wait 30 seconds after sending a photo.

Python 3.9 doesn’t support str | None syntax. Seems obvious in hindsight, but it’s easy to miss when you develop on a newer Python and deploy to an older one. Docker with Python 3.13 sidesteps this.

The device code OAuth flow is underrated. It works from anywhere - Docker, SSH, a phone - and doesn’t need a browser on the host machine.

Google killed service account storage quotas. As of late 2024, service accounts can’t own files in Drive. You need either a Shared Drive or OAuth delegation.

The Result

After /process, the bot replies with a formatted summary:

✅ Processed 2 receipt(s).

🧾 Costco Wholesale
📅 2026-02-06

  Avocado Oil 2pk  $12.99
  Chicken Breast 6lb x2  $23.98
  Tortillas 100ct  $6.49

Tax: $0.00
Total: $43.46
Paid: Visa

🧾 Restaurant Supply
📅 2026-02-07

  To-Go Containers 200ct  $34.99
  Aluminum Foil 1000ft  $28.50

Tax: $6.12
Total: $69.61
Paid: Debit

Now I can see at a glance what I spent where. The same data lands in the Google Sheet where I track monthly spending by vendor, calculate totals, and generate reports for my accountant. All from photos I took in three seconds.

The whole thing is about 400 lines of Python across 8 files. No frameworks, no abstractions, no database. Just a bot that takes photos and fills in a spreadsheet.

I’ve saved hours every week. No more manual data entry, no more faded receipts I can’t read, no more tax season panic. The total cost is whatever I’m already paying for my Claude subscription and the free tiers of Google Sheets and Drive.

Sometimes that’s all you need.