264 lines
7.0 KiB
Markdown
264 lines
7.0 KiB
Markdown
# ComicVine SQLite Database Updater
|
|
|
|
A Python script that maintains a local SQLite database of ComicVine data with automatic daily updates, email notifications, and intelligent cleanup.
|
|
|
|
Based on [/u/Logical-Feedback-543's](https://old.reddit.com/r/comicrackusers/comments/1q7rex8/here_it_is_an_offline_comicvine_tagger_for/) script.
|
|
|
|
## Features
|
|
|
|
- Incremental Updates: Only fetches changes since last sync (1-day safety margin)
|
|
- Dated Backups: Creates dated copies (e.g., `localcv-2026-01-26.db`) for version history
|
|
- Automatic Cleanup: Keeps only the last N days of databases to save disk space
|
|
- Email Reports: Sends HTML email reports after each sync
|
|
- Resume Capability: Recovers from interruptions without losing progress
|
|
- Cron-Ready: Fully non-interactive mode for automated daily runs
|
|
- Config File: All settings in one file in the script directory
|
|
|
|
## What Gets Synced
|
|
|
|
The script syncs these ComicVine resources:
|
|
- Publishers
|
|
- Volumes (series)
|
|
- Issues (with full credits and metadata)
|
|
- Persons (creators/writers/artists)
|
|
|
|
## Requirements
|
|
|
|
- Python 3.9+
|
|
- `requests` library: `pip install requests`
|
|
- ComicVine API key (free at https://comicvine.gamespot.com/api/)
|
|
- SQLite database with the appropriate schema (see [this reddit post](https://old.reddit.com/r/comicrackusers/comments/1q7rex8/here_it_is_an_offline_comicvine_tagger_for/))
|
|
|
|
## Quick Start
|
|
|
|
### 1. Setup
|
|
|
|
```bash
|
|
# Install dependencies
|
|
pip install requests
|
|
|
|
# Copy the example config
|
|
cp comicvine_config.env.example comicvine_config.env
|
|
|
|
# Edit config with your settings
|
|
vi comicvine_config.env
|
|
```
|
|
|
|
### 2. Configure `comicvine_config.env`
|
|
|
|
```env
|
|
# Required: ComicVine API Key
|
|
COMICVINE_API_KEY=your_api_key_here
|
|
|
|
# Optional: Database retention (default: 7 days)
|
|
KEEP_DAYS=7
|
|
|
|
# Optional: Email notifications
|
|
EMAIL_TO=your@email.com
|
|
EMAIL_FROM=comicvine@yourdomain.com
|
|
SMTP_HOST=mail.smtp2go.com
|
|
SMTP_PORT=2525
|
|
SMTP_USER=your_smtp_username
|
|
SMTP_PASS=your_smtp_password
|
|
```
|
|
|
|
### 3. Run
|
|
|
|
```bash
|
|
# Interactive mode (first time)
|
|
python3 sqlite_cv_updater.py
|
|
|
|
# Non-interactive mode (for cron)
|
|
python3 sqlite_cv_updater.py --non-interactive --db-dir /path/to/db
|
|
```
|
|
|
|
## How It Works
|
|
|
|
### Daily Workflow
|
|
|
|
```
|
|
Day 1 (Jan 25):
|
|
- Have: localcv-2026-01-25.db
|
|
|
|
Day 2 (Jan 26) - Script runs at 3 AM:
|
|
1. Finds yesterday's database: localcv-2026-01-25.db
|
|
2. Copies it to: localcv-2026-01-26.db
|
|
3. Updates new database with changes from Jan 25-26
|
|
4. Sends email report (if configured)
|
|
5. Deletes old databases (keeps last 7 days)
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Command Line Options
|
|
|
|
```bash
|
|
# Basic usage
|
|
python3 sqlite_cv_updater.py --non-interactive --db-dir /path/to/db
|
|
|
|
# Custom retention period
|
|
python3 sqlite_cv_updater.py --non-interactive --db-dir /path/to/db --keep-days 14
|
|
|
|
# Disable cleanup (keep all databases)
|
|
python3 sqlite_cv_updater.py --non-interactive --db-dir /path/to/db --keep-days 0
|
|
|
|
# Update specific database in-place (no dated copy)
|
|
python3 sqlite_cv_updater.py --non-interactive --db /path/to/localcv.db --no-copy
|
|
|
|
# Quiet mode (less output)
|
|
python3 sqlite_cv_updater.py --non-interactive --quiet --db-dir /path/to/db
|
|
|
|
# Custom start date (override automatic date detection)
|
|
python3 sqlite_cv_updater.py --non-interactive --db-dir /path/to/db --start-date 2026-01-01
|
|
```
|
|
|
|
### All Available Options
|
|
|
|
```
|
|
Arguments:
|
|
--db-dir DIR Directory containing dated databases
|
|
--db FILE Path to specific database file
|
|
--api-key KEY ComicVine API key
|
|
--start-date YYYY-MM-DD Custom start date for sync
|
|
--keep-days N Number of databases to keep (default: 7, 0=disable cleanup)
|
|
--no-copy Update database in place (no dated copy)
|
|
--non-interactive Run without prompts (required for cron)
|
|
--quiet Reduce output verbosity
|
|
|
|
Email Options:
|
|
--email-to EMAIL Email address for reports
|
|
--email-from EMAIL From address for emails
|
|
--smtp-host HOST SMTP server (default: mail.smtp2go.com)
|
|
--smtp-port PORT SMTP port (default: 2525)
|
|
--smtp-user USER SMTP username
|
|
--smtp-pass PASS SMTP password
|
|
--smtp-tls Use STARTTLS
|
|
--smtp-ssl Use SSL
|
|
```
|
|
|
|
## Cron Setup
|
|
|
|
### Basic Crontab Entry
|
|
|
|
```bash
|
|
# Edit crontab
|
|
crontab -e
|
|
|
|
# Run daily at 3 AM
|
|
0 3 * * * cd /home/user/scripts && /usr/bin/python3 sqlite_cv_updater.py --non-interactive --db-dir /home/user/db >> /home/user/comicvine_sync.log 2>&1
|
|
```
|
|
|
|
### Advanced Crontab with Logging
|
|
|
|
```bash
|
|
# Run daily at 3 AM with log rotation
|
|
0 3 * * * cd /home/user/scripts && /usr/bin/python3 sqlite_cv_updater.py --non-interactive --db-dir /home/user/db >> /home/user/comicvine_sync.log 2>&1
|
|
|
|
# Weekly log rotation (runs Sundays at 2:59 AM)
|
|
59 2 * * 0 mv /home/user/comicvine_sync.log /home/user/comicvine_sync.log.old 2>/dev/null
|
|
```
|
|
|
|
### Testing Before Adding to Cron
|
|
|
|
```bash
|
|
# Test the exact command cron will run
|
|
cd /home/user/scripts && /usr/bin/python3 sqlite_cv_updater.py --non-interactive --db-dir /home/user/db
|
|
|
|
# Check exit code (should be 0 for success)
|
|
echo $?
|
|
|
|
# Verify email was sent
|
|
# Check that today's database was created
|
|
ls -lh /home/user/db/localcv-$(date +%Y-%m-%d).db
|
|
```
|
|
|
|
## Email Reports
|
|
|
|
When configured, you'll receive HTML email reports with:
|
|
|
|
### Subject Line
|
|
- ✓ **Success**: `✓ ComicVine Sync Success - 2026-01-26`
|
|
- ✗ **Failure**: `✗ ComicVine Sync Failed - 2026-01-26`
|
|
|
|
### Report Contents
|
|
- Sync duration
|
|
- Statistics (items added/updated)
|
|
- Full timestamped logs
|
|
- Color-coded formatting
|
|
|
|
### Example Email Preview
|
|
```
|
|
ComicVine Database Sync Report
|
|
Duration: 2m 34s
|
|
|
|
Statistics:
|
|
- API calls made: 47
|
|
- Publishers added: 2
|
|
- Publishers updated: 5
|
|
- Volumes added: 15
|
|
- Volumes updated: 23
|
|
- Issues added: 142
|
|
- Issues updated: 8
|
|
|
|
[Full detailed logs with timestamps...]
|
|
```
|
|
|
|
## SMTP Provider Examples
|
|
|
|
### SMTP2GO (Default)
|
|
```env
|
|
SMTP_HOST=mail.smtp2go.com
|
|
SMTP_PORT=2525
|
|
SMTP_USER=your_username
|
|
SMTP_PASS=your_password
|
|
```
|
|
|
|
### Gmail (App Password Required)
|
|
```env
|
|
SMTP_HOST=smtp.gmail.com
|
|
SMTP_PORT=587
|
|
SMTP_USER=your@gmail.com
|
|
SMTP_PASS=your_app_specific_password
|
|
```
|
|
|
|
### SendGrid
|
|
```env
|
|
SMTP_HOST=smtp.sendgrid.net
|
|
SMTP_PORT=587
|
|
SMTP_USER=apikey
|
|
SMTP_PASS=your_sendgrid_api_key
|
|
```
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
/path/to/your/scripts/
|
|
├── sqlite_cv_updater.py # Main script
|
|
├── comicvine_config.env # Your configuration (not in git)
|
|
└── comicvine_config.env.example # Example template (in git)
|
|
|
|
/path/to/your/db/
|
|
├── localcv-2026-01-20.db
|
|
├── localcv-2026-01-21.db
|
|
├── localcv-2026-01-22.db
|
|
├── localcv-2026-01-23.db
|
|
├── localcv-2026-01-24.db
|
|
├── localcv-2026-01-25.db
|
|
└── localcv-2026-01-26.db ← Today's (newest)
|
|
```
|
|
|
|
## Configuration Priority
|
|
|
|
Settings are loaded in this order (highest priority first):
|
|
|
|
1. Command-line arguments (e.g., `--api-key`, `--keep-days`)
|
|
2. Config file (`comicvine_config.env`)
|
|
3. Environment variables (e.g., `COMICVINE_API_KEY`)
|
|
4. Default values
|
|
|
|
Example:
|
|
```bash
|
|
# Config file has: KEEP_DAYS=7
|
|
# This run will keep 14 days instead:
|
|
python3 sqlite_cv_updater.py --keep-days 14 --db-dir /path/to/db
|
|
``` |