Tony_tech/chess-project
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
55cb93244742ef1299760ca74061be2f40253b71
chess-project/scripts
T
History

README.md

Build Scripts

Overview

This directory contains setup and build scripts for the chess tutor application:

  1. Tactical Puzzles Setup - Downloads and configures tactical puzzles from Lichess
  2. Wikipedia Cache Builder - Downloads Wikipedia articles for chess openings

Tactical Puzzles Setup Script

Quick Start

Docker (Recommended for production)

Use the helper script from your host machine:

# Default: 100 puzzles per pattern (800 total)
./scripts/docker-setup-puzzles.sh

# Customize number of puzzles
./scripts/docker-setup-puzzles.sh --max-puzzles 500

# Force re-download
./scripts/docker-setup-puzzles.sh --force

The script will:

  • Check if your container is running
  • Download puzzles inside the container
  • Store them in persistent Docker volumes
  • Puzzles persist across container restarts

Local Development

Run this command once before using the tactical practice feature:

cd chess_tutor
python3 scripts/setup_tactical_puzzles.py

# Or specify the number of puzzles:
python3 scripts/setup_tactical_puzzles.py --max-puzzles 100
python3 scripts/setup_tactical_puzzles.py --max-puzzles 500

What It Does

The setup script automatically:

  1. ✅ Checks dependencies - Verifies zstd is installed (installs it if needed)
  2. ✅ Downloads database - Fetches the Lichess puzzle database (~500MB compressed)
  3. ✅ Decompresses - Extracts the CSV file (~3.5GB uncompressed)
  4. ✅ Filters puzzles - Extracts 20 high-quality puzzles for each tactical pattern:
    • PIN
    • FORK
    • SKEWER
    • DISCOVERED_CHECK
    • DOUBLE_ATTACK
    • OVERLOADING
    • BACK_RANK_WEAKNESS
    • TRAPPED_PIECE
  5. ✅ Converts format - Transforms Lichess format to our JSON fixture format
  6. ✅ Saves fixtures - Writes to fixtures/tactics/*.json
  7. ✅ Creates marker - Places .tactical_puzzles_configured file to prevent re-running

Command Line Options

python3 scripts/setup_tactical_puzzles.py [OPTIONS]

Options:
  --max-puzzles N    Number of puzzles to extract per pattern (default: 20)
  --force            Force re-run setup without prompting
  -h, --help         Show help message

Examples:
  # Default: 20 puzzles per pattern (160 total)
  python3 scripts/setup_tactical_puzzles.py

  # 100 puzzles per pattern (800 total)
  python3 scripts/setup_tactical_puzzles.py --max-puzzles 100

  # 500 puzzles per pattern (4000 total) - recommended for production
  python3 scripts/setup_tactical_puzzles.py --max-puzzles 500

  # Force re-run without prompting
  python3 scripts/setup_tactical_puzzles.py --max-puzzles 200 --force

Requirements

  • Python 3.7+ (with python-chess library - auto-installed if missing)
  • zstd (auto-installed on macOS/Linux if missing)
  • ~4GB disk space (for downloaded and decompressed database)
  • Internet connection (for downloading ~500MB file)

Time Estimate

  • First run: 5-10 minutes (depending on internet speed)
  • Subsequent runs: Instant (uses cached database)

Quality Criteria

The script filters puzzles based on:

  • Popularity: ≥ 50 (well-liked by users)
  • Rating: 1200-2000 (appropriate difficulty for learning)
  • Plays: ≥ 50 (well-tested)
  • Theme: Must match the tactical pattern

Only the top 20 puzzles (by popularity) are selected for each pattern.

File Structure

chess_tutor/
├── scripts/
│   ├── setup_tactical_puzzles.py    # Main setup script
│   └── README.md                     # This file
├── downloads/                        # Created by script
│   ├── lichess_db_puzzle.csv.zst    # Downloaded database (cached)
│   └── lichess_db_puzzle.csv        # Decompressed database (cached)
├── fixtures/
│   └── tactics/                      # Created by script
│       ├── pin.json                  # 20 PIN puzzles
│       ├── fork.json                 # 20 FORK puzzles
│       ├── skewer.json               # 20 SKEWER puzzles
│       └── ...                       # Other patterns
└── .tactical_puzzles_configured      # Marker file (created by script)

Re-running Setup

If you want to re-run the setup (e.g., to get fresh puzzles):

# Option 1: Delete the marker file and re-run
rm chess_tutor/.tactical_puzzles_configured
python3 scripts/setup_tactical_puzzles.py

# Option 2: The script will ask if you want to re-run
python3 scripts/setup_tactical_puzzles.py
# Answer 'y' when prompted

Cleaning Up

To save disk space after setup:

# Delete the downloaded database (keeps the fixtures)
rm -rf chess_tutor/downloads/

# The fixtures in chess_tutor/fixtures/tactics/ will remain

Troubleshooting

"zstd not found"

The script will attempt to auto-install zstd. If it fails:

macOS:

brew install zstd

Ubuntu/Debian:

sudo apt-get install zstd

Fedora/RHEL:

sudo dnf install zstd

"python-chess not found"

The script will attempt to auto-install python-chess. If it fails:

pip3 install python-chess

"Download failed"

Check your internet connection and try again. The Lichess database is updated daily, so temporary issues may occur.

"No puzzles found for pattern X"

This is rare but can happen if the database doesn't have enough puzzles matching the criteria. The script will warn you but continue with other patterns.

Manual Setup (Alternative)

If the automatic script doesn't work, you can manually:

  1. Download: https://database.lichess.org/lichess_db_puzzle.csv.zst
  2. Decompress with zstd -d lichess_db_puzzle.csv.zst
  3. Filter puzzles using the Python code in docs/LICHESS_PUZZLES.md
  4. Convert to JSON format and save to fixtures/tactics/

Source

  • Database: https://database.lichess.org/
  • License: Creative Commons CC0 (public domain)
  • Documentation: See docs/LICHESS_PUZZLES.md for detailed information

Support

If you encounter issues:

  1. Check the terminal output for specific error messages
  2. Ensure you have Python 3.7+ installed: python3 --version
  3. Ensure you have internet connectivity
  4. Try re-running the script
  5. Check docs/LICHESS_PUZZLES.md for manual setup instructions

Wikipedia Cache Builder

Overview

Downloads full Wikipedia articles for all chess opening families and caches them locally for offline use.

Quick Start

# Install dependencies (if not already done)
npm install

# Run the Wikipedia cache builder
npm run cache:wikipedia

Benefits

  • ✅ Offline access - Works completely offline once cached
  • ✅ Instant loading - No network delay
  • ✅ No rate limiting - Avoids Wikipedia API rate limits
  • ✅ Full content - Gets complete articles, not just summaries
  • ✅ Version controlled - Safe to commit to git

What It Does

  1. Extracts opening families from your opening database (public/openings/*.json)
  2. Searches Wikipedia for each family using the OpenSearch API
  3. Downloads full article content including:
    • Introduction
    • Main sections (History, Ideas, Variations, etc.)
    • Up to 5 relevant sections per opening
  4. Saves locally to public/wikipedia/*.json
  5. Creates an index at public/wikipedia/index.json

Output Structure

Each opening family gets a JSON file like public/wikipedia/french-defense.json:

{
  "openingFamily": "French Defense",
  "title": "French Defence",
  "url": "https://en.wikipedia.org/wiki/French_Defence",
  "sections": [
    {
      "title": "Introduction",
      "text": "The French Defence is a chess opening..."
    },
    {
      "title": "History",
      "text": "The first known game with the French Defense..."
    }
  ],
  "lastModified": "2025-01-15T...",
  "license": "CC BY-SA 3.0",
  "licenseUrl": "https://creativecommons.org/licenses/by-sa/3.0/",
  "fetchedAt": 1705334400000
}

Committing to Git

The generated files are safe to commit to your repository:

  • Wikipedia content is licensed under CC BY-SA 3.0 (allows redistribution with attribution)
  • Each file includes proper license information
  • Files are versioned, so you can review changes before committing

When to Run

  • Initial setup: Run once to cache all Wikipedia articles
  • After adding new openings: Re-run to fetch articles for new families
  • Periodically: Re-run every few months to get updated Wikipedia content

Rate Limiting

The script includes a 1-second delay between requests to be respectful to Wikipedia's servers.

Estimated time:

  • 50 opening families = ~1-2 minutes
  • 100 opening families = ~2-3 minutes

Fallback Strategy

The app uses a 3-tier approach for Wikipedia content:

  1. Local cache (public/wikipedia/*.json) - Fastest, always available
  2. localStorage cache - In-browser cache for API fetches
  3. Live API - Fallback if local cache missing

This ensures Wikipedia content is always available, even for openings not in your local cache.

License & Attribution

Wikipedia content is licensed under CC BY-SA 3.0:

  • ✅ Commercial use allowed
  • ✅ Modification allowed
  • ✅ Redistribution allowed
  • ⚠️ Attribution required (included in JSON files)
  • ⚠️ Derivative works must use same license

More info: https://creativecommons.org/licenses/by-sa/3.0/

Implementation Details

Script location: scripts/fetch-wikipedia-openings.ts

Dependencies:

  • Node.js 18+
  • tsx (TypeScript executor)

API endpoints used:

  • Wikipedia OpenSearch API (for finding articles)
  • Wikipedia MediaWiki Parse API (for full content)

Output directory: public/wikipedia/

Example Usage

# Run the cache builder
npm run cache:wikipedia

# Output:
# 🌐 Wikipedia Opening Cache Builder
# 📚 Extracting opening families from database...
# ✓ Found 47 unique opening families
#
# 📖 Processing: French Defense
#   Searching Wikipedia for: "French Defense"
#   ✓ Found: "French Defence"
#   Fetching full article...
#   ✓ Fetched 5 sections
#   ✓ Saved to: french-defense.json
# ...
# ✨ Wikipedia Cache Build Complete!
# ✓ Successful: 45
# ⚠️  Skipped:    2
# ❌ Failed:     0
Reference in New Issue View Git Blame Copy Permalink