scripts

2025-12-15 14:54:27 +01:00
parent 7383af7884
commit 35ba74e5c0
5 changed files with 1006 additions and 0 deletions
@@ -0,0 +1,365 @@
 # Build Scripts
 ## Overview
 This directory contains setup and build scripts for the chess tutor application:
 1. **Tactical Puzzles Setup** - Downloads and configures tactical puzzles from Lichess
 2. **Wikipedia Cache Builder** - Downloads Wikipedia articles for chess openings
 ---
 # Tactical Puzzles Setup Script
 ## Quick Start
 ### Docker (Recommended for production)
 **Use the helper script from your host machine:**
 ```bash
 # Default: 100 puzzles per pattern (800 total)
 ./scripts/docker-setup-puzzles.sh
 # Customize number of puzzles
 ./scripts/docker-setup-puzzles.sh --max-puzzles 500
 # Force re-download
 ./scripts/docker-setup-puzzles.sh --force
 ```
 The script will:
 - Check if your container is running
 - Download puzzles inside the container
 - Store them in persistent Docker volumes
 - Puzzles persist across container restarts
 ### Local Development
 **Run this command once before using the tactical practice feature:**
 ```bash
 cd chess_tutor
 python3 scripts/setup_tactical_puzzles.py
 # Or specify the number of puzzles:
 python3 scripts/setup_tactical_puzzles.py --max-puzzles 100
 python3 scripts/setup_tactical_puzzles.py --max-puzzles 500
 ```
 ## What It Does
 The setup script automatically:
 1. ✅ **Checks dependencies** - Verifies `zstd` is installed (installs it if needed)
 2. ✅ **Downloads database** - Fetches the Lichess puzzle database (~500MB compressed)
 3. ✅ **Decompresses** - Extracts the CSV file (~3.5GB uncompressed)
 4. ✅ **Filters puzzles** - Extracts 20 high-quality puzzles for each tactical pattern:
   - PIN
   - FORK
   - SKEWER
   - DISCOVERED_CHECK
   - DOUBLE_ATTACK
   - OVERLOADING
   - BACK_RANK_WEAKNESS
   - TRAPPED_PIECE
 5. ✅ **Converts format** - Transforms Lichess format to our JSON fixture format
 6. ✅ **Saves fixtures** - Writes to `fixtures/tactics/*.json`
 7. ✅ **Creates marker** - Places `.tactical_puzzles_configured` file to prevent re-running
 ## Command Line Options
 ```bash
 python3 scripts/setup_tactical_puzzles.py [OPTIONS]
 Options:
  --max-puzzles N    Number of puzzles to extract per pattern (default: 20)
  --force            Force re-run setup without prompting
  -h, --help         Show help message
 Examples:
  # Default: 20 puzzles per pattern (160 total)
  python3 scripts/setup_tactical_puzzles.py
  # 100 puzzles per pattern (800 total)
  python3 scripts/setup_tactical_puzzles.py --max-puzzles 100
  # 500 puzzles per pattern (4000 total) - recommended for production
  python3 scripts/setup_tactical_puzzles.py --max-puzzles 500
  # Force re-run without prompting
  python3 scripts/setup_tactical_puzzles.py --max-puzzles 200 --force
 ```
 ## Requirements
 - **Python 3.7+** (with `python-chess` library - auto-installed if missing)
 - **zstd** (auto-installed on macOS/Linux if missing)
 - **~4GB disk space** (for downloaded and decompressed database)
 - **Internet connection** (for downloading ~500MB file)
 ## Time Estimate
 - **First run**: 5-10 minutes (depending on internet speed)
 - **Subsequent runs**: Instant (uses cached database)
 ## Quality Criteria
 The script filters puzzles based on:
 - **Popularity**: ≥ 50 (well-liked by users)
 - **Rating**: 1200-2000 (appropriate difficulty for learning)
 - **Plays**: ≥ 50 (well-tested)
 - **Theme**: Must match the tactical pattern
 Only the top 20 puzzles (by popularity) are selected for each pattern.
 ## File Structure
 ```
 chess_tutor/
 ├── scripts/
 │   ├── setup_tactical_puzzles.py    # Main setup script
 │   └── README.md                     # This file
 ├── downloads/                        # Created by script
 │   ├── lichess_db_puzzle.csv.zst    # Downloaded database (cached)
 │   └── lichess_db_puzzle.csv        # Decompressed database (cached)
 ├── fixtures/
 │   └── tactics/                      # Created by script
 │       ├── pin.json                  # 20 PIN puzzles
 │       ├── fork.json                 # 20 FORK puzzles
 │       ├── skewer.json               # 20 SKEWER puzzles
 │       └── ...                       # Other patterns
 └── .tactical_puzzles_configured      # Marker file (created by script)
 ```
 ## Re-running Setup
 If you want to re-run the setup (e.g., to get fresh puzzles):
 ```bash
 # Option 1: Delete the marker file and re-run
 rm chess_tutor/.tactical_puzzles_configured
 python3 scripts/setup_tactical_puzzles.py
 # Option 2: The script will ask if you want to re-run
 python3 scripts/setup_tactical_puzzles.py
 # Answer 'y' when prompted
 ```
 ## Cleaning Up
 To save disk space after setup:
 ```bash
 # Delete the downloaded database (keeps the fixtures)
 rm -rf chess_tutor/downloads/
 # The fixtures in chess_tutor/fixtures/tactics/ will remain
 ```
 ## Troubleshooting
 ### "zstd not found"
 The script will attempt to auto-install `zstd`. If it fails:
 **macOS:**
 ```bash
 brew install zstd
 ```
 **Ubuntu/Debian:**
 ```bash
 sudo apt-get install zstd
 ```
 **Fedora/RHEL:**
 ```bash
 sudo dnf install zstd
 ```
 ### "python-chess not found"
 The script will attempt to auto-install `python-chess`. If it fails:
 ```bash
 pip3 install python-chess
 ```
 ### "Download failed"
 Check your internet connection and try again. The Lichess database is updated daily, so temporary issues may occur.
 ### "No puzzles found for pattern X"
 This is rare but can happen if the database doesn't have enough puzzles matching the criteria. The script will warn you but continue with other patterns.
 ## Manual Setup (Alternative)
 If the automatic script doesn't work, you can manually:
 1. Download: https://database.lichess.org/lichess_db_puzzle.csv.zst
 2. Decompress with `zstd -d lichess_db_puzzle.csv.zst`
 3. Filter puzzles using the Python code in `docs/LICHESS_PUZZLES.md`
 4. Convert to JSON format and save to `fixtures/tactics/`
 ## Source
 - **Database**: https://database.lichess.org/
 - **License**: Creative Commons CC0 (public domain)
 - **Documentation**: See `docs/LICHESS_PUZZLES.md` for detailed information
 ## Support
 If you encounter issues:
 1. Check the terminal output for specific error messages
 2. Ensure you have Python 3.7+ installed: `python3 --version`
 3. Ensure you have internet connectivity
 4. Try re-running the script
 5. Check `docs/LICHESS_PUZZLES.md` for manual setup instructions
 ---
 # Wikipedia Cache Builder
 ## Overview
 Downloads full Wikipedia articles for all chess opening families and caches them locally for offline use.
 ## Quick Start
 ```bash
 # Install dependencies (if not already done)
 npm install
 # Run the Wikipedia cache builder
 npm run cache:wikipedia
 ```
 ## Benefits
 - ✅ **Offline access** - Works completely offline once cached
 - ✅ **Instant loading** - No network delay
 - ✅ **No rate limiting** - Avoids Wikipedia API rate limits
 - ✅ **Full content** - Gets complete articles, not just summaries
 - ✅ **Version controlled** - Safe to commit to git
 ## What It Does
 1. **Extracts opening families** from your opening database (`public/openings/*.json`)
 2. **Searches Wikipedia** for each family using the OpenSearch API
 3. **Downloads full article content** including:
   - Introduction
   - Main sections (History, Ideas, Variations, etc.)
   - Up to 5 relevant sections per opening
 4. **Saves locally** to `public/wikipedia/*.json`
 5. **Creates an index** at `public/wikipedia/index.json`
 ## Output Structure
 Each opening family gets a JSON file like `public/wikipedia/french-defense.json`:
 ```json
 {
  "openingFamily": "French Defense",
  "title": "French Defence",
  "url": "https://en.wikipedia.org/wiki/French_Defence",
  "sections": [
    {
      "title": "Introduction",
      "text": "The French Defence is a chess opening..."
    },
    {
      "title": "History",
      "text": "The first known game with the French Defense..."
    }
  ],
  "lastModified": "2025-01-15T...",
  "license": "CC BY-SA 3.0",
  "licenseUrl": "https://creativecommons.org/licenses/by-sa/3.0/",
  "fetchedAt": 1705334400000
 }
 ```
 ## Committing to Git
 The generated files are **safe to commit** to your repository:
 - Wikipedia content is licensed under CC BY-SA 3.0 (allows redistribution with attribution)
 - Each file includes proper license information
 - Files are versioned, so you can review changes before committing
 ## When to Run
 - **Initial setup**: Run once to cache all Wikipedia articles
 - **After adding new openings**: Re-run to fetch articles for new families
 - **Periodically**: Re-run every few months to get updated Wikipedia content
 ## Rate Limiting
 The script includes a 1-second delay between requests to be respectful to Wikipedia's servers.
 **Estimated time:**
 - 50 opening families = ~1-2 minutes
 - 100 opening families = ~2-3 minutes
 ## Fallback Strategy
 The app uses a 3-tier approach for Wikipedia content:
 1. **Local cache** (`public/wikipedia/*.json`) - Fastest, always available
 2. **localStorage cache** - In-browser cache for API fetches  
 3. **Live API** - Fallback if local cache missing
 This ensures Wikipedia content is always available, even for openings not in your local cache.
 ## License & Attribution
 Wikipedia content is licensed under **CC BY-SA 3.0**:
 - ✅ Commercial use allowed
 - ✅ Modification allowed
 - ✅ Redistribution allowed
 - ⚠️ Attribution required (included in JSON files)
 - ⚠️ Derivative works must use same license
 More info: https://creativecommons.org/licenses/by-sa/3.0/
 ## Implementation Details
 **Script location:** `scripts/fetch-wikipedia-openings.ts`
 **Dependencies:** 
 - Node.js 18+
 - tsx (TypeScript executor)
 **API endpoints used:**
 - Wikipedia OpenSearch API (for finding articles)
 - Wikipedia MediaWiki Parse API (for full content)
 **Output directory:** `public/wikipedia/`
 ## Example Usage
 ```bash
 # Run the cache builder
 npm run cache:wikipedia
 # Output:
 # 🌐 Wikipedia Opening Cache Builder
 # 📚 Extracting opening families from database...
 # ✓ Found 47 unique opening families
 #
 # 📖 Processing: French Defense
 #   Searching Wikipedia for: "French Defense"
 #   ✓ Found: "French Defence"
 #   Fetching full article...
 #   ✓ Fetched 5 sections
 #   ✓ Saved to: french-defense.json
 # ...
 # ✨ Wikipedia Cache Build Complete!
 # ✓ Successful: 45
 # ⚠️  Skipped:    2
 # ❌ Failed:     0
 ```
@@ -0,0 +1,96 @@
 #!/bin/bash
 #
 # Helper script to download Lichess tactical puzzles in Docker container
 #
 # Usage:
 #   ./scripts/docker-setup-puzzles.sh [options]
 #
 # Options:
 #   --max-puzzles NUM    Number of puzzles per pattern (default: 100)
 #   --force              Force re-download even if already configured
 #   --container NAME     Container name (default: chess-tutor)
 #
 set -e
 # Default values
 CONTAINER_NAME="chess-tutor"
 MAX_PUZZLES=100
 FORCE_FLAG=""
 # Parse arguments
 while [[ $# -gt 0 ]]; do
  case $1 in
    --max-puzzles)
      MAX_PUZZLES="$2"
      shift 2
      ;;
    --force)
      FORCE_FLAG="--force"
      shift
      ;;
    --container)
      CONTAINER_NAME="$2"
      shift 2
      ;;
    *)
      echo "Unknown option: $1"
      echo "Usage: $0 [--max-puzzles NUM] [--force] [--container NAME]"
      exit 1
      ;;
  esac
 done
 echo "=========================================="
 echo "Chess Tutor - Puzzle Setup (Docker)"
 echo "=========================================="
 echo "Container: $CONTAINER_NAME"
 echo "Puzzles per pattern: $MAX_PUZZLES"
 echo "=========================================="
 echo ""
 # Check if container is running
 if ! docker ps --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then
  echo "❌ Error: Container '$CONTAINER_NAME' is not running"
  echo ""
  echo "Start it with: docker-compose up -d"
  exit 1
 fi
 echo "✅ Container is running"
 echo ""
 # Check if already configured (unless --force)
 if [ -z "$FORCE_FLAG" ]; then
  if docker exec "$CONTAINER_NAME" test -f .tactical_puzzles_configured 2>/dev/null; then
    echo "⚠️  Puzzles are already configured!"
    echo ""
    read -p "Do you want to re-download? (y/N): " -n 1 -r
    echo
    if [[ ! $REPLY =~ ^[Yy]$ ]]; then
      echo "Exiting..."
      exit 0
    fi
    FORCE_FLAG="--force"
  fi
 fi
 # Run the setup script inside the container
 echo "🎯 Starting puzzle download..."
 echo "This will download ~500MB and may take 5-10 minutes"
 echo ""
 docker exec -it "$CONTAINER_NAME" \
  python3 scripts/setup_tactical_puzzles.py --max-puzzles "$MAX_PUZZLES" $FORCE_FLAG
 echo ""
 echo "=========================================="
 echo "✅ Puzzle setup complete!"
 echo "=========================================="
 echo ""
 echo "The puzzles are now available in your Chess Tutor app."
 echo "They will persist across container restarts."
 echo ""
 echo "To check the downloaded puzzles:"
 echo "  docker exec $CONTAINER_NAME ls -lh fixtures/tactics/"
 echo ""
@@ -0,0 +1,66 @@
 #!/usr/bin/env node
 const fs = require('fs');
 const path = require('path');
 const ecoFiles = ['ecoA', 'ecoB', 'ecoC', 'ecoD', 'ecoE'];
 const missingWiki = [];
 for (const file of ecoFiles) {
  const filePath = path.join(__dirname, '..', 'public', 'openings', `${file}.json`);
  const data = JSON.parse(fs.readFileSync(filePath, 'utf8'));
  for (const [fen, opening] of Object.entries(data)) {
    if (opening.isEcoRoot && !opening.wikipediaSlug) {
      missingWiki.push({
        eco: opening.eco,
        name: opening.name,
        moves: opening.moves
      });
    }
  }
 }
 // Sort by ECO code
 missingWiki.sort((a, b) => a.eco.localeCompare(b.eco));
 console.log('Major openings (ECO roots) WITHOUT Wikipedia slugs:\n');
 console.log('Total:', missingWiki.length, '\n');
 // Group by ECO category
 const byCategory = {};
 for (const opening of missingWiki) {
  const category = opening.eco[0];
  if (!byCategory[category]) byCategory[category] = [];
  byCategory[category].push(opening);
 }
 for (const [cat, openings] of Object.entries(byCategory).sort()) {
  console.log(`\n=== ECO ${cat} (${openings.length} openings) ===`);
  openings.slice(0, 15).forEach(o => {
    console.log(`${o.eco.padEnd(4)} ${o.name}`);
  });
  if (openings.length > 15) {
    console.log(`... and ${openings.length - 15} more`);
  }
 }
 // Show some notable ones
 console.log('\n\n=== Notable Missing Openings ===');
 const notable = [
  'Sicilian', 'French', 'Caro-Kann', 'Pirc', 'Alekhine',
  'Scandinavian', 'Queen', 'English', 'Indian', 'Benoni',
  'Nimzo', 'Grunfeld', 'Dutch', 'Reti', 'Bird'
 ];
 const notableMatches = missingWiki.filter(o =>
  notable.some(n => o.name.toLowerCase().includes(n.toLowerCase()))
 );
 if (notableMatches.length > 0) {
  notableMatches.forEach(o => {
    console.log(`${o.eco.padEnd(4)} ${o.name}`);
  });
 } else {
  console.log('(None found - all major openings have Wikipedia slugs!)');
 }
@@ -0,0 +1,392 @@
 #!/usr/bin/env python3
 """
 One-time setup script to download and configure high-quality tactical puzzles from Lichess.
 This script:
 1. Downloads the Lichess puzzle database
 2. Extracts puzzles for each tactical pattern
 3. Converts them to our JSON fixture format
 4. Validates them with our tactical library
 5. Creates a marker file to indicate setup is complete
 Usage:
    python3 scripts/setup_tactical_puzzles.py
 """
 import os
 import sys
 import json
 import csv
 import subprocess
 import urllib.request
 from pathlib import Path
 from typing import List, Dict, Any
 # Add parent directory to path to import chess libraries
 sys.path.insert(0, str(Path(__file__).parent.parent))
 try:
    import chess
    import chess.pgn
 except ImportError:
    print("❌ Error: python-chess library not found.")
    print("Installing python-chess...")
    subprocess.check_call([sys.executable, "-m", "pip", "install", "python-chess"])
    import chess
    import chess.pgn
 # Configuration
 LICHESS_PUZZLE_URL = "https://database.lichess.org/lichess_db_puzzle.csv.zst"
 DOWNLOAD_DIR = Path(__file__).parent.parent / "downloads"
 FIXTURES_DIR = Path(__file__).parent.parent / "fixtures" / "tactics"
 SETUP_MARKER = Path(__file__).parent.parent / ".tactical_puzzles_configured"
 # Mapping of our patterns to Lichess themes
 PATTERN_THEMES = {
    "pin": ["pin"],
    "fork": ["fork"],
    "skewer": ["skewer"],
    "discovered_check": ["discoveredAttack"],
    "double_attack": ["doubleCheck", "fork"],
    "overloading": ["overloading"],
    "back_rank_weakness": ["backRankMate"],
    "trapped_piece": ["trappedPiece"],
 }
 # Quality criteria
 MIN_POPULARITY = 50
 MIN_RATING = 800   # Easy puzzles start here
 MAX_RATING = 2200  # Hard puzzles go up to here
 MIN_PLAYS = 50
 # Default number of puzzles per pattern (can be overridden via command line)
 DEFAULT_PUZZLES_PER_PATTERN = 20
 def check_zstd_installed() -> bool:
    """Check if zstd is installed for decompression."""
    try:
        subprocess.run(["zstd", "--version"], capture_output=True, check=True)
        return True
    except (subprocess.CalledProcessError, FileNotFoundError):
        return False
 def install_zstd():
    """Attempt to install zstd."""
    print("📦 Installing zstd...")
    # Detect OS and install accordingly
    if sys.platform == "darwin":  # macOS
        try:
            subprocess.check_call(["brew", "install", "zstd"])
            print("✅ zstd installed successfully via Homebrew")
        except (subprocess.CalledProcessError, FileNotFoundError):
            print("❌ Failed to install zstd. Please install Homebrew first:")
            print("   /bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"")
            sys.exit(1)
    elif sys.platform.startswith("linux"):
        try:
            # Try apt-get (Debian/Ubuntu)
            subprocess.check_call(["sudo", "apt-get", "update"])
            subprocess.check_call(["sudo", "apt-get", "install", "-y", "zstd"])
            print("✅ zstd installed successfully via apt-get")
        except (subprocess.CalledProcessError, FileNotFoundError):
            print("❌ Failed to install zstd. Please install it manually:")
            print("   Debian/Ubuntu: sudo apt-get install zstd")
            print("   Fedora/RHEL: sudo dnf install zstd")
            sys.exit(1)
    else:
        print("❌ Unsupported OS. Please install zstd manually:")
        print("   Windows: Download from https://github.com/facebook/zstd/releases")
        sys.exit(1)
 def download_puzzle_database() -> Path:
    """Download the Lichess puzzle database."""
    DOWNLOAD_DIR.mkdir(parents=True, exist_ok=True)
    compressed_file = DOWNLOAD_DIR / "lichess_db_puzzle.csv.zst"
    decompressed_file = DOWNLOAD_DIR / "lichess_db_puzzle.csv"
    # Check if already downloaded and decompressed
    if decompressed_file.exists():
        print(f"✅ Puzzle database already exists at {decompressed_file}")
        return decompressed_file
    # Download if not exists
    if not compressed_file.exists():
        print(f"📥 Downloading Lichess puzzle database from {LICHESS_PUZZLE_URL}")
        print("   This may take several minutes (file is ~500MB compressed)...")
        try:
            urllib.request.urlretrieve(LICHESS_PUZZLE_URL, compressed_file)
            print(f"✅ Downloaded to {compressed_file}")
        except Exception as e:
            print(f"❌ Failed to download: {e}")
            sys.exit(1)
    # Decompress
    print(f"📦 Decompressing {compressed_file.name}...")
    print("   This may take several minutes (decompressed file is ~3.5GB)...")
    try:
        subprocess.check_call(["zstd", "-d", str(compressed_file), "-o", str(decompressed_file)])
        print(f"✅ Decompressed to {decompressed_file}")
    except subprocess.CalledProcessError as e:
        print(f"❌ Failed to decompress: {e}")
        sys.exit(1)
    return decompressed_file
 def extract_puzzles_for_pattern(csv_file: Path, pattern: str, themes: List[str], max_puzzles: int) -> List[Dict[str, Any]]:
    """Extract high-quality puzzles for a specific tactical pattern."""
    print(f"🔍 Extracting {pattern.upper()} puzzles (max: {max_puzzles})...")
    puzzles = []
    with open(csv_file, 'r', encoding='utf-8') as f:
        reader = csv.DictReader(f)
        for row in reader:
            puzzle_themes = row['Themes'].split()
            rating = int(row['Rating'])
            popularity = int(row['Popularity'])
            nb_plays = int(row['NbPlays'])
            # Check if puzzle matches our criteria
            if (any(theme in puzzle_themes for theme in themes) and
                popularity >= MIN_POPULARITY and
                MIN_RATING <= rating <= MAX_RATING and
                nb_plays >= MIN_PLAYS):
                puzzles.append(row)
                # Stop when we have enough
                if len(puzzles) >= max_puzzles:
                    break
    # Sort by popularity (best first)
    puzzles.sort(key=lambda x: int(x['Popularity']), reverse=True)
    print(f"   Found {len(puzzles)} high-quality {pattern.upper()} puzzles")
    return puzzles[:max_puzzles]
 def lichess_to_fixture(puzzle_row: Dict[str, Any], pattern_type: str) -> Dict[str, Any]:
    """Convert Lichess puzzle to our fixture format.
    Lichess puzzle format:
    - FEN: Position BEFORE opponent's first move
    - Moves: Space-separated UCI moves alternating opponent/player
    - First move: Opponent's move (sets up the puzzle)
    - Remaining moves: Player move, opponent response, player move, etc.
    """
    fen = puzzle_row['FEN']
    moves_uci = puzzle_row['Moves'].split()
    # Apply first move (opponent's move) to get starting position
    board = chess.Board(fen)
    opponent_move = chess.Move.from_uci(moves_uci[0])
    board.push(opponent_move)
    initial_fen = board.fen()
    # Determine side to move from initial FEN
    side_to_move = "white" if " w " in initial_fen else "black"
    # Process all moves in the sequence
    # moves_uci[0] = opponent's setup move (already applied)
    # moves_uci[1] = player's first move (solution start)
    # moves_uci[2] = opponent's response
    # moves_uci[3] = player's second move
    # etc.
    move_sequence = []
    for i, move_uci in enumerate(moves_uci[1:], start=1):  # Skip first move (already applied)
        move = chess.Move.from_uci(move_uci)
        move_san = board.san(move)
        # Determine who makes this move
        # Odd indices (1, 3, 5...) = player moves
        # Even indices (2, 4, 6...) = opponent moves
        is_player_move = (i % 2 == 1)
        move_sequence.append({
            "uci": move_uci,
            "san": move_san,
            "player": is_player_move
        })
        board.push(move)
    # Get final position after all moves
    resulting_fen = board.fen()
    # First player move (for backward compatibility)
    first_player_move_uci = moves_uci[1]
    board_temp = chess.Board(initial_fen)
    first_player_move = chess.Move.from_uci(first_player_move_uci)
    first_player_move_san = board_temp.san(first_player_move)
    return {
        "id": puzzle_row['PuzzleId'],
        "initialFen": initial_fen,
        "sideToMove": side_to_move,
        "rating": int(puzzle_row['Rating']),  # Add rating for difficulty filtering
        "bestMove": {
            "san": first_player_move_san,
            "uci": first_player_move_uci
        },
        "moves": move_sequence,  # Full move sequence
        "resultingFen": resulting_fen,
        "expectedPattern": {
            "type": pattern_type.upper().replace("_", "_"),
            # Note: Exact squares will be detected by our tactical library
        },
        "context": f"Lichess puzzle {puzzle_row['PuzzleId']} (Rating: {puzzle_row['Rating']}, Popularity: {puzzle_row['Popularity']})",
        "tags": puzzle_row['Themes'].split()
    }
 def save_fixtures(pattern: str, fixtures: List[Dict[str, Any]]):
    """Save fixtures to JSON file."""
    FIXTURES_DIR.mkdir(parents=True, exist_ok=True)
    output_file = FIXTURES_DIR / f"{pattern}.json"
    fixture_data = {
        "description": f"High-quality {pattern.upper()} tactical puzzles from Lichess database",
        "source": "https://database.lichess.org/",
        "generatedAt": "auto-generated",
        "cases": fixtures
    }
    with open(output_file, 'w', encoding='utf-8') as f:
        json.dump(fixture_data, f, indent=2, ensure_ascii=False)
    print(f"✅ Saved {len(fixtures)} puzzles to {output_file}")
 def create_setup_marker():
    """Create a marker file to indicate setup is complete."""
    with open(SETUP_MARKER, 'w') as f:
        f.write("Tactical puzzles configured successfully\n")
    print(f"✅ Created setup marker at {SETUP_MARKER}")
    print("   (The app will auto-detect Lichess puzzles from fixture metadata)")
 def main():
    """Main setup function."""
    import argparse
    parser = argparse.ArgumentParser(
        description='Download and configure high-quality tactical puzzles from Lichess',
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
 Examples:
  # Download 20 puzzles per pattern (default)
  python3 scripts/setup_tactical_puzzles.py
  # Download 100 puzzles per pattern
  python3 scripts/setup_tactical_puzzles.py --max-puzzles 100
  # Download 500 puzzles per pattern for production
  python3 scripts/setup_tactical_puzzles.py --max-puzzles 500
        """
    )
    parser.add_argument(
        '--max-puzzles',
        type=int,
        default=DEFAULT_PUZZLES_PER_PATTERN,
        help=f'Maximum number of puzzles to extract per pattern (default: {DEFAULT_PUZZLES_PER_PATTERN})'
    )
    parser.add_argument(
        '--force',
        action='store_true',
        help='Force re-run setup without prompting'
    )
    args = parser.parse_args()
    print("=" * 70)
    print("🎯 Chess Tutor - Tactical Puzzles Setup")
    print("=" * 70)
    print(f"Configuration: {args.max_puzzles} puzzles per pattern")
    print("=" * 70)
    print()
    # Check if already configured
    if SETUP_MARKER.exists() and not args.force:
        print("⚠️  Tactical puzzles are already configured!")
        response = input("Do you want to re-run the setup? (y/N): ").strip().lower()
        if response != 'y':
            print("Exiting...")
            sys.exit(0)
        else:
            SETUP_MARKER.unlink()
    # Step 1: Check/install zstd
    print("Step 1: Checking dependencies...")
    if not check_zstd_installed():
        print("⚠️  zstd not found (required for decompression)")
        install_zstd()
    else:
        print("✅ zstd is installed")
    print()
    # Step 2: Download and decompress database
    print("Step 2: Downloading Lichess puzzle database...")
    csv_file = download_puzzle_database()
    print()
    # Step 3: Extract puzzles for each pattern
    print("Step 3: Extracting puzzles for each tactical pattern...")
    total_puzzles = 0
    for pattern, themes in PATTERN_THEMES.items():
        puzzles = extract_puzzles_for_pattern(csv_file, pattern, themes, args.max_puzzles)
        if len(puzzles) == 0:
            print(f"⚠️  Warning: No puzzles found for {pattern.upper()}")
            continue
        # Convert to fixture format
        fixtures = []
        for puzzle in puzzles:
            try:
                fixture = lichess_to_fixture(puzzle, pattern)
                fixtures.append(fixture)
            except Exception as e:
                print(f"   ⚠️  Skipping puzzle {puzzle['PuzzleId']}: {e}")
        # Save to file
        if fixtures:
            save_fixtures(pattern, fixtures)
            total_puzzles += len(fixtures)
    print()
    # Step 4: Create marker file
    print("Step 4: Finalizing setup...")
    create_setup_marker()
    print()
    print("=" * 70)
    print(f"✅ Setup complete! {total_puzzles} tactical puzzles are ready to use.")
    print("=" * 70)
    print()
    print("Next steps:")
    print("1. Refresh your Chess Tutor app (if running)")
    print("2. The warning banner will automatically disappear")
    print("   (The app detects Lichess puzzles by checking the fixture metadata)")
    print()
    print("3. Go to http://localhost:3050/learning")
    print("4. Select a coach and practice tactical patterns!")
    print()
    print("Note: The downloaded database is cached in the 'downloads' directory.")
    print("      You can delete it to save space if needed.")
 if __name__ == "__main__":
    main()
@@ -0,0 +1,87 @@
 #!/usr/bin/env node
 const fs = require('fs');
 const path = require('path');
 const ecoFiles = ['ecoA', 'ecoB', 'ecoC', 'ecoD', 'ecoE'];
 const withWiki = [];
 const withoutWiki = [];
 for (const file of ecoFiles) {
  const filePath = path.join(__dirname, '..', 'public', 'openings', `${file}.json`);
  const data = JSON.parse(fs.readFileSync(filePath, 'utf8'));
  for (const [fen, opening] of Object.entries(data)) {
    if (opening.isEcoRoot) {
      if (opening.wikipediaSlug) {
        withWiki.push({
          eco: opening.eco,
          name: opening.name,
          slug: opening.wikipediaSlug
        });
      } else {
        withoutWiki.push({
          eco: opening.eco,
          name: opening.name
        });
      }
    }
  }
 }
 // Sort
 withWiki.sort((a, b) => a.name.localeCompare(b.name));
 withoutWiki.sort((a, b) => a.name.localeCompare(b.name));
 console.log('━'.repeat(70));
 console.log('WIKIPEDIA INTEGRATION SUMMARY');
 console.log('━'.repeat(70));
 console.log();
 console.log(`✅ ECO roots WITH Wikipedia: ${withWiki.length}`);
 console.log(`❌ ECO roots WITHOUT Wikipedia: ${withoutWiki.length}`);
 console.log();
 // Show major openings with Wikipedia
 console.log('━'.repeat(70));
 console.log('MAJOR OPENINGS WITH WIKIPEDIA (sample):');
 console.log('━'.repeat(70));
 const majorOpenings = [
  'Sicilian', 'French', 'Caro-Kann', 'Pirc', 'Alekhine',
  'Scandinavian', 'Italian', 'Spanish', 'Scotch', 'Vienna',
  'English', 'Reti', 'Bird', 'Polish', 'Nimzo', 'Queen',
  'King', 'Benoni', 'Catalan', 'Grunfeld', 'Dutch', 'Ruy Lopez'
 ];
 const foundMajor = withWiki.filter(o =>
  majorOpenings.some(m => o.name.toLowerCase().includes(m.toLowerCase()))
 );
 foundMajor.slice(0, 40).forEach(o => {
  console.log(`  ${o.eco.padEnd(4)} ${o.name}`);
 });
 if (foundMajor.length > 40) {
  console.log(`  ... and ${foundMajor.length - 40} more major openings`);
 }
 console.log();
 console.log('━'.repeat(70));
 console.log('OPENINGS MISSING WIKIPEDIA:');
 console.log('━'.repeat(70));
 if (withoutWiki.length === 0) {
  console.log('  None! All ECO roots have Wikipedia articles.');
 } else {
  withoutWiki.forEach(o => {
    const isVariation = o.name.includes(':');
    const marker = isVariation ? '  └─' : '  ';
    console.log(`${marker} ${o.eco.padEnd(4)} ${o.name}`);
  });
 }
 console.log();
 console.log('━'.repeat(70));
 console.log(`Coverage: ${withWiki.length}/${withWiki.length + withoutWiki.length} ECO roots (${Math.round(withWiki.length / (withWiki.length + withoutWiki.length) * 100)}%)`);
 console.log('━'.repeat(70));