Automated Global Sports Data Extraction & Historical Database Backfill System

======================================================================== TECHNICAL REQUIREMENTS SPECIFICATION & MILESTONE AGREEMENT ======================================================================== Project Name: Phase 1 Sports Data Extraction & Backfill Total Budget: $100 USD (Includes 1-Month Free Technical Maintenance Warranty) 1. SYSTEM ARCHITECTURE: DUAL-SCRIPT MODULAR DESIGN To optimize system stability, simplify debugging, and facilitate modular maintenance, the software must be developed as TWO completely independent executable scripts: - Script A ([login to view URL]): Daily Match Opening Odds Ingestion Module. - Script B ([login to view URL]): 5-Year Historical Whitelist Database Backfill Module. 2. GLOBAL LEAGUE WHITELIST FILTER CONFIGURATION Both scripts must enforce the following league whitelist. The crawler MUST check the league name first. Only matches belonging to these precise leagues shall be processed; all unspecified or minor leagues must be skipped entirely: TARGET_LEAGUES = [ # ── EUROPE ── '英超', '西甲', '意甲', '德甲', '法甲', # Big 5 '荷甲', '葡超', '土超', '比甲', '苏超', # Mainstream Europe '瑞典超', '挪超', '丹超', '瑞士超', '奥甲', '希超', '捷甲', '波兰甲', '克甲', '俄超', '欧冠', '欧联杯', '欧协联', # UEFA Tournaments # ── AMERICA ── '美职联', '墨超', # North America '巴甲', '阿甲', '哥伦甲', '智利甲', '厄甲', '秘鲁甲', '巴拉甲', '乌拉甲', # South America '解放者杯', '南美杯', '美职联杯', '美金杯', # Cups # ── ASIA & OCEANIA ── '日职联', '日职乙', '天皇杯', '日联杯', # Japan Core '韩职联', '韩足总', # Korea Core '中超', '澳超', '沙特联', '卡塔尔联', '阿联酋超', # Other Asia '亚冠', '亚洲杯' # Asia Tournaments ] 3. DETAILED EXECUTION LOGIC & DATA SCHEMA 3.1 Script A: Daily Match Scraping - Target: Scan the daily fixtures/live matches list page for the current local date. - Frequency: This data represents static opening lines. Scrape ONCE per match when the market opens (or via a few timed intervals per day). No high-frequency real-time updates or WebSocket reversing required. - Tab Switching: Must switch and extract data from both [全场 (Full-Time)] and [半场 (Half-Time)] tabs. - Target Columns: Structurally extract the 3 "Opening" columns (Asian Handicap, 1X2 Match Odds, Over/Under Total Goals) for ALL 9 major bookmakers (Crown, Bet365, Ladbrokes, William Hill, Macauslot, Easybet, Victor Chandler, Mansion88, Interwetten). - Storage: Automatically export into a structured text/CSV file on the local machine, saved using the current date as the filename (e.g., [login to view URL]). 3.2 Script B: 5-Year Historical Backfill - Time Depth: Comprehensive backfill spanning the past 5 full seasons (specifically from the 2021/2022 season up to the current 2025/2026 season). - Execution Path: Loop through Season ➔ Round ➔ Match Result Page ➔ Extract 9-Bookmaker Full/Half Time Opening Lines. - Target Fields: Historical Final Scores (both Full-Time & Half-Time) must be perfectly mapped to their respective 9-bookmaker opening odds matrices. - Storage Path: All historical datasets must be structured and saved automatically directly to my local computer's D: Drive (e.g., D:\SportsData_History\). 4. CORE FAIL-SAFE & SAFETY MECHANISMS - Resume from Breakpoint (Checkpoint): The historical script MUST implement a local state-saving mechanism (e.g., [login to view URL]) written to the D: Drive. If interrupted by network drops, manual termination, or PC reboots, re-running the script must seamlessly resume from the last saved state without duplication or starting from scratch. - Polite Scraping Delay: Enforce a random sleep delay of `[login to view URL]([login to view URL](1.0, 3.0))` between match page requests to guarantee a 100% stable, ban-free operation on local residential IPs. 5. MILESTONE PLATFORM DISTRIBUTION & ACCEPTANCE CRITERIA MILESTONE 1: Initial Local Setup & Verification ($40 USD) - Deliverables: Transfer complete Python source code with clear comments (no encrypted wrappers or compiled .exe files) and a step-by-step local runtime configuration guide. - Criteria: The script successfully boots on my PC and correctly writes the first structured test batch into my D: Drive path. MILESTONE 2: 24-Hour Stability & Integrity Test ($60 USD) - Criteria: 1. The scraper runs continuously for 24 hours on my machine without any script crashes, memory leaks, or local IP bans. 2. The breakpoint resume system passes manual interruption testing (intentionally disconnecting the network/killing the process and verifying seamless recovery). 3. Full-volume 5-year historical data for whitelisted leagues is completely populated into the D: Drive. 1-MONTH FREE MAINTENANCE WARRANTY - Requirement: Provide 1 month of free technical support from the final payout date. If the target site updates its layout or selectors causing script failures during this month, patches must be applied within 24 hours for free.

Project ID: 40467708