Required stack
필요 기술
PHPPythonWeb ScrapingMySQLLaravelData ExtractionSeleniumPlaywrightPuppeteerAPI IntegrationNext.jsFastAPI
Project brief
프로젝트 내용
We are looking for an experienced web scraping developer to build a robust and scalable scraping system for extracting product data from G2A and integrating it into our Laravel-based platform.
We already have an integration with the G2A API. However, some critical data is NOT available or NOT accurate via the API, which is why scraping is required.
---
Project Scope:
* Scrape and process data for 240,000+ products
* Integrate scraping output with an existing Laravel backend
* Ensure data consistency and correct mapping with existing API data
---
Important Clarification (API vs Scraping):
* We already receive the following from G2A API:
* Product ID (same as G2A)
* Slug (same as G2A)
* Price, currency
* Images
* Platform, region
* System requirements
* However:
* Categories from the API are NOT accurate and do not match G2A website structure
* Some important content (description sections, alert boxes, activation guides) is NOT available via API
Therefore, scraping is required to:
* Rebuild the correct categories structure exactly as on G2A
* Extract missing product content
All scraped data must be correctly matched with API data using the same product ID and slug.
---
Core Requirements:
1. Categories Mapping (Very Important)
* Extract the full categories structure exactly as on G2A website
* Include main categories, subcategories, and full hierarchy
* Replace API categories completely
* Each product must be mapped accurately to its category path
2. Product Data Extraction
* Basic product information
* Full product description (HTML)
* Yellow alert / warning box under “About this item” (must be extracted بالكامل with formatting)
* Activation guide (especially for software products)
3. Data Alignment
* Must correctly match scraped data with API data (using product ID and slug)
* No duplicates or mismatched records
---
Technical Requirements:
* Python (preferred with FastAPI or similar)
* Experience with headless browsers (Playwright / Selenium / Puppeteer)
* Ability to handle dynamic content and anti-bot protections
* Experience with scalable scraping (parallel workers, batching, queues)
* Strong error handling, retry logic, and logging
---
Infrastructure:
* Scraper will run on a dedicated VPS (8 CPU / 32GB RAM)
* Must support parallel execution
* Must not affect the main Laravel application
---
Milestones:
1. Initial Test (Mandatory)
* Scrape 50 varied products (games, software, gift cards)
* Display results on the live website
* Validate:
* Data accuracy
* Categories mapping (must match G2A exactly)
* Alert box extraction
* Activation guide
2. Scaling Phase
* Gradual scaling after validation
* Full scraping of 240K+ products
---
Payment Terms:
* No upfront payment without results
* Payment after successful delivery of the initial test (50 products)
* Further payments based on validated milestones
---
To Apply:
Please include:
* Examples of similar scraping projects (especially large-scale)
* Technologies used
* Your approach for handling this project (short explanation)
---
We are looking for someone who can deliver real, scalable results — not partial implementations.
We already have an integration with the G2A API. However, some critical data is NOT available or NOT accurate via the API, which is why scraping is required.
---
Project Scope:
* Scrape and process data for 240,000+ products
* Integrate scraping output with an existing Laravel backend
* Ensure data consistency and correct mapping with existing API data
---
Important Clarification (API vs Scraping):
* We already receive the following from G2A API:
* Product ID (same as G2A)
* Slug (same as G2A)
* Price, currency
* Images
* Platform, region
* System requirements
* However:
* Categories from the API are NOT accurate and do not match G2A website structure
* Some important content (description sections, alert boxes, activation guides) is NOT available via API
Therefore, scraping is required to:
* Rebuild the correct categories structure exactly as on G2A
* Extract missing product content
All scraped data must be correctly matched with API data using the same product ID and slug.
---
Core Requirements:
1. Categories Mapping (Very Important)
* Extract the full categories structure exactly as on G2A website
* Include main categories, subcategories, and full hierarchy
* Replace API categories completely
* Each product must be mapped accurately to its category path
2. Product Data Extraction
* Basic product information
* Full product description (HTML)
* Yellow alert / warning box under “About this item” (must be extracted بالكامل with formatting)
* Activation guide (especially for software products)
3. Data Alignment
* Must correctly match scraped data with API data (using product ID and slug)
* No duplicates or mismatched records
---
Technical Requirements:
* Python (preferred with FastAPI or similar)
* Experience with headless browsers (Playwright / Selenium / Puppeteer)
* Ability to handle dynamic content and anti-bot protections
* Experience with scalable scraping (parallel workers, batching, queues)
* Strong error handling, retry logic, and logging
---
Infrastructure:
* Scraper will run on a dedicated VPS (8 CPU / 32GB RAM)
* Must support parallel execution
* Must not affect the main Laravel application
---
Milestones:
1. Initial Test (Mandatory)
* Scrape 50 varied products (games, software, gift cards)
* Display results on the live website
* Validate:
* Data accuracy
* Categories mapping (must match G2A exactly)
* Alert box extraction
* Activation guide
2. Scaling Phase
* Gradual scaling after validation
* Full scraping of 240K+ products
---
Payment Terms:
* No upfront payment without results
* Payment after successful delivery of the initial test (50 products)
* Further payments based on validated milestones
---
To Apply:
Please include:
* Examples of similar scraping projects (especially large-scale)
* Technologies used
* Your approach for handling this project (short explanation)
---
We are looking for someone who can deliver real, scalable results — not partial implementations.