Google Play Store Review Scraper

Executive Summary

This project demonstrates a production-ready web scraper for collecting app reviews from the Google Play Store. The implementation showcases the evolution from a basic script to a robust, maintainable solution using modern Python best practices.

---

The Challenge

Collecting app reviews at scale from Google Play Store presents several challenges:

• Dynamic Content: Reviews are loaded via JavaScript (lazy loading)
• Pagination: “Load More” button reveals additional reviews
• Rate Limiting: Aggressive scraping can trigger blocks
• Data Quality: Reviews may be truncated (short vs. full versions)

---

Before: The Original Approach

The initial implementation had several issues common in quick prototypes:

# Hardcoded paths and URLs
PATH = "C:\\Program Files (x86)\\chromedriver.exe"
driver = webdriver.Chrome(PATH)

# User input instead of CLI args
namefile = input('Input file name to export data : ')
count = input("Amount of data : ")

# Deprecated Selenium API
button = driver.find_element_by_css_selector('span.CwaK9')

# Duplicated logic
def dataload():
    try:
        for i in range(50):
            button = driver.find_element_by_css_selector('span.CwaK9')
            # ... same code repeated in except block
    except:
        try:
            # ... duplicated logic here
        except:
            scrolldown()

Problems identified:

1. ❌ Hardcoded ChromeDriver path (breaks on different systems)
2. ❌ Deprecated find_element_by_* methods
3. ❌ Repetitive code blocks
4. ❌ Bare except clauses (catches everything, including KeyboardInterrupt)
5. ❌ No CLI arguments (requires interactive input)
6. ❌ No logging or proper progress tracking

---

After: The Optimized Solution

1. Configuration with Dataclasses

@dataclass
class ScraperConfig:
    """Configuration for the scraper."""
    app_id: str
    output_file: str
    target_count: int = 1000
    language: str = "id"
    headless: bool = False
    scroll_pause: float = 2.0
    max_retries: int = 3
    backup_interval: int = 500

2. Modern Selenium 4+ API

# Before (deprecated)
driver.find_element_by_css_selector('span.CwaK9')

# After (modern)
from selenium.webdriver.common.by import By
driver.find_element(By.CSS_SELECTOR, 'span.CwaK9')

3. Centralized Selectors

class PlayStoreReviewScraper:
    SELECTORS = {
        "review_container": "div.LXrl4c",
        "review_item": "div.UD7Dzf",
        "short_comment": 'span[jsname="bN97Pc"]',
        "long_comment": 'span[jsname="fbQN7e"]',
        "load_more_button": "span.CwaK9",
    }

4. Robust Error Handling

from selenium.common.exceptions import (
    NoSuchElementException,
    TimeoutException,
    ElementClickInterceptedException,
)

def _click_load_more(self) -> bool:
    try:
        button = WebDriverWait(self.driver, 5).until(
            EC.element_to_be_clickable(
                (By.CSS_SELECTOR, self.SELECTORS["load_more_button"])
            )
        )
        button.click()
        return True
    except (NoSuchElementException, TimeoutException, ElementClickInterceptedException):
        return False

5. CLI Interface with argparse

# Now runs with simple command-line arguments
python webscrape.py --app-id com.gojek.app --output reviews.csv --count 1000 --headless

---

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    PlayStoreReviewScraper                   │
├─────────────────────────────────────────────────────────────┤
│  ScraperConfig (dataclass)                                  │
│  ├── app_id: str                                            │
│  ├── output_file: str                                       │
│  ├── target_count: int                                      │
│  ├── language: str                                          │
│  └── headless: bool                                         │
├─────────────────────────────────────────────────────────────┤
│  Core Methods                                               │
│  ├── scrape() → Main loop                                   │
│  ├── _load_more_reviews() → Pagination handling             │
│  ├── _extract_review_text() → Text extraction               │
│  ├── _export_reviews() → CSV export                         │
│  └── _create_backup() → Automatic backups                   │
└─────────────────────────────────────────────────────────────┘

---

Key Improvements Summary

Aspect	Before	After
Configuration	Hardcoded values	Dataclass + CLI args
Selenium API	Deprecated methods	Modern By locators
Error Handling	Bare except	Specific exceptions
Code Structure	Procedural script	OOP with single responsibility
Logging	Print statements	Python logging module
Progress Tracking	Manual prints	Structured progress updates
Backups	Manual	Automatic interval-based
Portability	Windows-only path	webdriver-manager fallback

---

Usage

# Install dependencies
pip install selenium pandas webdriver-manager

# Basic usage
python webscrape.py --app-id com.gojek.app --count 500

# Full options
python webscrape.py \
    --app-id com.tokopedia.tkpd \
    --output tokopedia_reviews.csv \
    --count 2000 \
    --language en \
    --headless

Conclusion

This refactoring demonstrates how a quick prototype can be transformed into a maintainable, production-ready tool. The key principles applied were:

1. Single Responsibility: Each method does one thing well
2. Configuration as Data: Dataclasses make settings explicit
3. Explicit Error Handling: Catch only what you expect
4. CLI Over Input: Scripts should be automatable
5. Progressive Enhancement: Backups and logging for reliability