# doi_search

**Repository Path**: joneyhsiao/doi_search

## Basic Information

- **Project Name**: doi_search
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-03-18
- **Last Updated**: 2025-03-19

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# DOI Search Automation

This script automates the process of searching for DOIs on the Chinese DOI system and retrieving information from various Chinese academic resources.

## Features

- Process multiple DOIs in batch mode
- Take screenshots at each step of the process
- Handle redirections to various Chinese academic platforms (Wanfang, CNKI, etc.)
- Save detailed results in JSON, HTML, and text formats
- Handle human verification (CAPTCHA) prompts
- **NEW**: Automatically solve slider-based verification challenges

## Requirements

- Python 3.7+
- Playwright
- Internet connection

## Installation

1. Install the required packages:
   ```
   pip install -r requirements.txt
   ```

2. Install Playwright browsers:
   ```
   playwright install
   ```

## Usage

1. Create a file named `doi_values.txt` with one DOI per line.

2. Run the script:
   ```
   python search_doi.py
   ```

### Command-line Options

- `--limit N`: Process only N DOIs (default: process all)
- `--start N`: Start from the Nth DOI in the list (0-based index)
- `--headless`: Run in headless mode (no browser UI)
- `--slow_mo N`: Slow down operations by N milliseconds
- `--debug`: Enable debug mode
- `--full_page`: Take full-page screenshots
- `--width N`: Set browser viewport width (default: 1280)
- `--height N`: Set browser viewport height (default: 800)
- `--verification_timeout N`: Maximum time to wait for human verification in seconds (default: 300)
- `--wait_between N`: Wait time in seconds between DOI requests (default: 3)
- `--auto_verify`: Enable automatic verification for supported challenge types

## Handling Human Verification

When accessing certain Chinese academic resources, human verification (CAPTCHA) is sometimes required. The script can:

1. Automatically solve slider-based verification challenges when using the `--auto_verify` flag
2. Detect verification screens automatically 
3. Take a screenshot of the verification page
4. Pause and wait for human intervention (if automatic solving fails or isn't enabled)
5. Monitor for verification completion
6. Continue processing once verification is completed

### Automatic Verification

The script can automatically solve certain types of verification challenges:

- **Slider-based verification**: The script can simulate human-like mouse movements to slide the verification slider to the correct position
- **Image CAPTCHA**: Currently requires human intervention
- **Click verification**: Currently requires human intervention

To enable automatic verification:

```
python search_doi.py --auto_verify
```

### Manual Verification Fallback

If automatic verification fails or isn't enabled:

1. Run the script in non-headless mode (`--headless` flag omitted)
2. When verification is required, the browser window will show the verification screen
3. Complete the verification manually in the browser
4. The script will automatically detect completion and continue processing
5. Adjust the verification timeout with `--verification_timeout` if needed

## Output

Results are saved to the `doi_results` directory with the following structure:

- `{DOI}/` - Directory for each DOI (with / replaced by _)
  - `1_initial.png` - Screenshot of initial state
  - `2_input_clicked.png` - Screenshot after clicking input
  - `3_doi_entered.png` - Screenshot after entering DOI
  - `4_results.png` - Screenshot of results page
  - `5_*` - Additional screenshots based on redirection
  - `verification_required.png` - Screenshot of verification screen (if encountered)
  - `verification_automation_attempt.png` - Screenshot of automation attempt (if --auto_verify)
  - `verification_automation_completed.png` - Screenshot after automation attempt
  - `verification_automation_success.png` - Screenshot if automation succeeds
  - `verification_completed.png` - Screenshot after verification completed
- `{DOI}.json` - JSON data with results
- `{DOI}.html` - HTML content of the final page
- `{DOI}.txt` - Plain text results
- `{DOI}_result.png` - Final screenshot
- `errors.log` - Log of any errors
- `process_log.txt` - Processing log

## Examples

Process all DOIs:
```
python search_doi.py
```

Process only the first 5 DOIs:
```
python search_doi.py --limit 5
```

Process all DOIs with automatic verification:
```
python search_doi.py --auto_verify
```

Start from the 10th DOI and process 20 DOIs with automatic verification:
```
python search_doi.py --start 10 --limit 20 --auto_verify
```

Process all DOIs with longer wait between requests:
```
python search_doi.py --wait_between 10
```

Customize browser viewport and take full-page screenshots:
```
python search_doi.py --width 1920 --height 1080 --full_page
```