# stock_select

**Repository Path**: gendai/stock_select

## Basic Information

- **Project Name**: stock_select
- **Description**: Stock selection algorithm.
- **Primary Language**: Python
- **License**: GPL-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2023-11-26
- **Last Updated**: 2026-10-11

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# stock_select

A small quantitative-investment data pipeline for A-share blue chips.

This repo downloads **monthly prices + quarterly financials** for the current CSI300 constituents, reconstructs a historical membership panel, and merges them into a single panel dataset. The goal is to keep the dataset small enough for local quant experiments while covering the most representative large-cap stocks since 2003.

## Project structure

```
stock_select/
├── database/                    # data pipeline (downloads data only)
│   ├── scripts/                  # data download scripts (01~12)
│   ├── data/                     # generated data (gitignored)
│   └── README.md                 # detailed docs: script table, verification, known caveats
├── 组合策略/                     # portfolio strategy
│   ├── 年化10.md                 # strategy document
│   ├── scripts/                  # strategy backtest scripts
│   │   └── portfolio_backtest.py # core+satellite portfolio backtest (reads database data)
│   └── backtest/                 # backtest outputs (gitignored)
│       ├── portfolio_nav.csv     # NAV history
│       ├── trades.csv            # trade log
│       ├── summary.csv           # summary metrics
│       ├── nav_curve.png         # NAV + drawdown chart
│       └── yearly_returns.png    # annual returns chart
└── 定投选股/
    └── hs300.py                  # 定投增强策略回测（读取 database 生成的数据）
```

> `database/data/` is **not** tracked by git; it can be regenerated locally with the scripts.

## Install dependencies

You can install the required packages directly into your global Windows Python, or create one shared virtual environment.

### Option A: install globally (simplest)

```bash
cd database
pip install -r requirements.txt
```

### Option B: one shared virtual environment

If you prefer to avoid version conflicts, create a single shared venv anywhere (for example `D:/venvs/stock_select`) and use it for all runs:

```bash
python -m venv D:/venvs/stock_select
D:/venvs/stock_select/Scripts/python.exe -m pip install -r database/requirements.txt
```

Then activate it before running the scripts:

```bash
D:/venvs/stock_select/Scripts/activate
```

The included `update_db.sh` / `update_db.bat` will also auto-activate a venv placed at `database/.venv` or `stock_select/.venv`.

## How to update the database

### 1. One-click update

In Git Bash:

```bash
cd database
bash update_db.sh
```

Or in Windows Command Prompt:

```cmd
cd database
update_db.bat
```

This runs all five scripts in order:

1. `01_get_membership.py` — fetches current CSI300 constituents and builds a monthly membership table from their join dates.
2. `02_download_prices.py` — downloads daily prices from Sina and resamples to monthly bars (OHLCV + turnover).
3. `03_download_financials.py` — downloads quarterly performance reports from Eastmoney for the CSI300 pool.
4. `04_merge.py` — forward-fills financials to each month-end and merges with prices and membership.
5. `05_validate.py` — writes `data_summary.json` with row counts, date range and missing-value rates.

### 2. Update step by step

You can also run each script manually from the `database/` directory:

```bash
cd database
python scripts/01_get_membership.py
python scripts/02_download_prices.py
python scripts/03_download_financials.py
python scripts/04_merge.py
python scripts/05_validate.py
```

### 3. Resume after interruption

`02_download_prices.py` and `03_download_financials.py` save progress files under `data/processed/_progress_*.json`. If a run is interrupted, rerun the same script and it will continue from where it left off.

## Main output

- **`database/data/processed/csi300_dataset.parquet`**
  - Rows: ~205,000
  - Stocks: 931 stocks that were ever CSI300 constituents since 2006 (point-in-time membership from BaoStock; includes delisted/removed stocks)
  - Date range: 2003-01 to 2026-09 (month-end); reliable membership from 2006-01
  - Columns: monthly OHLCV + turnover + forward-filled quarterly financials + point-in-time `in_csi300` flag + name

Typical size after a full update:

| file | size | description |
|---|---|---|
| `data/processed/csi300_dataset.parquet` | ~18 MB | merged panel |
| `data/processed/prices_monthly.parquet` | ~3 MB | monthly prices (original pool) |
| `data/processed/prices_monthly_extra.parquet` | ~8 MB | monthly prices (expanded pool) |
| `data/processed/financials_quarterly.parquet` | ~2 MB | quarterly financials (original pool) |
| `data/processed/financials_quarterly_extra.parquet` | ~4 MB | quarterly financials (expanded pool) |
| `data/processed/financials_quarterly_delisted.parquet` | ~0.1 MB | quarterly financials (delisted stocks via BaoStock) |
| `data/membership/csi300_membership_baostock.parquet` | ~0.4 MB | point-in-time monthly membership (74,700 rows) |

## Field reference

### Price / market fields

`date, code, open, high, low, close, volume, amount, amplitude, pct_change, change, turnover`

### Financial fields (forward-filled from latest report)

- `report_date` — the report period end date
- `股票简称` — stock name
- `每股收益`
- `营业总收入-营业总收入`
- `营业总收入-同比增长`
- `营业总收入-季度环比增长`
- `净利润-净利润`
- `净利润-同比增长`
- `净利润-季度环比增长`
- `每股净资产`
- `净资产收益率`
- `每股经营现金流量`
- `销售毛利率`
- `所处行业`
- `最新公告日期`

### Membership fields

- `in_csi300` — whether the stock was a constituent in that month (point-in-time, BaoStock)

## Important caveats

1. **Blue-chip proxy**: we use the CSI300 constituents as a proxy for A-share blue chips, reconstructed point-in-time from BaoStock (free) and cross-validated against the official CSIndex weight file.
2. **Membership history is point-in-time since 2006-01**: monthly snapshots capture real entries/exits (incl. delistings). 2005 (index launch year) has no snapshot, so treat pre-2006 membership as unavailable.
3. **Residual survivorship gap (now negligible)**: prices cover all 931 ever-constituents; financials cover 924/931 — the 7 uncovered codes were in the index for under a year during 2006-2007 (BaoStock has no records for them). Re-running `pe15roe15` after the delisted-financials fill produced identical results.
4. **Financial frequency**: quarterly/annual performance snapshots (`stock_yjbb_em`), not full balance/cash-flow statements.
5. **qfq price anchoring**: pre-adjusted prices are anchored to the download date and drift after dividends; re-run `02`/`09` before precise backtests.

## License

See [LICENSE](./LICENSE).