# cube

**Repository Path**: wilenwu/cube

## Basic Information

- **Project Name**: cube
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2024-11-14
- **Last Updated**: 2026-08-27

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# MiniCube：端到端机器学习框架

MiniCube 是一套全面的模块化机器学习工作流框架，涵盖以下功能：

- 数据加载、预处理与验证
- 特征工程（选择、构建、编码、插补、降维）
- 模型训练（梯度提升、神经网络）与超参数优化
- 模型评估、比较与文档化
- 通过用户自定义函数（UDF）、回调函数及损失函数实现定制化

## 核心特性

### 🔧 模块化与可扩展

- 全流程插件化组件（数据处理、预处理、建模、评估），支持按需组合
- 支持通过自定义函数（UDF）、损失函数与回调函数扩展功能
- 兼容梯度提升树（LightGBM/XGBoost/CatBoost）与神经网络模型

### 🚀 高效工作流

- 常用任务一键式 API（如模型训练、特征选择）
- 大型数据集延迟加载（Lazy Loading），节省内存开销
- 贝叶斯超参数优化（支持 Optuna/Hyperopt/Skopt）

### 📊 全面工具链

- 自动化特征工程（特征选择、编码、构造、降维）
- 鲁棒的评估指标（分类任务：AUC/KS/Lift；回归任务：MAE/R²）
- 群体稳定性指数（PSI），用于监控数据漂移
- 自动化 Excel 报告生成，内置可视化图表

### 🔍 可复现与透明化

- 模型、预处理流水线与配置的内置序列化功能
- 特征谱系追踪（记录原始特征、衍生特征、剔除特征）
- 详细日志与实验文档记录

## 安装指南

### 本地开发环境安装 (推荐)

```
# Install only core features
pip install -e .

# Install full features
pip install -e .[full]

# Developer installation (with testing, formatting tools)
pip install -e .[dev,full]
```

### 生产环境安装

```
# Install full features
pip install minicube[full]

# Install only core features
pip install minicube
```

### 可选依赖

| Feature               | Installation Command           |
| --------------------- | ------------------------------ |
| Neural Networks       | `pip install minicube[nn]`     |
| Hyperparameter Tuning | `pip install minicube[tuning]` |

## Quick Start Workflow

### 1. Load & Split Data

```python
from minicube import DataReader, col

# Load CSV and split into train/test/oot (ordered split by date)
reader = DataReader(
    "data.csv",
    delimiter=",",
    header=0
).split(
    by="transaction_date",
    method="ordered",
    weights={"train": 0.7, "test": 0.2, "oot": 0.1}
).fit(
    filters=(col("age") >= 18) & (col("income") > 30000)  # Declarative filter
)

train_data = reader["train"]
test_data = reader["test"]
oot_data = reader["oot"]
```

### 2. Preprocess Features

```python
from minicube import Preprocessor
from minicube.feature.selection import (
    DropHighMissingFeatures,
    SelectByInformationValue
)

# Define preprocessing pipeline
preprocessor = Preprocessor(
    steps=[
        DropHighMissingFeatures(threshold=0.3),  # Drop features with >30% missing values
        SelectByInformationValue(threshold=0.02)    # Keep features with IV ≥ 0.02
    ],
    max_features=50,  # Retain top 50 features by IV
    reference="iv"    # Use Information Value for ranking
)

# Fit preprocessor and transform data
preprocessor.fit(train_data, input_cols=train_data.columns.drop("target"), target="target")
train_processed = preprocessor.transform(train_data)
test_processed = preprocessor.transform(test_data)
```

### 3. Train Model

```python
from minicube import Cube

# Initialize model pipeline (binary classification with LightGBM)
cube = Cube(
    problem_type="binary",
    algorithm="lightgbm",
    optimizer="optuna",
    save_dir="model_output",
    feature_desc=pd.Series({"age": "Customer age", "income": "Annual income"})
)

# Train with hyperparameter optimization
cube.fit(
    data=train_processed,
    input_cols=preprocessor.selected_features,
    target="target",
    cv=3,
    n_trials=50,
    early_stopping_rounds=10
)
```

### 4. Evaluate & Generate Report

```python
# Make predictions
predictions = cube.predict(
    test_processed,
    retained_cols=["customer_id", "transaction_date"]
)

# Evaluate performance
cube.evaluate(
    prediction=predictions,
    sets_col="dataset_split",
    train_set="train"
)

# Generate comprehensive Excel report
cube.report(
    data=predictions,
    sets_col="dataset_split",
    n_features=10  # Highlight top 10 features
)
```

## 推荐项目结构

```
.
├── dataset/
├── explore/                  # Data exploration reports (stats, plots)
├── preprocessing/            # Saved preprocessing pipeline (for inference)
├── models/
│   ├── network_random/       # Model trained with validation split
│   │   ├── model.pkl         # Trained neural network model
│   │   ├── prediction.csv    # Predictions for train/test sets
│   │   └── report.xlsx       # Detailed evaluation report
│   └── network_all/          # Model trained on full train set
├── model_selection.xlsx      # Model comparison results
└── main.py                   # Main workflow script
```


## 关键组件说明

### Data Handling

| Component      | Purpose                                          |
| -------------- | ------------------------------------------------ |
| `DataReader`   | Load data, split (random/ordered), and filter    |
| `Column`/`col` | Declarative column reference for building filters |
| `Expression`   | Create complex filters (e.g. `col("age") >= 18`) |

### Feature Engineering

| Category     | Components                                                               |
| ------------ |--------------------------------------------------------------------------|
| Selection    | `DropHighMissingFeatures`, `SelectByInformationValue`, `SelectFromModel` |
| Construction | `AnomalyDetector`, `Cluster`, `DatetimeDelta`, etc                       |
| Encoding     | `InfrequencyEncoder`, `CategoryIndexer`                                  |
| Imputation   | `AutoEncoderImputer` (neural network-based)                              |
| Reduction    | `PCA`, `ManifoldEmbedding`                                                       |

### Modeling

| Component               | Purpose                                                      |
| ----------------------- | ------------------------------------------------------------ |
| `Cube`                  | End-to-end pipeline (training, evaluation, reporting)        |
| `GradientBoostingModel` | Wrapper for LightGBM/XGBoost/CatBoost                        |
| `NeuralNetwork`         | Custom neural network (requires `minicube[nn]`)              |
| `BayesianSearchCV`      | Bayesian hyperparameter optimization (Optuna/Hyperopt/Skopt) |

### Evaluation & Documentation

| Component        | Purpose                                                                 |
| ---------------- | ----------------------------------------------------------------------- |
| `ModelEvaluator` | Calculate metrics (AUC/KS/Lift/PSI) and compare splits                  |
| `Document`       | Generate Excel reports with model overview, metrics, and visualizations |
| `compare_models` | Side-by-side performance comparison of multiple models                  |

## 完整工作流示例

```python
# 1. Setup
import pandas as pd
from minicube import DataReader, Preprocessor, Cube, col
from minicube.feature.selection import DropHighCorrelationFeatures, SelectByMutualInformation
from minicube.model.callback import LightGBMOverfitStopping

# 2. Load and prepare data
reader = DataReader("credit_risk.csv").split(
    by="application_date",
    method="ordered",
    weights={"train": 0.7, "test": 0.3}
).fit(filters=col("loan_amount") > 0)

train = reader["train"]
test = reader["test"]

# 3. Preprocess features
preprocessor = Preprocessor(
    steps=[
        DropHighCorrelationFeatures(threshold=0.85),
        SelectByMutualInformation(min_mi=0.01)
    ],
    max_features=30,
    reference="mutual_information"
).fit(train, input_cols=train.columns.drop("default"), target="default")

# 4. Train model with custom callback
cube = Cube(
    problem_type="binary",
    algorithm="lightgbm",
    save_dir="credit_risk_model",
    sample_date="application_date"
)

cube.fit(
    data=preprocessor.transform(train),
    input_cols=preprocessor.selected_features,
    target="default",
    cv=5,
    n_trials=80,
    callbacks=[LightGBMOverfitStopping(gap_thresh=0.05)],  # Prevent overfitting
    focal_loss=BinaryFocalLoss(gamma=2)  # Handle class imbalance
)

# 5. Evaluate on test and OOT data
test_preds = cube.predict(preprocessor.transform(test), retained_cols=["customer_id", "application_date"])
test_preds["dataset_split"] = "test"

# 6. Generate final report
cube.evaluate(prediction=test_preds, sets_col="dataset_split", train_set="train")
cube.report(data=test_preds, sets_col="dataset_split", n_features=15)
```

## License

MiniCube 基于 MIT 许可证开源。

***

*Built with ❤️ for machine learning practitioners*

*Last updated: 2025-11-06*