# unicode-segmentation-cj

**Repository Path**: iJetLi/unicode-segmentation-cj

## Basic Information

- **Project Name**: unicode-segmentation-cj
- **Description**: Rust unicode-segmentation库的仓颉cangjie实现
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-07-08
- **Last Updated**: 2026-07-12

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# unicode-segmentation-cj

Unicode 文本边界分割算法的仓颉(Cangjie)实现，完整移植自 Rust [unicode-segmentation](https://github.com/rust-lang/unicode-segmentation) crate v1.13.3。

实现了 Unicode Standard Annex #29 定义的文本边界分割算法，包括：

- **Grapheme Cluster 边界分割** — 按字素簇（用户感知的字符）分割
- **Word 边界分割** — 按词边界分割
- **Sentence 边界分割** — 按句子边界分割

## Unicode 版本

Unicode 17.0.0

## 功能

### Grapheme Cluster 分割

```cangjie
let s = "a̐éö̲\r\n"

// 正向迭代
for (g in s.graphemes(true)) {
    println(g)  // "a̐", "é", "ö̲", "\r\n"
}

// 反向迭代
var iter = s.graphemes(true)
var g = iter.next_back()
while (g.isSome()) {
    println(g.getOrThrow())
    g = iter.next_back()
}

// 偏移量迭代
for (pair in s.grapheme_indices(true)) {
    let (offset, grapheme) = (pair[0], pair[1])
    println("${offset}: ${grapheme}")
}
```

### Word 边界分割

```cangjie
let s = "The quick (\"brown\") fox can't jump 32.3 feet, right?"

// 仅字母数字词
for (w in s.unicode_words()) {
    println(w)  // "The", "quick", "brown", "fox", "can't", "jump", "32.3", "feet", "right"
}

// 包含非字母数字的完整边界
for (w in s.split_word_bounds()) {
    println(w)  // "The", " ", "quick", " ", "(", ...
}
```

### Sentence 边界分割

```cangjie
let s = "Mr. Fox jumped. The dog was too lazy."

// 仅字母数字句
for (sent in s.unicode_sentences()) {
    println(sent)
}

// 完整边界（含空格和标点）
for (sent in s.split_sentence_bounds()) {
    println(sent)
}
```

### GraphemeCursor 分块处理

```cangjie
// 适用于 rope/buffer 等非连续内存场景
var cursor = GraphemeCursor(0, full_len, true)

try {
    let is_break = cursor.is_boundary(chunk, chunk_start)
} catch (e: GraphemeIncomplete) {
    // 需要提供更多上下文
    cursor.provide_context(prev_chunk, prev_chunk_start)
    let is_break = cursor.is_boundary(chunk, chunk_start)
}
```

## 公共 API

### UnicodeSegmentation 接口

所有方法通过 `extend String <: UnicodeSegmentation` 添加到 String：

| 方法 | 说明 |
|------|------|
| `graphemes(is_extended)` | Grapheme Cluster 迭代器 |
| `grapheme_indices(is_extended)` | Grapheme Cluster + 字节偏移量 |
| `split_word_bounds()` | Word 全边界迭代器 |
| `split_word_bound_indices()` | Word 全边界 + 偏移量 |
| `unicode_words()` | 仅含字母数字的词迭代器 |
| `unicode_word_indices()` | 词 + 偏移量 |
| `split_sentence_bounds()` | Sentence 全边界迭代器 |
| `split_sentence_bound_indices()` | Sentence + 偏移量 |
| `unicode_sentences()` | 仅含字母数字的句迭代器 |

### 公共类/枚举

- `GraphemeCursor` — 分块式 Grapheme 边界判断
- `GraphemeIncomplete` — 分块处理错误枚举
- `UNICODE_VERSION` — (17, 0, 0)

## 构建与测试

```bash
cjpm build
cjpm test
```

## 许可证

MIT