# catar

**Repository Path**: mirrors_regular/catar

## Basic Information

- **Project Name**: catar
- **Description**: stream of streams archive format sepc and tools
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-07-27
- **Last Updated**: 2026-10-20

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# catar(5) - concatenatable archive format

## SYNOPSIS

**catar** - a stream-of-streams archive format designed for simplicity, efficient streaming and concatenation

**splat** - split catar archives and execute commands on each member

## DESCRIPTION

The catar format stores multiple, optionally named, binary streams in a single file. Unlike tar(1), zip(1), and other traditional archives, catar files can be concatenated with cat(1) to produce a valid merged archive. No special tools are required to create catar archives - they can be written by hand or with simple shell scripts.

Each stream (called a "member") consists of a boundary marker, optional headers, and binary data. The format is self-synchronizing: corruption in one member does not prevent reading subsequent members.

## DESIGN GOALS

1. **Trivial to create** - No special tools required
2. **Concatenatable** - `cat a.catar b.catar > c.catar` just works
3. **Streamable** - Process members while receiving the file
4. **Self-synchronizing** - Recover from corruption
5. **Human-readable headers** - Inspect with less(1) or grep(1)
6. **Binary-safe** - Store any data without encoding
7. **Adaptable** authors select boundary string length for their use-case

## FORMAT SPECIFICATION

### Grammar

```
archive         = first-member member*

first-member    = boundary-init newline header body

member          = boundary newline header body

boundary-init   = non-ws-char non-ws-char non-ws-char+
                  ; at least 3 chars, no whitespace, NO metadata

boundary        = non-ws-char non-ws-char non-ws-char+ [space metadata]
                  ; subsequent boundaries can have metadata
                  
metadata        = word [space word]*
word            = non-ws-char+

header          = [header-line newline]* newline
                  ; ends with empty line

header-line     = key colon space+ value
                  ; at least one space after colon
key             = char-no-colon+
value           = char-no-newline*
                  ; can be empty after colon and space

body            = [byte]* terminator
                  
terminator      = newline boundary newline

; Character classes
newline         = '\n' (0x0A)
space           = ' ' (0x20)  
non-ws-char     = printable ASCII except whitespace
char-no-colon   = any except ':' and '\n'
char-no-newline = any except '\n'
byte            = any (0x00-0xFF)
```

### Rules

1. Boundaries must be at least 3 characters, contain no whitespace
2. First boundary in an archive cannot have metadata
3. Headers require `key: value` format with at least one space after colon
4. Headers end with an empty line
5. Bodies are terminated by newline + boundary + newline

### Simple Examples

A minimal archive with two empty members:
```
---

---

---
```

Archive with headers:
```
===
name: README
type: text

This is a text file.
===
name: data.bin  
type: binary
encoding: base64

SGVsbG8gV29ybGQh
===
```

Archive with metadata (checksums computed after streaming):
```
%%%
name: bigfile.iso

[2GB of binary data]
%%% sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
name: smallfile.txt

Hello
%%%
```

## THE SPLAT TOOL

### Synopsis

```
splat COMMAND
```

### Description

splat reads a catar archive from stdin and executes COMMAND once per member. The command receives:
- Member data on stdin
- Header fields as environment variables

### Special Variables

- `$MEMBER_NUM` - Sequential member number (starting at 0)

### Examples

Extract all members to files named by their 'name' header:
```
$ cat archive.catar | splat 'cat >$name'
```

Process only text files:
```
$ cat archive.catar | splat '[ "$type" = "text" ] && wc -l'
```

Verify checksums:
```
$ cat archive.catar | splat 'echo "$name: $(sha256sum | cut -d\  -f1)"'
```

### Creating Archives

By hand:
```
$ cat >example.catar <<'EOF'
===
name: greeting.txt
lang: en

Hello, World!
===
name: greeting_de.txt
lang: de  

Hallo, Welt!
===
EOF
```

Programmatically:
```
$ (echo "###"
   echo "name: kernel.log"
   echo ""
   cat /var/log/kern.log
   echo ""
   echo "###"
   echo "name: system.log"  
   echo ""
   cat /var/log/syslog
   echo ""
   echo "###") > logs.catar
```

### Concatenating Archives

```
$ cat imgs1.catar imgs2.catar imgs3.catar > all-images.catar
```

### Streaming Processing

Count lines in all text files without extracting:
```
$ curl http://example.com/data.catar | \
  splat '[ "$type" = "text/plain" ] && echo "$name: $(wc -l)"'
```

## RATIONALE

Traditional archive formats require special tools and complex parsing. They typically cannot be concatenated without rewriting headers or rebuilding indices. The catar format solves these problems through careful design:

- **No global header** - Each member is self-contained
- **Boundary markers** - Like MIME multipart, but simpler
- **Required terminators** - Enable reliable concatenation
- **Metadata placement** - After data, when checksums are known
- **First boundary restriction** - Prevents ambiguity during concatenation

The name "catar" combines "cat" (concatenate) and "ar" (archive). The tool "splat" suggests the inverse of concatenation - splitting streams apart.

## SEE ALSO

cat(1), tar(1), ar(1), cpio(1), shar(1)

MIME multipart format (RFC 2046)

## AUTHORS

regular@github.com

# catar(5) - concatenatable archive format

## SYNOPSIS

**catar** - a stream-of-streams archive format designed for simplicity and concatenation

**splat** - split catar archives and execute commands on each stream

## DESCRIPTION

The catar format stores multiple named binary streams in a single file. Unlike tar(1), zip(1), and other traditional archives, catar files can be concatenated with cat(1) to produce a valid merged archive. No special tools are required to create catar archives - they can be written by hand or with simple shell scripts.

Each stream (called a "member") consists of a boundary marker, optional headers, and binary data. The format is self-synchronizing: corruption in one member does not prevent reading subsequent members.

## DESIGN GOALS

1. **Trivial to create** - No special tools required
2. **Concatenatable** - `cat a.catar b.catar > c.catar` just works
3. **Streamable** - Process members without reading entire file
4. **Self-synchronizing** - Recover from corruption
5. **Human-readable headers** - Inspect with less(1) or grep(1)
6. **Binary-safe** - Store any data without encoding

## FORMAT SPECIFICATION

### Grammar

```
archive         = first-member member*

first-member    = boundary-init newline header body

member          = boundary newline header body

boundary-init   = non-ws-char non-ws-char non-ws-char+
                  ; at least 3 chars, no whitespace, NO metadata

boundary        = non-ws-char non-ws-char non-ws-char+ [space metadata]
                  ; subsequent boundaries can have metadata
                  
metadata        = word [space word]*
word            = non-ws-char+

header          = [header-line newline]* newline
                  ; ends with empty line

header-line     = key colon space+ value
                  ; at least one space after colon
key             = char-no-colon+
value           = char-no-newline*
                  ; can be empty after colon and space

body            = [byte]* terminator
                  
terminator      = newline boundary newline

; Character classes
newline         = '\n' (0x0A)
space           = ' ' (0x20)  
non-ws-char     = printable ASCII except whitespace
char-no-colon   = any except ':' and '\n'
char-no-newline = any except '\n'
byte            = any (0x00-0xFF)
```

### Rules

1. Boundaries must be at least 3 characters, contain no whitespace
2. First boundary in an archive cannot have metadata
3. Headers require `key: value` format with at least one space after colon
4. Headers end with an empty line
5. Bodies are terminated by newline + boundary + newline

### Simple Examples

A minimal archive with two empty members:
```
---

---

---
```

Archive with headers:
```
===
name: README
type: text

This is a text file.
===
name: data.bin  
type: binary
encoding: base64

SGVsbG8gV29ybGQh
===
```

Archive with metadata (checksums computed after streaming):
```
%%%
name: bigfile.iso

[2GB of binary data]
%%% sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
name: smallfile.txt

Hello
%%%
```

## THE SPLAT TOOL

### Synopsis

```
splat COMMAND
```

### Description

splat reads a catar archive from stdin and executes COMMAND once per member. The command receives:
- Member data on stdin
- Header fields as environment variables
- Metadata fields as environment variables (prefixed with `META_`)

### Special Variables

- `$BOUNDARY` - The boundary string for this archive
- `$MEMBER_NUM` - Sequential member number (starting at 1)

### Examples

Extract all members to files named by their 'name' header:
```
$ cat archive.catar | splat 'cat >$name'
```

Process only text files:
```
$ cat archive.catar | splat '[ "$type" = "text" ] && wc -l'
```

Verify checksums:
```
$ cat archive.catar | splat 'echo "$name: $(sha256sum | cut -d\  -f1)"'
```

Convert to tar (with some limitations):
```
$ cat archive.catar | splat 'tar cf - --add-file=- --transform="s:-:$name:"' | tar tf -
```

## EXAMPLES

### Creating Archives

By hand:
```
$ cat >example.catar <<'EOF'
===
name: greeting.txt
lang: en

Hello, World!
===
name: greeting_de.txt
lang: de  

Hallo, Welt!
===
EOF
```

Programmatically:
```
$ (echo "###"
   echo "name: kernel.log"
   echo ""
   cat /var/log/kern.log
   echo ""
   echo "###"
   echo "name: system.log"  
   echo ""
   cat /var/log/syslog
   echo ""
   echo "###") > logs.catar
```

### Concatenating Archives

```
$ cat imgs1.catar imgs2.catar imgs3.catar > all-images.catar
```

### Streaming Processing

Count lines in all text files without extracting:
```
$ curl http://example.com/data.catar | \
  splat '[ "$type" = "text/plain" ] && echo "$name: $(wc -l)"'
```

## RATIONALE

Traditional archive formats require special tools and complex parsing. They typically cannot be concatenated without rewriting headers or rebuilding indices. The catar format solves these problems through careful design:

- **No global header** - Each member is self-contained
- **Boundary markers** - Like MIME multipart, but simpler
- **Required terminators** - Enable reliable concatenation
- **Metadata placement** - After data, when checksums are known
- **First boundary restriction** - Prevents ambiguity during concatenation

The name "catar" combines "cat" (concatenate) and "ar" (archive). The tool "splat" suggests the inverse of concatenation - splitting streams apart.

## SEE ALSO

cat(1), tar(1), ar(1), cpio(1), shar(1)

MIME multipart format (RFC 2046)

## AUTHORS

regular@github.com
This document: An AI with too much time on its hands