# gocelery

**Repository Path**: yuebanlaosiji/gocelery

## Basic Information

- **Project Name**: gocelery
- **Description**: go python celery 构建开发
- **Primary Language**: Go
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-03-15
- **Last Updated**: 2025-06-11

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# GoCelery

GoCelery is a Go implementation of the Celery distributed task queue system, designed to enable seamless communication between Go and Python services using the Celery protocol.

## Features

- Send and receive Celery tasks from Go code
- Interoperate with Python Celery workers and clients
- Support for Redis broker (with plans for RabbitMQ)
- Asynchronous task execution
- Task result retrieval
- Modular architecture with clean interfaces

## Installation

```bash
go get gitee.com/yuebanlaosiji/gocelery
```

## Quick Start

### Worker Example

```go
package main

import (
    "fmt"
    "log"
    
    "gitee.com/yuebanlaosiji/gocelery/pkg/gocelery"
)

// Define a task function
func addTask(args []interface{}, kwargs map[string]interface{}) (interface{}, error) {
    a, _ := args[0].(float64)
    b, _ := args[1].(float64)
    return a + b, nil
}

func main() {
    // Create a new worker
    worker, err := gocelery.NewWorker(gocelery.Config{
        BrokerURL:  "redis://localhost:6379/0",
        BackendURL: "redis://localhost:6379/0",
    })
    if err != nil {
        log.Fatal(err)
    }
    
    // Register the task
    worker.Register("tasks.add", addTask)
    
    // Start the worker
    worker.Start()
    
    // Block until interrupted
    select {}
}
```

### Client Example

```go
package main

import (
    "fmt"
    "log"
    "time"
    
    "gitee.com/yuebanlaosiji/gocelery/pkg/gocelery"
)

func main() {
    // Create a new client
    client, err := gocelery.NewClient(gocelery.Config{
        BrokerURL:  "redis://localhost:6379/0",
        BackendURL: "redis://localhost:6379/0",
    })
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()
    
    // Send a task
    taskID, err := client.SendTask("tasks.add", []interface{}{1, 2}, nil)
    if err != nil {
        log.Fatal(err)
    }
    
    fmt.Printf("Task sent. ID: %s\n", taskID)
    
    // Wait for the result
    result, err := client.WaitForResult(taskID, 5*time.Second)
    if err != nil {
        log.Fatal(err)
    }
    
    fmt.Printf("Result: %v\n", result)
}
```

## Connection URLs

### Redis Connection URL Format

GoCelery supports Redis connection URLs with the following format:

```
redis://[username:password@]host:port/db
```

Examples:
- `redis://localhost:6379/0` - Connect to Redis without authentication
- `redis://:mypassword@localhost:6379/0` - Connect with just a password
- `redis://username:mypassword@localhost:6379/0` - Connect with username and password (Redis 6.0+)

## Python Interoperability

GoCelery can communicate with Python Celery workers and clients. For example, a Go client can send tasks to a Python worker:

### Python Worker (tasks.py)

```python
from celery import Celery

app = Celery('tasks', 
             broker='redis://localhost:6379/0',
             backend='redis://localhost:6379/0')

app.conf.update(
    task_serializer='json',
    accept_content=['json'],
    result_serializer='json',
)

@app.task(name='tasks.add')
def add(x, y):
    return x + y
```

Run the Python worker:

```bash
celery -A tasks worker --loglevel=info
```

### Go Client

```go
package main

import (
    "fmt"
    "log"
    "time"
    
    "gitee.com/yuebanlaosiji/gocelery/pkg/gocelery"
)

func main() {
    client, _ := gocelery.NewClient(gocelery.Config{
        BrokerURL:  "redis://localhost:6379/0",
        BackendURL: "redis://localhost:6379/0",
    })
    defer client.Close()
    
    // Call Python task
    taskID, _ := client.SendTask("tasks.add", []interface{}{1, 2}, nil)
    
    // Get result
    result, _ := client.WaitForResult(taskID, 5*time.Second)
    fmt.Printf("Result from Python: %v\n", result)
}
```

### Keys to successful Python interoperability:

1. **Message Format Compatibility**: GoCelery ensures that the task messages sent to Python Celery workers follow the Celery protocol:
   - Task messages include a full envelope with `body`, `headers`, and `properties`
   - The task body is encoded as base64
   - Keyword arguments (`kwargs`) are always provided as an empty map (`{}`) even when not used, never as `null`

2. **Result Handling**: When retrieving results from Python Celery workers:
   - Results are stored in Redis with the key format `celery-task-meta-{taskID}`
   - The result format includes status, result value, and other metadata
   - GoCelery handles the parsing of this format to extract the result value

3. **Data Type Compatibility**: 
   - JSON is used as the serialization format for maximum compatibility
   - Go numeric types are automatically handled when communicating with Python
   - String, boolean, and array/slice types are directly compatible

### Troubleshooting Python Interoperability

If you encounter issues with Python interoperability, check the following:

1. **Task Not Received by Python Worker**:
   - Ensure the task name matches exactly between Go and Python
   - Verify the Redis connection details are correct
   - Check if your task message format is correct (run with debug logs)

2. **Result Not Retrieved**:
   - Verify the Python worker is storing results (check Redis)
   - Ensure the task ID used for retrieval matches the one from task submission
   - Confirm the result backend URL is correct

3. **Data Type Issues**:
   - Complex Python objects may not serialize/deserialize correctly
   - Stick to basic types (numbers, strings, booleans, arrays, maps) for best compatibility

For a complete example of Go-Python interoperability, see the `examples/python_interop` directory.

## Examples

See the `examples` directory for more detailed examples:

- `examples/simple`: Basic worker and client
- `examples/python_interop`: Interoperability with Python Celery

## Architecture

GoCelery is built with a modular architecture:

- **Broker**: Handles message transport (Redis, RabbitMQ)
- **Backend**: Manages task result storage
- **Worker**: Processes tasks from the queue
- **Client**: Submits tasks to the queue
- **Serializer**: Handles data serialization/deserialization
- **Protocol**: Implements the Celery wire protocol
- **Registry**: Stores and manages task registrations

## Development

### Prerequisites

- Go 1.20 or later
- Redis server
- For interoperability testing: Python with Celery

### Building

```bash
go build -o gocelery ./...
```

### Testing

```bash
# Run all tests
go test ./...

# Skip integration tests
go test -short ./...
```

## License

MIT License