# foundational-storage

**Repository Path**: mirrors_getsentry/foundational-storage

## Basic Information

- **Project Name**: foundational-storage
- **Description**: Object storage with meta data, expiration, and auditing
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-07-12
- **Last Updated**: 2026-02-22

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Objectstore

Objectstore is a data storage platform for blobs, files, and other unstructured
data at Sentry. It comprises a service with an RPC interface that internally
manages multiple backends, client libraries for easy integration, and a set of
utilities to manage stored data.

<img width="531" height="716" alt="overview" src="https://github.com/user-attachments/assets/e9a4df55-591c-495f-b2a6-60d76d49958e" />

## Internals

The platform is split into the following core components:

- `objectstore-server`: An `HTTP` server that exposes blob storage and calls
  functionality from the internal services. This crate creates the `objectstore`
  binary.
- `objectstore-service`: The core object storage logic.
- `clients`: The Rust and Python client library SDKs, which expose
  high-performance blob storage access.

Additionally, it contains a number of utilities:

- `stresstest`: A stresstest binary that can run various workloads against
  storage backends.

## Usage

Objectstore consists of a stateless service that connects to one or more storage
backends, and client libraries for Python and Rust. To use objectstore,
configure the desired backends, run the service, and connect to it using one of
the provided clients.

### Configuration

The objectstore service is configured through a configuration file and
environment variables. If both are provided, environment variables take
precedence and override the configuration file settings. See the [config docs]
for a complete list of available backends and options.

An example configuration file that uses the local file system for storage can be
found in this [example file].

[config docs]: https://getsentry.github.io/objectstore/rust/objectstore_server/config/
[example file]: objectstore-server/config/local.example.yaml

### Docker

A pre-built Docker container is published at `ghcr.io/getsentry/objectstore`,
supporting both `amd64` and `arm64` architectures. To run the container, mount
your configuration file and data volume:

```sh
docker run -d \
  --name objectstore \
  --volume ./config.yaml:/etc/objectstore/config.yaml \
  --volume data-volume:/data \
  --publish 127.0.0.1:8888:8888 \
  ghcr.io/getsentry/objectstore:latest \
  --config /etc/objectstore/config.yaml run
```

The command above assumes a configuration file named `config.yaml` in the current
directory and publishes the service on port `8888`. Adjust the volume mount path
and port mapping as needed for your setup.

### Devservices

For Sentry development environments, we provide a [devservice] that can be
referenced from any project that needs to run objectstore. To use it, add the
following to the `dependencies` section in your `devservices/config.yml`:

```yaml
objectstore:
  description: Storage for files and blobs
  remote:
    repo_name: objectstore
    branch: main
    repo_link: https://github.com/getsentry/objectstore.git
    mode: containerized
```

In the Sentry backend, objectstore is integrated and can be started using
`devservices up --mode=objectstore`. No additional configuration is necessary,
simply import a client from `sentry.objectstore`.

[devservice]: https://develop.sentry.dev/development-infrastructure/devservices/

### Clients

To integrate objectstore into your application, you can either use the HTTP API
directly or leverage one of the higher-level client SDKs. We provide official
client libraries for both [Python] and [Rust], which handle connection
management, retries, and compression for you. Refer to the SDK documentation
for detailed setup and usage instructions.

[Python]: https://getsentry.github.io/objectstore/python/
[Rust]: https://getsentry.github.io/objectstore/rust/objectstore_client/

## Building

Ensure `protoc` and the latest stable Rust toolchain are installed on your
machine. A release build can be created with:

```sh
cargo build --release --locked
```

The server binary will be located at `target/release/objectstore`.

### Docker Container

To build the Docker container, first build the release binary, then build the
container:

```sh
docker build -f Dockerfile target/release/
```

The last argument must be the path to the directory that contains the
`objectstore` binary.

### Cross-building for AMD64 from ARM64

We have utilities to cross-build an AMD64 container from an ARM64 host, such as
a macOS machine running on Apple Silicon. To do this, run:

```sh
scripts/build-cross.sh
```

This script will take care of the entire cross-compilation build process inside
an ARM-based Docker container and use the project's `target/` directory for
caching and build output. The build results in two artifacts:

- The binary at `target/x86_64-unknown-linux-gnu/release/objectstore`
- The docker container, tagged with `objectstore:latest`

## Development

### Environment Setup

Before starting development, ensure you have development tools and environment
configured:

```bash
# Option 1: Use direnv (recommended)
direnv allow

# Option 2: Manually source the environment
source .envrc
```

This will create and activate the `.venv` virtual environment with the correct
Python version, install development tools, and export environment variables for
development.

### Devservices

Our test suite also tests backends that require development services to connect
to. We recommend to use [`devservices`] and start objectstore in "full" mode.
Note that this does not start objectstore itself, but only its dependencies:

[`devservices`]: https://github.com/getsentry/devservices

```sh
devservices up objectstore --mode=full
```

Devservices continue to run in the background until explicitly stopped. If you
prefer to start containers manually, please check `devservices/config.yml` for
the required images and configuration, such as port mapping.

For **Google BigTable**, we automatically create a table with the required
column families. Use this configuration to connect (recommended as high-volume
backend):

```yaml
high_volume_storage:
  type: bigtable
  endpoint: localhost:8086
  project_id: testing
  instance_name: objectstore
  table_name: objectstore
```

For **Google Cloud Storage** (GCS), a test bucket is already configured in the
dev container. Use this configuration to connect (recommended as long-term
backend):

```yaml
long_term_storage:
  type: gcs
  endpoint: http://localhost:8087
  bucket: test-bucket
```

### Editor Setup

We recommend using **Visual Studio Code** with the recommended extensions. The
project includes VSCode configuration that will:

- Automatically format code on save
- Automatically organize imports on save
- Show linting errors inline

When you open this project in VSCode, it will prompt you to install the
recommended extensions if they're not already installed.

### Code Quality

We recommend using Rust Analyzer and clippy, which will apply formatting and
warn about lints during development. If you use the recommended VSCode setup,
this is done automatically for you.

To run linters manually, use:

```sh
# Check and fix formatting
cargo fmt

# Lint all features
cargo clippy --workspace --all-features
```

### Development Server

You can run a development build of the server with `cargo run`. Objectstore can
be configured through a configuration file or environment variables. Note that
if both are given, environment variables override the config file. For this
reason, we do **not** provide default configuration variables in `.envrc`.

```sh
# Option 1: Configuration file
cargo run -- -c objectstore-server/config/local.example.yaml run

# Option 2: Environment variables
export OS__HIGH_VOLUME_STORAGE__TYPE=filesystem
export OS__HIGH_VOLUME_STORAGE__PATH=data/high-volume
export OS__LONG_TERM_STORAGE__TYPE=filesystem
export OS__LONG_TERM_STORAGE__PATH=data/long-term
cargo run -- run
```

You can copy and save additional config files into next to the examples in
`objectstore-server/config`. All other files are ignored by git.

### Tests

To run tests:

```sh
cargo test --workspace --all-features
```

## Utilities

Run the stresstest binary against the running server with:

```sh
cargo run --release -p stresstest -- -c stresstest/config/example.yaml
```

Similar to the objectstore server, you can find example configuration files in
the `config` subfolder. Copy and save your own files there, as they will not be
tracked by git.

## License

Like Sentry, Objectstore is licensed under the FSL. See the `LICENSE.md` file
and [this blog post](https://blog.sentry.io/introducing-the-functional-source-license-freedom-without-free-riding/)
for more information.