# gcloud-node-bigquery-manystreams-bug

**Repository Path**: mirrors_stephenplusplus/gcloud-node-bigquery-manystreams-bug

## Basic Information

- **Project Name**: gcloud-node-bigquery-manystreams-bug
- **Description**: to reproduce bug with google cloud node bigquery - silently drops streams when many streams are used.
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2020-11-23
- **Last Updated**: 2026-04-05

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# gcloud-node-bigquery-manystreams-bug
To reproduce a bug with google cloud node bigquery - it sometimes silently drops a variable subset of streams when many streams are used in parallel.

When the bug reproduces and `n` streams of size `s` are dropped bigquery will have `n` * `s` missing rows. So if `n=2` and `s=150` bigquery will be missing 300 rows. In other words, the problem does not appear to be that a subset of stream data is missing, but rather one or more entire streams are missing.

This seems to reproduce reliably sometimes, and other times it reliably does not reproduce. To try and get the opposite result, try again later and/or change the load with the env variables.

This small bug reproduction project is the result of troubleshooting missing big query data in a production system.

## How to reproduce the bug

### setup
```bash
git clone https://github.com/chrishiestand/gcloud-node-bigquery-manystreams-bug.git
cd gcloud-node-bigquery-manystreams-bug
npm install
cp .env.example .env
# copy a gcp credentials file into secret/ and update .env keyFilePath and projectId
```

### Run the test

```bash
npm test
```

If the test passes, the bug has not reproduced. Try again and try increasing the env values of `totalStreams`, `streamLength`, and `queryLatencyMs`

If the test fails, the bug might have reproduced or there may have not been enough wait time between the write and the subsequent query. Look at the Big Query UI and verify that the table still does not contain the number of expected rows.