# conspired

**Repository Path**: mirrors_UKPLab/conspired

## Basic Information

- **Project Name**: conspired
- **Description**: Data and source code for the CONSPIR Evaluation Dataset (CONSPIRED).
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-10-21
- **Last Updated**: 2026-05-24

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# 🧠 ConspirED Dataset Overview

Welcome to the repo for [ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety](https://arxiv.org/abs/2508.20468). 

**ConspirED** is a dataset for identifying cognitive traits of conspiracy theories in text. It contains annotated conspiracy snippets labeled with the [CONSPIR](https://skepticalscience.com/docs/ConspiracyTheoryHandbook.pdf) traits of conspiratorial ideation.

---
## 📁 Directory Structure

The repository is organized as follows:

```
arxiv2025-conspired/
├── data/                              # Dataset files
│   ├── context_training.xlsx          # Training split
│   ├── context_testing.xlsx           # Test split
│   ├── val_splits/                    # Validation splits
│   └── LICENSE-CC-BY-4.0.txt          # CC-BY-4.0 license for datasets
├── .github/                           # GitHub Actions and workflows
├── static/                            # Project page assets
├── conspir_tils.py                    # Utility functions for prompting experiments
├── main.py                            # Main script for LLM prompting experiments
├── train_clf.py                       # Script for fine-tuning LaGoNN classifiers
├── lagonn.py                          # LaGoNN model implementation
├── setup_utils.py                     # Setup and evaluation utilities
├── finetuning_environment.yml         # Conda environment for LaGoNN
├── prompting_environment.yml          # Conda environment for prompting
├── README.md                          # This file
├── LICENSE                            # Apache 2.0 license for code
├── NOTICE.txt                         # Copyright notices
├── .gitignore                         # Git ignore rules
├── .nojekyll                          # GitHub Pages configuration
└── index.html                         # Project landing page
```

### `data/`
Contains all processed dataset files:
#### Dataset Fields

Each dataset file contains the following columns:

- **`doc_id`**: Unique identifier for the source document
- **`snippet`**: The annotated text snippet exhibiting conspiratorial thinking
- **`context500`**: Surrounding context (500 tokens) around the snippet
- **`context1000`**: Surrounding context (1000 tokens) around the snippet
- **`labels`**: Multi-hot encoded list of conspiracy traits (0/1 for each of 6 traits)
- **`consolidated_trait`**: Human-readable list of trait names present in the snippet
- **`dominant_consol_trait`**: The most salient/dominant trait in the snippet
- **`single_dominant_one_hot_dm`**: One-hot encoded dominant trait vector
- **`OverallTrait`**: Original annotation of overall conspiracy traits
- **`DominantTrait`**: Original annotation of the dominant trait
- **`Justification`**: Annotator's justification for trait assignment
- **`Confidence`**: Annotator confidence score
- **`ConspiracyTheoryorMainstream`**: Classification of source as conspiracy theory or mainstream
- **`annotated_text`**: Original annotated text with markup
- **`linkingpassage`**: Context linking the snippet to broader narrative
- **`begin`/`end`**: Character offsets of the snippet in the source document
- **`name`**: Source annotator where relevant
- **`id`**: Snippet identifier
- **`Label`**: Additional label information
- **`remove_row`**: Flag for data quality filtering

The six conspiracy traits are (in order): **Contradictory**, **Overriding suspicion**, **Nefarious intent**, **Persecuted victim**, **Immune to evidence**, and **Re-interpreting randomness**.

---

## ⚙️ Environment Setup

To set up the environment, use the provided Conda YAML files:

- `finetuning_environment.yml` — for training and evaluating LaGoNN.
- `prompting_environment.yml` — for prompting experiments with LLaMA or GPT models.

### ✅ Create a Conda environment

Use the following commands:

#### LaGoNN (fine-tuning)
```bash
conda env create -f finetuning_environment.yml
conda activate lagonn-env
```

#### Prompting
```bash
conda env create -f prompting_environment.yml
conda activate prompting-env
```

### 🔑 API Configuration

If you plan to use OpenAI models (GPT-4, etc.), you need to set your API key as an environment variable:

```bash
export OPENAI_API_KEY='your-api-key-here'
```

For permanent configuration, add this line to your `~/.bashrc` or `~/.zshrc` file.

---

## 🏋️‍♂️ Fine-Tuning LaGoNN

Use the `train_clf.py` script to fine-tune LaGoNN classifiers on the ConspirED dataset.

### 🔧 Command-Line Arguments

| Argument | Description |
|----------|-------------|
| `--model` | HuggingFace model to fine-tune (e.g., `paraphrase-mpnet-base-v2`). |
| `--model_seed` | Random seed for model initialization. |
| `--num_iter` | Number of LaGoNN message-passing iterations. |
| `--epochs` | Number of training epochs. |
| `--multilab` | Set to `True` for multi-label classification; `False` for single-label. |
| `--lagonn_config` | Graph configuration (e.g., `LABEL`, `TEXT`, etc.). |
| `--lagonn_mode` | Name of the experimental setup (e.g., `LAGONN_EXP`). |
| `--NUM_NEIGHBORS` | Number of neighbors per node in the graph. |
| `--DISTANCE_PRECISION` | Optional: precision mode for node distance (default: `None`). |
| `--context` | Whether to include surrounding context in the input. |
| `--window` | Token window size used when `--context` is enabled. |

### 🚀 Example Usage

```bash
python train_clf.py \
  --model paraphrase-mpnet-base-v2 \
  --model_seed 4 \
  --num_iter 17 \
  --epochs 3 \
  --multilab True \
  --lagonn_config LABEL \
  --lagonn_mode LAGONN_EXP \
  --NUM_NEIGHBORS 1 \
  --setfit False \
  --context True \
  --window 1000
```

---

## 💬 Prompting with LLaMA and GPT Models

The `main.py` script runs prompting experiments to identify conspiratorial traits using LLaMA, GPT, or other LLMs.

### 🔧 Command-Line Arguments

| Argument | Description |
|----------|-------------|
| `--strategy` | Prompting strategy used (e.g., `what_to_look_for`). |
| `--icl` | In-context learning setup: `zero_shot`, `few_shot_similar`, `few_shot_dissimilar`, or `few_shot_both`. |
| `--k` | Number of examples used in few-shot prompting. |
| `--cot` | Whether to use chain-of-thought prompting (`True` or `False`). |
| `--dev` | Run on development set instead of test set (`True` or `False`). |
| `--model` | Path to the local LLaMA model (if not using OpenAI). |
| `--context` | Whether to include surrounding context in the prompt (`True` or `False`). |
| `--window` | Token window size for included context (ignored if `--context=False`). |
| `--openai` | Whether to use OpenAI API models (`True`) or local models (`False`). |
| `--openai_model` | OpenAI model name (e.g., `gpt-4o`.). |

### 🚀 Example Usage

```bash
python main.py \
  --strategy what_to_look_for \
  --icl few_shot_both \
  --k 20 \
  --cot True \
  --dev False \
  --model path/to/llama-2-7b-chat \
  --context True \
  --window 1000 \
  --openai False
```

---

## 📊 Expected Results

After running experiments, results will be automatically saved to disk in JSON format with the following structure:

### Standard Evaluation
Results are saved to: `llm_trainclf_jsons/{seed}/{model}/{strategy}/{icl}/{k}/{cot}/{dev}/{context}/{window}/{openai}/{openai_model}/`

Each experiment produces JSON files containing:
- **Classification report**: Per-class precision, recall, and F1-scores for each conspiracy trait
- **Aggregated metrics**: Macro, micro, samples, and weighted averages for:
  - F1-score
  - Precision  
  - Recall

### Relaxed Evaluation
Relaxed evaluation files (`*_relaxed_results.json`) assess whether the model correctly identifies the **dominant trait** when multiple traits are present. This evaluation considers a prediction correct if the model assigns a probability ≥ 0.5 to the dominant trait, providing a less strict measure of model performance focused on identifying the most salient conspiracy trait in each instance.

Both standard and relaxed evaluation results are printed to the console during execution and saved as JSON files for further analysis.
---
## 📄 License

This repository uses dual licensing:
- **Code**: Licensed under the Apache License 2.0 (see `LICENSE`)
- **Data**: Licensed under Creative Commons Attribution 4.0 International (CC-BY-4.0) (see `data/LICENSE-CC-BY-4.0.txt`)
---

## 📖 Citation

If our work was helpful for your work, please be so kind as to cite us:

```bibtex
@article{bates2025conspired,
  title={ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety},
  author={Bates, Luke and Glockner, Max and Nakov, Preslav and Gurevych, Iryna},
  journal={arXiv preprint arXiv:2508.20468},
  year={2025},
  url={https://arxiv.org/abs/2508.20468}
}
```

---

## 👥 Contact

- **Maintainer**: Luke Bates (lukes_first_name.lukes_last_name@tu-darmstadt.de)
- **UKP Lab**: [https://www.ukp.tu-darmstadt.de](https://www.ukp.tu-darmstadt.de)
- **TU Darmstadt**: [https://www.tu-darmstadt.de](https://www.tu-darmstadt.de)

Don't hesitate to send us an email or report an issue if something is broken or if you have further questions.

---

## ⚠️ Disclaimer

This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.