# promediate **Repository Path**: ipcun/promediate ## Basic Information - **Project Name**: promediate - **Description**: No description available - **Primary Language**: Unknown - **License**: MIT - **Default Branch**: main - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2026-10-01 - **Last Updated**: 2026-10-01 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # ProMediate **A socio-cognitive testbed for evaluating proactive AI mediator agents in multi-party negotiation.** This repository is an open-source, pip-installable implementation of the benchmark described in: > Liu Z., Sarrafzadeh B., Zhou P., Yang L., Zhao J., Sharma A. (2026). > *ProMediate: A Socio-cognitive framework for evaluating proactive agents in multi-party negotiation.* > arXiv:2510.25224 ProMediate ships: - **A simulation environment** that runs goal-directed multi-party, multi-topic negotiations using the InnerThought framework (Liu et al., 2025a) for participant agents and a plug-and-play AI mediator that *decides when and how to intervene*. - **Six negotiation scenarios** drawn from Harvard Law School's Program on Negotiation (PON) materials, each with multiple parties, multiple topics, and multiple options per topic. - **Three difficulty levels** (Easy / Medium / Hard) realised through three conflict modes (Accommodating / Avoiding, no persona, Competing). - **Two reference mediator agents** — a generic Group-Copilot-style baseline and a socially-intelligent mediator grounded in mediation theory. - **A socio-cognitive evaluation suite** with five metrics: Consensus Change (CC), Topic-Level Efficiency (TLE), Response Latency (RL), Mediator Effectiveness (ME), and Mediator Intelligence (MI). The implementation is faithful to the paper: every prompt template in Appendix E is included verbatim in [`promediate/prompts/`](promediate/prompts/), and the consensus tracker is a direct port of Algorithm 1. --- ## Install ```bash pip install -e ".[dev]" # development install with test deps # or, once published: # pip install promediate ``` Set provider keys (any of OpenAI / Anthropic — pick whichever you want to use for which role): ```bash cp .env.example .env # then edit OPENAI_API_KEY / ANTHROPIC_API_KEY ``` ## Quick start List the bundled scenarios: ```bash promediate scenarios list ``` Run one simulation with the socially-intelligent mediator on Hopkins HMO, **Hard** difficulty (Competing participants), GPT-4.1 as mediator, Claude Sonnet-4 as the human simulator, GPT-4.1 as the LLM-as-judge: ```bash promediate run \ --scenario hopkins_hmo \ --difficulty hard \ --mediator social \ --human-model anthropic/claude-sonnet-4-20250514 \ --mediator-model openai/gpt-4.1 \ --judge-model openai/gpt-4.1 \ --output runs/hopkins_hard_social.json ``` This writes a single JSON file containing the full conversation and every metric (CC, TLE, RL, ME, MI, plus the per-turn consensus trajectory). To re-evaluate an existing transcript with a different judge model: ```bash promediate evaluate runs/hopkins_hard_social.json --judge-model openai/gpt-4.1 ``` --- ## Programmatic use ```python from promediate import SimulationConfig, load_scenario, run_and_evaluate from promediate.llm import build_client cfg = SimulationConfig( scenario=load_scenario("francis_hospital"), human_client=build_client("anthropic/claude-sonnet-4-20250514"), mediator_client=build_client("openai/gpt-4.1"), mediator_kind="social", difficulty="medium", max_turns=50, ) result = run_and_evaluate(cfg, judge_client=build_client("openai/gpt-4.1")) print(f"CC: {result.metrics.consensus_change * 100:+.2f}%") print(f"MI: {result.metrics.mediator_intelligence:.2f}") ``` --- ## Reproducing the paper | Setting | CLI flags | |----------------------|-----------------------------------------------------------| | `PROMEDIATE-Easy` | `--difficulty easy` (Accommodating / Avoiding parties) | | `PROMEDIATE-Medium` | `--difficulty medium` (no fixed persona) | | `PROMEDIATE-Hard` | `--difficulty hard` (Competing parties) | | `NoAgent` baseline | `--mediator none` | | Generic mediator | `--mediator generic` | | Social mediator | `--mediator social` | The paper reports five independent runs per scenario and conflict mode; just loop over `--scenario` (six values) and re-run. > **Note on scenario fidelity** — the paper draws full negotiation briefs from > the Harvard PON Store (each is 2–3 hours of student materials and is *not* > redistributable). Each bundled scenario in `promediate/scenarios/data/` > contains: > > - The verbatim background, issues, and option lists from Appendix B.1 > (which is freely included in the paper). > - Reasonable placeholder party identities and preference rankings derived > from the scenario's stated structure. To run a fully PON-aligned > experiment, replace the per-party `description`, `main_goal`, and > `preferences` fields with the official confidential briefs. --- ## Architecture ``` promediate/ ├── prompts/ # Verbatim Appendix-E prompts (Tables 7-18) ├── llm/ # Provider-neutral LLMClient + OpenAI / Anthropic backends ├── scenarios/ # 6 negotiation scenarios (JSON) + loader ├── agents/ # Participant (InnerThought), GenericMediator, SocialMediator ├── simulator/ # Per-turn orchestration: when -> who -> how ├── metrics/ # Algorithm 1 (consensus tracker) + CC, TLE, RL, ME, MI ├── runner.py # Convenience: simulate + evaluate -> RunResult └── cli.py # `promediate` command-line interface ``` ### Per-turn loop 1. The mediator (if any) is consulted first via `decide_when`. If it elects to intervene, it speaks; this turn ends. 2. Otherwise every participant generates `num_thoughts` candidate thoughts (Table 10), a meta-evaluator scores them for speaking motivation, and the participant whose top thought has the highest score speaks next (InnerThought, Section B.2). 3. The chosen utterance is appended to the recorded conversation and the loop advances. The socially-intelligent mediator uses the four-stage pipeline from Section 4.1: **when** ➜ **generate three candidate strategies** ➜ **evaluate each** ➜ **articulate the highest-scoring strategy**. The generic mediator only does **when** ➜ **how** (Table 11). ### Metrics - **Consensus tracker** ([`metrics/tracker.py`](promediate/metrics/tracker.py)) is a direct port of Algorithm 1: per-party / per-topic attitudes are extracted from each utterance with the Table-16 prompt and pairwise agreement is scored with Table 17 across five socio-cognitive dimensions. - **Consensus Change (CC)** uses 10-turn windowed averages. - **Topic-Level Efficiency (TLE)** = per-topic consensus delta divided by the number of mention turns. - **Response Latency (RL)** counts turns between a *drop event* (consensus falling by `tau=0.1` within the next `W=10` turns) and the mediator's next utterance. - **Mediator Effectiveness (ME)** = post-intervention slope minus pre-intervention slope of per-topic consensus around each mediator turn (5-turn windows). - **Mediator Intelligence (MI)** uses the Table-18 LLM-as-judge to rate every mediator turn 1–5 on Perception alignment, Emotional dynamics, Cognitive challenges, and Communication breakdowns. --- ## Tests ```bash pytest -q ``` End-to-end tests use a `FakeLLM` (`tests/conftest.py`) so no API keys are required for CI. --- ## Citation ```bibtex @misc{liu2026promediate, title = {ProMediate: A Socio-cognitive framework for evaluating proactive agents in multi-party negotiation}, author = {Liu, Ziyi and Sarrafzadeh, Bahar and Zhou, Pei and Yang, Longqi and Zhao, Jieyu and Sharma, Ashish}, year = {2026}, eprint = {2510.25224}, archivePrefix = {arXiv}, primaryClass = {cs.CL} } ``` ## License [MIT](LICENSE). This implementation is independent of the original authors. Negotiation materials remain the property of the Harvard Program on Negotiation; the bundled scenario JSON files only include the publicly-printed appendix descriptions, structural metadata, and placeholder identities.