Skip to content

Reviewer-Proposal Matching System

The Waldur proposal module includes an automated reviewer-proposal matching system that computes expertise affinity scores and uses them to suggest reviewers and to generate reviewer assignments. This ensures qualified reviewers are matched with proposals in their area of expertise.

Architecture Overview

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
│                           Reviewer-Proposal Matching Architecture                                │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                   REVIEWER DISCOVERY                                             │
│                                                                                                 │
│  ┌─────────────────────┐      ┌─────────────────────┐      ┌─────────────────────┐            │
│  │  Published Profiles │─────▶│  Affinity Algorithm │─────▶│ ReviewerSuggestion  │            │
│  │  (is_published=true)│      │  compute_suggestions │      │    (pending)        │            │
│  └─────────────────────┘      └─────────────────────┘      └──────────┬──────────┘            │
│                                                                        │                       │
│                                              ┌────────────────┬────────┴────────┐              │
│                                              ▼                ▼                 ▼              │
│                                        ┌──────────┐    ┌──────────┐    ┌────────────┐        │
│                                        │ CONFIRMED│    │ REJECTED │    │  INVITED   │        │
│                                        │(approved)│    │(declined)│    │(pool added)│        │
│                                        └──────────┘    └──────────┘    └────────────┘        │
└─────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                │
                                                ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                   AFFINITY SCORING                                               │
│                                                                                                 │
│  ┌─────────────────────┐      ┌─────────────────────┐      ┌─────────────────────┐            │
│  │   Reviewer Profile  │      │  MatchingConfig     │      │     Proposals       │            │
│  │   • Expertise       │      │  • affinity_method  │      │   • Title           │            │
│  │   • Publications    │      │  • keyword_weight   │      │   • Summary         │            │
│  │   • Biography       │      │  • text_weight      │      │   • Description     │            │
│  └──────────┬──────────┘      └──────────┬──────────┘      └──────────┬──────────┘            │
│             │                            │                            │                        │
│             └────────────────────────────┼────────────────────────────┘                        │
│                                          ▼                                                     │
│                           ┌──────────────────────────────┐                                     │
│                           │     Affinity Computation     │                                     │
│                           │  ┌─────────┐   ┌─────────┐  │                                     │
│                           │  │ Keyword │ + │ TF-IDF  │  │                                     │
│                           │  │  Score  │   │  Score  │  │                                     │
│                           │  └─────────┘   └─────────┘  │                                     │
│                           └──────────────┬───────────────┘                                     │
│                                          ▼                                                     │
│                           ┌──────────────────────────────┐                                     │
│                           │  ReviewerProposalAffinity    │                                     │
│                           │  (cached score matrix)       │                                     │
│                           └──────────────────────────────┘                                     │
└─────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                │
                                                ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                   ASSIGNMENT (generate-assignments)                             │
│                                                                                                 │
│  Greedy, per proposal: highest stored affinity first, skipping reviewers with a pending or      │
│  recused conflict and reviewers already assigned; then accepted pool members in pool order.     │
│                                          │                                                     │
│                                          ▼                                                     │
│                           ┌──────────────────────────────┐                                     │
│                           │  AssignmentBatch (draft)     │                                     │
│                           │  → AssignmentItem per pair   │                                     │
│                           └──────────────────────────────┘                                     │
└─────────────────────────────────────────────────────────────────────────────────────────────────┘

Affinity Scoring Methods

The system computes affinity scores between reviewers and proposals using configurable methods.

Keyword-Based Scoring

Matches reviewer expertise keywords against proposal text content.

How it works:

  1. Extracts reviewer's expertise keywords with proficiency weights
  2. Searches proposal text (title, summary, description) for keyword matches
  3. Calculates weighted match score

Proficiency weights:

Proficiency Level Weight
Expert 1.0
Familiar 0.7
Basic 0.3 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Example: Keyword affinity computation
reviewer_keywords = {
    "machine learning": 1.0,      # Expert
    "neural networks": 0.7,       # Familiar
    "data science": 0.3           # Basic
}

proposal_text = "Machine learning approaches for neural network optimization..."
# Matches: "machine learning" (1.0), "neural networks" (0.7)
# Score: (1.0 + 0.7) / (1.0 + 0.7 + 0.3) = 0.85

TF-IDF Text Similarity

Computes semantic similarity between reviewer expertise and proposal content using Term Frequency-Inverse Document Frequency (TF-IDF) vectors.

Reviewer text sources:

  • Expertise keywords (weighted by proficiency)
  • Recent publication titles and abstracts (last 5 years)
  • Biography text

Proposal text sources:

  • Proposal name/title
  • Project summary
  • Project description

Algorithm:

  1. Tokenize text (lowercase, remove stopwords)
  2. Compute TF-IDF vectors for reviewer and proposal
  3. Calculate cosine similarity between vectors
1
2
3
4
5
6
# TF-IDF similarity example
reviewer_vector = {"machine": 0.3, "learning": 0.4, "neural": 0.2, ...}
proposal_vector = {"machine": 0.5, "learning": 0.3, "optimization": 0.4, ...}

# Cosine similarity = dot_product / (magnitude1 * magnitude2)
similarity = 0.72

Combined Method

Default method that combines keyword and text-based scoring with configurable weights.

1
2
3
4
5
affinity_score = (keyword_weight × keyword_score) + (text_weight × text_score)

# Default weights
keyword_weight = 0.4
text_weight = 0.6

Matching Configuration

Each call can have its own matching configuration via MatchingConfiguration:

Field Type Default Description
affinity_method choice combined Scoring method: keyword, tfidf, or combined
keyword_weight float 0.4 Weight for keyword scoring (0-1)
text_weight float 0.6 Weight for TF-IDF scoring (0-1)
min_reviewers_per_proposal int 3 Minimum reviewers per proposal
max_reviewers_per_proposal int 5 Maximum reviewers per proposal
min_proposals_per_reviewer int 3 Minimum proposals per reviewer
max_proposals_per_reviewer int 10 Maximum proposals per reviewer
algorithm choice minmax Reserved; not used by assignment today
min_affinity_threshold float 0.1 Minimum affinity for suggestions
use_reviewer_bids bool true Reserved; bids are not used today
bid_weight float 0.3 Reserved; bids are not used today

Validation: keyword_weight + text_weight must equal 1.0

Assignment

POST /api/proposal-protected-calls/{uuid}/generate-assignments/ creates draft assignment batches (AssignmentBatch, one per reviewer, with an AssignmentItem per reviewer–proposal pair) that the call manager reviews and sends. For each submitted or in-review proposal it picks reviewers in one greedy pass:

  1. Accepted pool members ordered by their stored affinity for the proposal, highest first.
  2. Skipped: reviewers with a pending or recused conflict of interest on the proposal (dismissed and waived conflicts do not block), and reviewers already assigned to it.
  3. If affinity yields too few reviewers, remaining accepted pool members are added in pool order.

The number of reviewers per proposal comes from the request or, if absent, the call's min_reviewers_per_proposal (2 when the call has no matching configuration). There is no global optimisation across proposals: the algorithm setting (minmax, fairflow, hungarian) is stored but not read.

Reviewer Bids

Reviewers can record a preference for each proposal (eager, willing, not_willing, conflict) through /api/reviewer-bids/. Bids are stored and listed only: neither affinity scoring nor assignment reads them, and a conflict bid does not create a conflict-of-interest record. Declare conflicts through COI disclosure instead.

Reviewer Discovery Workflow

The system supports two paths for finding reviewers:

Path A: Algorithm-Based Discovery

For reviewers with published profiles:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
1. Reviewer publishes profile
   ├── POST /api/reviewer-profiles/me/publish/
   └── Sets is_published=true, available_for_reviews=true

2. Manager triggers suggestion generation
   └── POST /api/proposal-protected-calls/{uuid}/generate-suggestions/

3. Algorithm evaluates all published profiles
   ├── Excludes reviewers already in pool
   ├── Excludes reviewers already suggested
   └── Creates ReviewerSuggestion records with affinity scores

4. Manager reviews suggestions
   ├── Confirm: POST /api/reviewer-suggestions/{uuid}/confirm/
   └── Reject: POST /api/reviewer-suggestions/{uuid}/reject/ (with reason)

5. Manager sends invitations to confirmed suggestions
   └── POST /api/proposal-protected-calls/{uuid}/send-invitations/

6. Reviewer views invitation details
   ├── GET /api/reviewer-invitations/{token}/
   └── Returns: call info, COI config, proposals (based on disclosure level)

7. Reviewer accepts/declines with optional COI disclosure
   ├── POST /api/reviewer-invitations/{token}/accept/
   │   └── Can include declared_conflicts for specific proposals
   └── POST /api/reviewer-invitations/{token}/decline/

Path B: Direct Email Invitation

For reviewers without existing profiles:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
1. Manager invites by email
   └── POST /api/proposal-protected-calls/{uuid}/invite-by-email/

2. Invitation sent to email address
   └── CallReviewerPool created (reviewer=null, invited_email set)

3. Invited person clicks invitation link
   ├── GET /api/reviewer-invitations/{token}/
   └── Response indicates profile_status: "missing" or "unpublished"

4. Invited person creates and publishes profile
   ├── POST /api/reviewer-profiles/ (create)
   └── POST /api/reviewer-profiles/me/publish/ (publish)

5. Accept invitation with optional COI disclosure
   ├── POST /api/reviewer-invitations/{token}/accept/
   └── Profile automatically linked to CallReviewerPool

Reviewer Profile Visibility

Reviewer profiles have visibility controls:

Field Default Description
is_published false Profile discoverable by algorithm
available_for_reviews true Currently accepting review requests
published_at null When profile was published

Visibility rules:

  • Own profile: Always visible to the profile owner
  • Pool members: Managers see ACCEPTED pool members only
  • Suggestions: Managers see full profile in suggestion list
  • Discovery: Algorithm only considers published + available profiles

Suggestion Status Workflow

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
   ┌─────────┐
   │ PENDING │◀──────────────── (algorithm generates)
   └────┬────┘
        │
        ├─────────────────────────────────────┐
        ▼                                     ▼
   ┌───────────┐                        ┌──────────┐
   │ CONFIRMED │                        │ REJECTED │
   │ (approved │                        │(declined)│
   │  by mgr)  │                        └──────────┘
   └─────┬─────┘
         │
         ▼
   ┌───────────┐
   │  INVITED  │
   │(invitation│
   │   sent)   │
   └───────────┘

API Endpoints

Affinity Computation

1
POST /api/proposal-protected-calls/{uuid}/compute-affinities/

Computes affinity scores for all reviewer-proposal pairs in the call.

Response:

1
2
3
4
5
{
  "affinities_computed": 150,
  "reviewers": 10,
  "proposals": 15
}

Get Affinity Matrix

1
GET /api/proposal-protected-calls/{uuid}/affinity-matrix/

Returns the complete affinity matrix for visualization.

Response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
  "reviewers": [
    {"uuid": "...", "name": "Dr. Smith"}
  ],
  "proposals": [
    {"uuid": "...", "name": "Proposal A"}
  ],
  "matrix": [
    [0.85, 0.32, 0.67, ...]
  ]
}

Generate Suggestions

1
POST /api/proposal-protected-calls/{uuid}/generate-suggestions/

Runs affinity algorithm on all published profiles to generate reviewer suggestions.

Response:

1
2
3
4
5
{
  "suggestions_created": 15,
  "reviewers_evaluated": 42,
  "suggestions": ["uuid1", "uuid2", "..."]
}

View Suggestions

1
GET /api/proposal-protected-calls/{uuid}/suggestions/

Lists all suggestions for a call with affinity scores.

Filter parameters:

  • status: Filter by status (pending, confirmed, rejected, invited)
  • min_affinity_score: Minimum affinity score (0-1)
  • reviewer_uuid: Filter by specific reviewer

Manage Suggestions

1
2
POST /api/reviewer-suggestions/{uuid}/confirm/
POST /api/reviewer-suggestions/{uuid}/reject/

Manager confirms or rejects suggestions. Rejection requires a reason.

Send Invitations

1
POST /api/proposal-protected-calls/{uuid}/send-invitations/

Sends invitations to all confirmed suggestions.

Invite by Email

1
POST /api/proposal-protected-calls/{uuid}/invite-by-email/

Invites a reviewer by email address (profile not required initially).

Request:

1
2
3
4
5
{
  "email": "reviewer@example.com",
  "invitation_message": "We invite you to review...",
  "max_assignments": 5
}

View Invitation Details

1
GET /api/reviewer-invitations/{token}/

Returns invitation details for reviewers to review before accepting.

Response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
{
  "call": {
    "uuid": "...",
    "name": "2025 Spring HPC Allocation Call",
    "description": "..."
  },
  "invitation_status": "pending",
  "profile_status": "published",
  "requires_profile": false,
  "coi_configuration": {
    "recusal_required_types": ["REL_FAMILY", "FIN_DIRECT"],
    "management_allowed_types": ["COLLAB_ACTIVE", "COAUTH_RECENT"],
    "disclosure_only_types": ["INST_SAME"],
    "proposal_disclosure_level": "titles_and_summaries"
  },
  "coi_types": [["ROLE_NAMED", "Named in proposal"], ...],
  "proposals": [
    {"uuid": "...", "name": "Quantum Computing Research", "summary": "..."}
  ]
}

Accept Invitation

1
POST /api/reviewer-invitations/{token}/accept/

Accepts an invitation with optional COI self-declaration.

Request:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
{
  "declared_conflicts": [
    {
      "proposal_uuid": "<uuid>",
      "coi_type": "COAUTH_RECENT",
      "severity": "apparent",
      "description": "Co-authored 2 papers with PI in 2024"
    }
  ]
}

Notes:

  • declared_conflicts is optional
  • Creates ConflictOfInterest records with detection_method='self_disclosed'
  • Acceptance NOT blocked by declared conflicts (manager handles via COI workflow)
  • For email invitations, requires published profile first

Decline Invitation

1
POST /api/reviewer-invitations/{token}/decline/

Declines a reviewer invitation.

Request:

1
2
3
{
  "reason": "Schedule conflict during review period"
}

Manage Reviewer Bids

1
2
3
GET /api/reviewer-bids/
POST /api/reviewer-bids/
PATCH /api/reviewer-bids/{uuid}/

Manage reviewer preferences for proposals.

Data Models

ReviewerProposalAffinity

Cached affinity scores between reviewers and proposals.

Field Type Description
call FK Call for this affinity
reviewer FK Reviewer profile
proposal FK Proposal
affinity_score float Combined score (0-1)
keyword_score float Keyword-based score
text_score float TF-IDF score

ReviewerSuggestion

Algorithm-generated reviewer suggestions.

Field Type Description
call FK Call for suggestion
reviewer FK Suggested reviewer profile
affinity_score float Combined score (0-1)
keyword_score float Keyword match score
text_score float Text similarity score
status choice pending/confirmed/rejected/invited
reviewed_by FK Manager who reviewed
reviewed_at datetime When reviewed
rejection_reason text Reason for rejection

ProposedAssignment

Reserved for algorithm-proposed assignments. No code path writes it today; the proposed-assignments endpoint only lists it. Assignments are created as AssignmentBatch / AssignmentItem.

Field Type Description
call FK Call for assignment
reviewer FK Assigned reviewer
proposal FK Assigned proposal
affinity_score float Score at assignment time
algorithm_used choice minmax/fairflow/hungarian
rank int Assignment priority (1=best)
is_deployed bool Assignment finalized
deployed_at datetime When deployed
deployed_by FK Who deployed

ReviewerBid

Reviewer preferences for proposals.

Field Type Description
call FK Call
reviewer FK Reviewer profile
proposal FK Proposal
bid choice eager/willing/not_willing/conflict
comment text Optional explanation

Integration with COI Detection

The matching system integrates with Conflict of Interest Detection:

  1. generate-assignments skips reviewers with a pending or recused conflict on the proposal
  2. dismissed and waived conflicts do not block assignment
  3. Reviewer suggestions do not check conflicts; the affinity matrix reports each pair's conflict status
  4. A conflict bid is not a conflict record and is not checked
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
┌──────────────────────────────────────────────────────────────────────────┐
│                    Matching + COI Integration                             │
├──────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────────────────────┐ │
│  │  Affinity   │───▶│ COI Filter  │───▶│ Assignment batches          │ │
│  │ Computation │    │ (exclude    │    │ (suggestions not filtered)  │ │
│  └─────────────┘    │  conflicts) │    └─────────────────────────────┘ │
│                     └─────────────┘                                     │
│                                                                          │
│  Excluded from assignment:                                               │
│  • PENDING conflicts                                                     │
│  • RECUSED reviewers                                                     │
│                                                                          │
│  Not blocking:                                                           │
│  • WAIVED (with management plan) and DISMISSED conflicts                 │
│                                                                          │
└──────────────────────────────────────────────────────────────────────────┘

Performance Considerations

Affinity Computation

  • Computation is O(reviewers × proposals)
  • Results cached in ReviewerProposalAffinity table
  • Recompute when profiles or proposals change significantly

Corpus IDF

  • TF-IDF uses corpus-wide IDF for better term weighting
  • Corpus includes all reviewer texts and proposal texts
  • Computed once per call, reused for all pairs

Large Calls

  • For calls with 100+ proposals and 50+ reviewers:
  • Use batch processing
  • Consider incremental updates
  • Monitor computation time