Repository files navigation

^{* عادلة, feminine Arabic given name, meaning just and fair}

Team Formation aims to automate forming teams of experts who can successfully solve difficult tasks. While state-of-the-art neural team formation methods are able to efficiently analyze massive collections of experts to form effective collaborative teams, they largely ignore the fairness in recommended teams of experts. Fairness breeds innovation and increases teams' success by enabling a stronger sense of community, reducing conflict, and stimulating more creative thinking. InAdila, we study the application offairness-aware team formation algorithms to mitigate the potential popularity bias in the neural team formation models. Our experiments show that, first, neural team formation models are biased towardpopular andmale experts. Second, although deterministic re-ranking algorithms mitigatepopularity XORgender bias substantially, they severely hurt the efficacy of teams. On the other hand, probabilistic greedy re-ranking algorithms mitigatepopularity bias significantly and maintain utility. Finally, due to extreme bias in the dataset in terms ofgender, probabilistic greedy re-ranking algorithms also fail to achieve fair and efficient teams.

We have studied the application of state-of-the-artdeterministic greedy re-ranking methods [Geyik et al. KDD'19] in addition toprobabilistic greedy re-ranking methods [Zehlike et al. IP&M'22]to mitigatepopulairty bias andgender bias based onequality of opportunity anddemographic parity notions of fairness for state-of-the-art neural team formation methods fromOpeNTF. Our experiments show that:

• Neural team formation models are biased toward popular experts;
• Although deterministic re-ranking algorithms mitigate bias substantially, they severely hurt the efficacy of teams.
• Probabilistic greedy re-ranking methods are able to mitigate bias while maintaining the utility of the teams as well.

Currently, we are investigating:

• Other fairness factors like demographic attributes, including age, race, and gender;
• Developing machine learning-based models using Learning-to-Rank (L2R) techniques to mitigate bias as opposed to deterministic greedy algorithms.

• 1. Setup
• 2. Quickstart
• 3. Pipeline
  • 3.1. Labeling
  • 3.2. Gender Distribution
  • 3.3. Reranking
  • 3.4. Evaluations
• 4. Result
• 5. Acknowledgement
• 6. License
• 7. Citation

Adila needsPython=3.8 and others packages listed inrequirements.txt:

Bypip, clone the codebase and install the required packages:

git clone https://github.com/Fani-Lab/Adilacd Adilapip install -r requirements.txt

Byconda:

git clone https://github.com/Fani-Lab/Adilacd Adilaconda env create -f environment.ymlconda activate adila

To runAdila, you can use./src/main.py:

cd srcpython -u main.py \  -fteamsvecs ../data/preprocessed/dblp/toy.dblp.v12.json/teamsvecs.pkl \  -fsplit ../output/toy.dblp.v12.json/splits.json \  -fpred ../output/toy.dblp.v12.json/bnn/ \  -np_ratio 0.5 \  -reranker det_cons \  -output ../output/toy.dblp.v12.json/

Where the arguements are:

fteamsvecs: the sparse matrix representation of all teams in a pickle file, including the teams whose members are predicted in--pred. It should contain a dictionary of threelil_matrix with keys[id] of size[#teams × 1],[skill] of size[#teams × #skills],[member] of size[#teams × #experts]. Simply, each row of a metrix shows the occurrence vector of skills and experts in a team. For a toy example, try

import picklewith open(./data/preprocessed/dblp/toy.dblp.v12.json/teamsvecs.pkl) as f: teams=pickle.load(f)

fsplit: the split.json file that indicates the index (rowid) of teams whose members are predicted in--pred. For a toy example, seeoutput/toy.dblp.v12.json/splits.json

fpred: a file or folder that includes the prediction files of a neural team formation methods in the format oftorch.ndarray. The file name(s) should be*.pred and the content is[#test × #experts] probabilities that shows the membership probability of an expert to a team in test set. For a toy example, try

import torchtorch.load(./output/toy.dblp.v12.json/bnn/t31.s11.m13.l[100].lr0.1.b4096.e20.s1/f0.test.pred)

np_ratio: the desirednonpopular ratio among members of predicted teams after mitigation process by re-ranking algorithms. E.g., 0.5.

reranker: fairness-aware reranking algorithm from {det_greedy,det_cons,det_relaxed,fa-ir}. Eg.det_cons.

output: the path to the reranked predictions of members for teams, as well as, the teams' success and fairness evaluationbefore andafter mitigation process.

Adila needs preprocessed information about the teams in the form of sparse matrix representation (-fteamsvecs) and neural team formation prediction file(s) (-fpred), obtained fromOpeNTF:

├── data│   └── preprocessed│       └── dblp│           └── toy.dblp.v12.json│               └── teamsvecs.pkl#sparse matrix representation of teams├── output    └── toy.dblp.v12.json        ├── bnn        │   └── t31.s11.m13.l[100].lr0.1.b4096.e20.s1        │       ├── f0.test.pred        │       ├── f1.test.pred        │       ├── f2.test.pred        └── splits.json#rowids of team instances in n-fold train-valid splits, and a final test split

Adila has three main steps:

Based on the distribution of experts on teams, which is power law (long tail) as shown in the figure, we label those in thetail asnonpopular and those in thehead as popular.

To find the cutoff betweenhead andtail, we calculate the average number of teams per expert over the whole dataset. As seen in the table, this number is62.45 and the popular/nonpopular ratio is0.426/0.574. The result is a Boolean value in{popular: True, nonpopular: False} for each expert and is save in{output}/popularity.csv like./output/toy.dblp.v12.json/bnn/t31.s11.m13.l[100].lr0.1.b4096.e20.s1/rerank/popularity.csv

Future: We will consider equal area under the curve for the cutoff.

The following figures will demonstrate the gender distributions inimdb,dblp anduspt datasets.

We apply rerankers fromdeterministic greedy re-ranking methods [Geyik et al. KDD'19], including{'det_greedy', 'det_cons', 'det_relaxed'} to mitigatepopulairty bias. The reranker needs a cutoffk_max which is set to10 by default.

The result of predictions after reranking is saved in{output}/rerank/{fpred}.{reranker}.{k_max}.rerank.pred like./output/toy.dblp.v12.json/bnn/t31.s11.m13.l[100].lr0.1.b4096.e20.s1/rerank/f0.test.pred.det_cons.10.rerank.pred .

We evaluatefairness andutility metricsbefore andafter applying rerankers on team predictions to answer two research questions (RQs):

RQ1: Do state-of-the-art neural team formation models produce fair teams of experts in terms of popularity bias? To this end, we measure the fairness scores of predicted teamsbefore applying rerankers.

RQ2: Do state-of-the-art deterministic greedy re-ranking algorithms improve the fairness of neural team formation models while maintaining their accuracy? To this end, we measure thefairness andutility metricsbefore andafter applying rerankers.

The result offairness metricsbefore andafter will be stored in{output}.{algorithm}.{k_max}.{faireval}.csv like./output/toy.dblp.v12.json/bnn/t31.s11.m13.l[100].lr0.1.b4096.e20.s1/rerank/f2.test.pred.det_cons.10.faireval.csv .

The result ofutility metricsbefore andafter will be stored in{output}.{algorithm}.{k_max}.{utileval}.csv like./output/toy.dblp.v12.json/bnn/t31.s11.m13.l[100].lr0.1.b4096.e20.s1/rerank/f1.test.pred.det_cons.10.utileval.csv.

Future: We will consider other fairness metrics.

After successful run of all steps,./output contains:

├── output    └── toy.dblp.v12.json        ├── bnn        │   └── t31.s11.m13.l[100].lr0.1.b4096.e20.s1        │       ├── f0.test.pred        │       ├── f1.test.pred        │       ├── f2.test.pred        │       └── rerank/{popularity, gender}        │           ├── f0.test.pred.det_cons.10.faireval.csv        │           ├── f0.test.pred.det_cons.10.utileval.csv        │           ├── f0.test.pred.det_cons.10.rerank.csv        │           ├── f0.test.pred.det_cons.10.rerank.pred        │           ├── f1.test.pred.det_cons.10.faireval.csv        │           ├── f1.test.pred.det_cons.10.utileval.csv        │           ├── f1.test.pred.det_cons.10.rerank.csv        │           ├── f1.test.pred.det_cons.10.rerank.pred        │           ├── f2.test.pred.det_cons.10.faireval.csv        │           ├── f2.test.pred.det_cons.10.utileval.csv        │           ├── f2.test.pred.det_cons.10.rerank.csv        │           ├── f2.test.pred.det_cons.10.rerank.pred        │           ├── labels.csv        │           ├── rerank.time        │           └── stats.pkl        └── splits.json

Our results show that although we improve fairness significantly, our utility metric drops extensively. Part of this phenomenon is described inFairness in Ranking, Part I: Score-Based Ranking [Zehlike et al. ACM Computing Surveys'22]. When we apply representation constraints on individual attributes, like race , popularity and gender and we want to maximize a score with respect to these constraints, utility loss can be particularly significant in historically disadvantaged intersectional groups. The following tables contain the results of our experiments on thebnn,bnn_emb andrandom baselines withgreedy,conservative andrelaxed re-ranking algorithms withdemographic parity fairness notion.

The files containing the rest of our experiment results with various notions, datasets ,and algorithms are as follows:

We benefit frompytrec,reranking, and other libraries. We would like to thank the authors of these libraries and helpful resources.

©2024. This work is licensed under aCC BY-NC-SA 4.0 license.

@inproceedings{DBLP:conf/bias/LoghmaniF23,  author    = {Hamed Loghmani and Hossein Fani},  title     = {Bootless Application of Greedy Re-ranking Algorithms in Fair Neural Team Formation},  booktitle = {Advances in Bias and Fairness in Information Retrieval - Fourth International Workshop, {BIAS} 2023, Dublin, Irland, April 2, 2023, Revised Selected Papers},  pages     = {108--118},  publisher = {Springer Nature Switzerland},  year      = {2023},  url       = {https://doi.org/10.1007/978-3-031-37249-0_9},  doi       = {10.1007/978-3-031-37249-0_9},  bibsource = {dblp computer science bibliography, https://dblp.org}}