Effects of performance feedback on users' evaluations of an interactive IR system

Diane Kelly, Chirag Shah, Cassidy R. Sugimoto, Earl W. Bailey, Rachael A. Clemens, Ann K. Irvine, Nicholas A. Johnson, Weimao Ke, Sanghee Oh, Anezka Poljakova, Marcos A. Rodriguez, Megan G. Van Noord, Yan Zhang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

In this study, we seek to understand how providing feedback to users about their performances with an interactive information retrieval (IIR) system impacts their evaluations of that system. Sixty subjects completed three recall-based searching tasks with an experimental IIR system and were asked to evaluate the system after each task and after finishing all three tasks. Before completing the final evaluation, three-fourths of the subjects were provided with feedback about their performances. Subjects were assigned randomly to one of four feedback conditions: a baseline condition where no feedback was provided; an actual feedback condition where subjects were provided with their real performances; and two conditions where subjects were deceived and told that they performed very well or very poorly. Results show that the type of feedback provided significantly affected subjects' system evaluations; most importantly there was a significant difference in subjects' satisfaction ratings before and after feedback was provided in the actual feedback condition.

Original languageEnglish (US)
Title of host publicationIIiX'08
Subtitle of host publicationProceedings of the 2nd International Symposium on Information Interaction in Context
Pages75-82
Number of pages8
DOIs
StatePublished - Dec 1 2008
Externally publishedYes
Event2nd International Symposium on Information Interaction in Context 2008, IIiX'08 - London, United Kingdom
Duration: Oct 14 2008Oct 17 2008

Publication series

NameIIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context

Conference

Conference2nd International Symposium on Information Interaction in Context 2008, IIiX'08
CountryUnited Kingdom
CityLondon
Period10/14/0810/17/08

Fingerprint

Feedback
Information retrieval systems

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Information Systems

Cite this

Kelly, D., Shah, C., Sugimoto, C. R., Bailey, E. W., Clemens, R. A., Irvine, A. K., ... Zhang, Y. (2008). Effects of performance feedback on users' evaluations of an interactive IR system. In IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context (pp. 75-82). (IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context). https://doi.org/10.1145/1414694.1414712

Effects of performance feedback on users' evaluations of an interactive IR system. / Kelly, Diane; Shah, Chirag; Sugimoto, Cassidy R.; Bailey, Earl W.; Clemens, Rachael A.; Irvine, Ann K.; Johnson, Nicholas A.; Ke, Weimao; Oh, Sanghee; Poljakova, Anezka; Rodriguez, Marcos A.; Van Noord, Megan G.; Zhang, Yan.

IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context. 2008. p. 75-82 (IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Kelly, D, Shah, C, Sugimoto, CR, Bailey, EW, Clemens, RA, Irvine, AK, Johnson, NA, Ke, W, Oh, S, Poljakova, A, Rodriguez, MA, Van Noord, MG & Zhang, Y 2008, Effects of performance feedback on users' evaluations of an interactive IR system. in IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context. IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context, pp. 75-82, 2nd International Symposium on Information Interaction in Context 2008, IIiX'08, London, United Kingdom, 10/14/08. https://doi.org/10.1145/1414694.1414712
Kelly D, Shah C, Sugimoto CR, Bailey EW, Clemens RA, Irvine AK et al. Effects of performance feedback on users' evaluations of an interactive IR system. In IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context. 2008. p. 75-82. (IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context). https://doi.org/10.1145/1414694.1414712
Kelly, Diane ; Shah, Chirag ; Sugimoto, Cassidy R. ; Bailey, Earl W. ; Clemens, Rachael A. ; Irvine, Ann K. ; Johnson, Nicholas A. ; Ke, Weimao ; Oh, Sanghee ; Poljakova, Anezka ; Rodriguez, Marcos A. ; Van Noord, Megan G. ; Zhang, Yan. / Effects of performance feedback on users' evaluations of an interactive IR system. IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context. 2008. pp. 75-82 (IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context).
@inproceedings{632440b0fb6c4a0f94992acf758b4dac,
title = "Effects of performance feedback on users' evaluations of an interactive IR system",
abstract = "In this study, we seek to understand how providing feedback to users about their performances with an interactive information retrieval (IIR) system impacts their evaluations of that system. Sixty subjects completed three recall-based searching tasks with an experimental IIR system and were asked to evaluate the system after each task and after finishing all three tasks. Before completing the final evaluation, three-fourths of the subjects were provided with feedback about their performances. Subjects were assigned randomly to one of four feedback conditions: a baseline condition where no feedback was provided; an actual feedback condition where subjects were provided with their real performances; and two conditions where subjects were deceived and told that they performed very well or very poorly. Results show that the type of feedback provided significantly affected subjects' system evaluations; most importantly there was a significant difference in subjects' satisfaction ratings before and after feedback was provided in the actual feedback condition.",
author = "Diane Kelly and Chirag Shah and Sugimoto, {Cassidy R.} and Bailey, {Earl W.} and Clemens, {Rachael A.} and Irvine, {Ann K.} and Johnson, {Nicholas A.} and Weimao Ke and Sanghee Oh and Anezka Poljakova and Rodriguez, {Marcos A.} and {Van Noord}, {Megan G.} and Yan Zhang",
year = "2008",
month = "12",
day = "1",
doi = "10.1145/1414694.1414712",
language = "English (US)",
isbn = "9781605583105",
series = "IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context",
pages = "75--82",
booktitle = "IIiX'08",

}

TY - GEN

T1 - Effects of performance feedback on users' evaluations of an interactive IR system

AU - Kelly, Diane

AU - Shah, Chirag

AU - Sugimoto, Cassidy R.

AU - Bailey, Earl W.

AU - Clemens, Rachael A.

AU - Irvine, Ann K.

AU - Johnson, Nicholas A.

AU - Ke, Weimao

AU - Oh, Sanghee

AU - Poljakova, Anezka

AU - Rodriguez, Marcos A.

AU - Van Noord, Megan G.

AU - Zhang, Yan

PY - 2008/12/1

Y1 - 2008/12/1

N2 - In this study, we seek to understand how providing feedback to users about their performances with an interactive information retrieval (IIR) system impacts their evaluations of that system. Sixty subjects completed three recall-based searching tasks with an experimental IIR system and were asked to evaluate the system after each task and after finishing all three tasks. Before completing the final evaluation, three-fourths of the subjects were provided with feedback about their performances. Subjects were assigned randomly to one of four feedback conditions: a baseline condition where no feedback was provided; an actual feedback condition where subjects were provided with their real performances; and two conditions where subjects were deceived and told that they performed very well or very poorly. Results show that the type of feedback provided significantly affected subjects' system evaluations; most importantly there was a significant difference in subjects' satisfaction ratings before and after feedback was provided in the actual feedback condition.

AB - In this study, we seek to understand how providing feedback to users about their performances with an interactive information retrieval (IIR) system impacts their evaluations of that system. Sixty subjects completed three recall-based searching tasks with an experimental IIR system and were asked to evaluate the system after each task and after finishing all three tasks. Before completing the final evaluation, three-fourths of the subjects were provided with feedback about their performances. Subjects were assigned randomly to one of four feedback conditions: a baseline condition where no feedback was provided; an actual feedback condition where subjects were provided with their real performances; and two conditions where subjects were deceived and told that they performed very well or very poorly. Results show that the type of feedback provided significantly affected subjects' system evaluations; most importantly there was a significant difference in subjects' satisfaction ratings before and after feedback was provided in the actual feedback condition.

UR - http://www.scopus.com/inward/record.url?scp=62949084183&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=62949084183&partnerID=8YFLogxK

U2 - 10.1145/1414694.1414712

DO - 10.1145/1414694.1414712

M3 - Conference contribution

AN - SCOPUS:62949084183

SN - 9781605583105

T3 - IIiX'08: Proceedings of the 2nd International Symposium on Information Interaction in Context

SP - 75

EP - 82

BT - IIiX'08

ER -