A primer on gathering and analysing multi-level quantitative evidence for differential student outcomes in higher education

Article


Balloo, Kieran and Winstone, Naomi E.. 2021. "A primer on gathering and analysing multi-level quantitative evidence for differential student outcomes in higher education." Frontline Learning Research. 9 (2), pp. 121-144. https://doi.org/10.14786/flr.v9i2.675
Article Title

A primer on gathering and analysing multi-level quantitative evidence for differential student outcomes in higher education

ERA Journal ID212580
Article CategoryArticle
AuthorsBalloo, Kieran (Author) and Winstone, Naomi E. (Author)
Journal TitleFrontline Learning Research
Journal Citation9 (2), pp. 121-144
Number of Pages24
Year2021
PublisherEuropean Association for Research on Learning and Instruction
Place of PublicationBelgium
ISSN2295-3159
Digital Object Identifier (DOI)https://doi.org/10.14786/flr.v9i2.675
Web Address (URL)https://journals.sfu.ca/flr/index.php/journal/article/view/675
Abstract

A significant challenge currently facing the higher education sector is how to address differential student outcomes in terms of attainment and continuation gaps at various stages of students’ transitions. Worryingly, there appears to be a ‘deficit’ discourse among some university staff in which differential outcomes are perceived to be due to student deficiencies. This may be exacerbated by institutional analyses placing an over-emphasis on the presence of the gaps rather than the causes. The purpose of this primer is to provide advice about how institutions can carry out far more nuanced analyses of their institutional data without requiring specialist software or expertise. Drawing on a multi-level framework for explaining differential outcomes, we begin with guidance for gathering quantitative data on explanatory factors for attainment and continuation gaps, largely by linking sources of internal data that have not previously been connected. Using illustrative examples, we then provide tutorials for how to model explanatory factors employing IBM SPSS Statistics (IBM Corp., Armonk, NY, USA) to perform and interpret regression and meta-regression analyses of individual- and group-level (aggregated) student data, combined with data on micro- and meso-level factors. We propose that university staff with strategic responsibilities could use these approaches with their institutional data, and the findings could then inform the design of context-specific interventions that focus on changing practices associated with gaps. In doing so, institutions could enhance the evidence-base, raise awareness, and further ‘embed the agenda’ when it comes to understanding potential reasons for differential student outcomes during educational transitions.

KeywordsAttainment gaps; Continuation gaps; Diversity; Micro, meso and macro level; Transitions
ANZSRC Field of Research 2020390303. Higher education
Byline AffiliationsUniversity of Surrey, United Kingdom
Institution of OriginUniversity of Southern Queensland
Permalink -

https://research.usq.edu.au/item/q6z9w/a-primer-on-gathering-and-analysing-multi-level-quantitative-evidence-for-differential-student-outcomes-in-higher-education

  • 76
    total views
  • 4
    total downloads
  • 3
    views this month
  • 0
    downloads this month

Export as

Related outputs

Trends and global power of research on physical activity, depression, and anxiety in adolescents and young adults: Science mapping the literature
Balloo, Kieran, Delaney, Patrick, Hosein, Anesa and Liza, Farhana Ferdousi. 2024. "Trends and global power of research on physical activity, depression, and anxiety in adolescents and young adults: Science mapping the literature." International Journal of Adolescence and Youth. 29 (1). https://doi.org/10.1080/02673843.2024.2375356
Research trends in the higher education feedback literature since the ‘new paradigm’ shift to learner-centred feedback practices
Balloo, Kieran. 2023. "Research trends in the higher education feedback literature since the ‘new paradigm’ shift to learner-centred feedback practices." Society for Research into Higher Education (SRHE) International Conference 2023. Birmingham, United Kingdom 06 - 08 Dec 2023 United Kingdom.
True encounters with the fictional university: collectively rewriting the script of filmic dark academia from the academic margins
Hopkins, Susan, Balloo, Kieran, Ramos, Fabiane, Salmeron, Raquel, Singh, Niharika and Wilson, Victoria E.. 2024. "True encounters with the fictional university: collectively rewriting the script of filmic dark academia from the academic margins ." Culture and Organization. https://doi.org/10.1080/14759551.2024.2394461
The many colours of student support: perspectives and practice of enabling education practitioners
Westacott, Marguerite, Subramaniam, Juliette, Mann, Gemma, Larsen, Ana, James, Trixie, Hopkins, Susan and Balloo, Kieran. 2024. "The many colours of student support: perspectives and practice of enabling education practitioners." 9th Students Transitions Achievement Retention & Success Conference (STARS 2024) . Melbourne, Australia 01 - 03 Jul 2024 Australia.
Defining and Delimiting Student Support: Insights and Challenges in Enhancing Support and Wellbeing for Educators and Students
Hopkins, S., James, T., Larsen, A., Balloo, K., Westacott, M., Mann, G. and Subramaniam, J.. 2024. "Defining and Delimiting Student Support: Insights and Challenges in Enhancing Support and Wellbeing for Educators and Students." National Association of Enabling Educators of Australia 2024 (NAEEA 2024) Webinar Series. Australia.
Surfacing the complexity of students’ experiences of assessment and feedback processes using a rich picture approach
Balloo, Kieran, Barnett, Laura, Gravett, Karen, Ali, Xeina, Tatam, James and Winstone, Naomi E.. 2024. "Surfacing the complexity of students’ experiences of assessment and feedback processes using a rich picture approach." Journal of Further and Higher Education. 48 (4), pp. 467-480. https://doi.org/10.1080/0309877X.2024.2347535
Efficacy of an Antimicrobial Reality Simulator (AMRSim) as an Educational Tool for Teaching Antimicrobial Stewardship to Veterinary Medicine Undergraduates
Subasinghe, Dona Wilani Dynatra, Balloo, Kieran, Dale, Emily, Lygo-Baker, Simon, La Ragione, Roberto La and Chambers, Mark. 2024. "Efficacy of an Antimicrobial Reality Simulator (AMRSim) as an Educational Tool for Teaching Antimicrobial Stewardship to Veterinary Medicine Undergraduates." Journal of Veterinary Medical Education. https://doi.org/10.3138/jvme-2023-0148
Assessment design to support sound academic integrity practices for Pathways students
Judith, Kate, Balloo, Kieran, Chang,Heejin, Park, Sang-Soon and Padro, Fernando. 2023. "Assessment design to support sound academic integrity practices for Pathways students." A New Agenda for Pathways Education and Research Colloquium. University of Southern Queensland 26 - 26 Oct 2023
Exploring the identities of pathways educators through the lens of Third Space Theory
Balloo, Kieran, Ramos, Fabiane, Crank, Russell, Crane, Daniel, Hopkins, Susan, McGovern, Mary, Parkes, Frey, Penno, Julie, Singh, Niharika, Todd, Nicholas, Wilson, Victoria, Windsor, Angela and Worsley, Sue. 2023. "Exploring the identities of pathways educators through the lens of Third Space Theory." Pedagogy Culture and Society. https://doi.org/10.1080/14681366.2023.2230986
Measuring psychological wellbeing and mental health in university student cohorts
Heron, Paul, Balloo, Kieran, Barkham, Michael, Bennett, acks, Berry, Clio, Bewick, Bridgette, Dutta, Snigdha, Edwards, Lisa, Foster, Juliet, Gardani, Maria, Hosein, Anesa, Knowles, Louise, Kumar, Sanjay, Linton, Myles-Jay, McLafferty, Margaret, Lucock, Mike, McMillan, Dean, Paton, Lewis, Rees, Sarah, ..., Broglia, Emma. 2023. Measuring psychological wellbeing and mental health in university student cohorts. United Kingdom. SMaRteN.
Observational feedback literacy: designing post observation feedback for learning
Heron, Marion, O’Donaghue, Helen and Balloo, Kieran. 2023. "Observational feedback literacy: designing post observation feedback for learning." Teaching in Higher Education. https://doi.org/10.1080/13562517.2023.2191786
The role of the university environment in shaping education and employment inequalities
Hosein, Anesa, Balloo, Kieran, Byrom, Nicola and Essau, Cecilia A.. 2023. "The role of the university environment in shaping education and employment inequalities." Journal of Higher Education Policy and Management. 45 (2), pp. 223-242. https://doi.org/10.1080/1360080X.2023.2180170
Poems and pedagogic frailty: uncovering the affective within teacher development through collective biography
Kinchin, Ian, Balloo, Kieran, Barnett, Laura, Gravett, Karen, Heron, Marion, Hosein, Anesa, Lygo-Baker, Simon, Medland, Emma, Winstone, Naomi and Yakovchuk, Nadya. 2023. "Poems and pedagogic frailty: uncovering the affective within teacher development through collective biography." Arts and Humanities in Higher Education: an international journal of theory, research and practice. 22 (3), pp. 305-321. https://doi.org/10.1177/14740222221147483
Who stands to benefit? Wellbeing, belonging and challenges to equity in engagement in extra-curricular activities at university
Winstone, Naomi, Balloo, Kieran, Gravett, Karen, Jacobs, Daniel and Keen, Harry. 2022. "Who stands to benefit? Wellbeing, belonging and challenges to equity in engagement in extra-curricular activities at university." Active Learning in Higher Education. 23 (2), p. 81–96. https://doi.org/10.1177/1469787420908209
The development of academics’ feedback literacy: experiences of learning from critical feedback via scholarly peer review
Gravett, Karen, Kinchin, Ian M., Winstone, Naomi E., Balloo, Kieran, Heron, Marion, Hosein, Anesa, Lygo-Baker, Simon and Medland, Emma. 2020. "The development of academics’ feedback literacy: experiences of learning from critical feedback via scholarly peer review." Assessment and Evaluation in Higher Education. 45 (5), pp. 651-665. https://doi.org/10.1080/02602938.2019.1686749
Discipline‑specific feedback literacies: A framework for curriculum design
Winstone, Naomi E., Balloo, Kieran and Carless, David. 2022. "Discipline‑specific feedback literacies: A framework for curriculum design." Higher Education. 83 (1), pp. 57-77. https://doi.org/10.1007/s10734-020-00632-0
The focus, function and framing of feedback information: linguistic and content analysis of intext feedback comments
Derham, Cathrine, Balloo, Kieran and Winstone, Naomi. 2022. "The focus, function and framing of feedback information: linguistic and content analysis of intext feedback comments." Assessment and Evaluation in Higher Education. 47 (6), p. 896–909. https://doi.org/10.1080/02602938.2021.1969335
‘I’m not sure where home is’: narratives of student mobilities into and through higher education
Balloo, Kieran, Gravett, Karen and Erskine, Gemma. 2021. "‘I’m not sure where home is’: narratives of student mobilities into and through higher education." British Journal of Sociology of Education. 42 (7), pp. 1022-1036. https://doi.org/10.1080/01425692.2021.1959298
Mapping the values and identities of academics who teach into widening participation university preparation programs
Balloo, Kieran, Crane, Daniel, Crank, Russell, Hopkins, Susan, McGovern, Mary, Parkes, Geoffrey, Penno, Julie, Ramos, Fabiane, Singh, Niharika, Todd, Nicholas, Wilson, Victoria, Windsor, Angela and Worsley, Sue. 2022. "Mapping the values and identities of academics who teach into widening participation university preparation programs." Society for Research into Higher Education International Research (SRHE) Conference 2022. 05 - 09 Dec 2022 United Kingdom.
Differences in mental health inequalities based on university attendance: Intersectional multilevel analyses of individual heterogeneity and discriminatory accuracy
Balloo, Kieran, Hosein, Anesa, Byrom, Nicola and Essau, Cecilia A.. 2022. "Differences in mental health inequalities based on university attendance: Intersectional multilevel analyses of individual heterogeneity and discriminatory accuracy." Social Science and Medicine – Population Health. 19, pp. 1-10. https://doi.org/10.1016/j.ssmph.2022.101149
An intersectional analysis of inequalities in young people’s mental health within the higher education context
Balloo, Kieran, Hosein, Anesa, Byrom, Nicola and Essau, Cecilia A.. 2022. "An intersectional analysis of inequalities in young people’s mental health within the higher education context." Health Studies User Conference (2022). Manchester, United Kingdom 12 Jul 2022 Manchester, United Kingdom.
Mental distress during adolescence and participation in higher education
Balloo, Kieran, Hosein, Anesa, Byrom, Nicola and Essau, Cecilia A.. 2022. "Mental distress during adolescence and participation in higher education." 44th Higher Education Research and Development Society of Australasia Annual Conference (HERDSA 2022). Melbourne, Australia 27 - 30 Jun 2022 Australia.
Precedents to feedback literacy: using an evidence-based approach to reflect on feedback talk
Medland, Emma, Heron, Marion and Balloo, Kieran. 2022. "Precedents to feedback literacy: using an evidence-based approach to reflect on feedback talk." International Assessment in Higher Education Conference (AHE 2022). Manchester, United Kingdom 22 - 24 Jun 2022 Manchester, United Kingdom.
Maximising student success through the development of self-regulation
Evans, Carol, Zhu, Xiaotong, Winstone, Naomi, Balloo, Kieran, Hughes, Annie and Bright, Chantal. 2019. Maximising student success through the development of self-regulation. United Kingdom. University of Southampton. https://doi.org/10.13140/RG.2.2.21930.44481
Academic pressures and mental distress during adolescence: associations with participation in higher education
Balloo, Kieran, Hosein, Anesa, Byrom, Nicola and Essau, Cecilia A.. 2021. "Academic pressures and mental distress during adolescence: associations with participation in higher education." Symposium Conducted at the International Conference on Research into Higher Education 2021: (Re)connecting, (Re)building: Higher Education in Transformative Times. United Kingdom 06 - 10 Dec 2021 United Kingdom.
Precedents to feedback literacy: uncoupling feedback from assessment using an evidence-based approach to reflect on and validate the Feedback Talk Framework
Medland, Emma, Heron, Marion and Balloo, Kieran. 2021. "Precedents to feedback literacy: uncoupling feedback from assessment using an evidence-based approach to reflect on and validate the Feedback Talk Framework." SRHE Conference 2021: (Re)connecting, (Re)building: Higher Education in Transformative Times. United Kingdom 06 - 10 Dec 2021 United Kingdom.
Efficacy of a Microbial Reality Simulator (AMRSim) as an education tool for antimicrobial stewardship teaching for veterinary undergraduates
Subasinghe, D.W., Sofokleous, S., Howgate, M., Bartlett, K., Trace, C., Balloo, K., Lygo-Baker, S., Cockcroft, P., Wyles, K., Macdonald, A. and Chambers, M.. 2020. "Efficacy of a Microbial Reality Simulator (AMRSim) as an education tool for antimicrobial stewardship teaching for veterinary undergraduates." GW4 - Multidisciplinary approaches to AMR. Virtual 30 Nov - 04 Dec 2020
Managing complex assessment interventions: research within research
Evans, Carol A., Winstone, Naomi E., Hughes, Annie, Zhu, Xiaotong, Balloo, Kieran and Kyei, Chantal. 2018. "Managing complex assessment interventions: research within research." Society for Research into Higher Education International Research (SRHE) Conference 2019 . Newport, United Kingdom 05 - 07 Dec 2018 United Kingdom.
Evaluation of a large-scale inclusive assessment intervention: a novel approach to quantifying perceptions about assessment literacy
Balloo, Kieran, Norman, Maria and Winstone, Naomi E.. 2018. "Evaluation of a large-scale inclusive assessment intervention: a novel approach to quantifying perceptions about assessment literacy." Society for Research into Higher Education International Research (SRHE) Conference 2019 . Newport, United Kingdom 05 - 07 Dec 2018 United Kingdom.
The intervening role of assessment literacy in relationships between feedback literacy and self-regulation
Balloo, Kieran, Winstone, Naomi E. and Evans, Carol. 2018. "The intervening role of assessment literacy in relationships between feedback literacy and self-regulation." 9th Biennial Conference of the European Association for Research on Learning and Instruction SIG1: Assessment and Evaluation (EARLI 2018). Helsinki, Finland 29 - 31 Aug 2018 https://doi.org/10.13140/RG.2.2.27769.93283
Undergraduates’ conceptions of research methods learning
Balloo, Kieran. 2017. "Undergraduates’ conceptions of research methods learning." 2017 Biennial European Association for Research on Learning and Instruction (EARLI 2017). Tampere, Finland 29 Aug - 02 Sep 2017 Belgium.
In-depth profiles of the expectations of undergraduate students commencing university: a Q methodological analysis
Balloo, Kieran. 2018. "In-depth profiles of the expectations of undergraduate students commencing university: a Q methodological analysis." Studies in Higher Education. 43 (12), pp. 2251-2262. https://doi.org/10.1080/03075079.2017.1320373
Effectiveness of the Super Skills for Life (SSL) programme in enhancing children’s emotional wellbeing, emotional regulation and cognitive control: a randomized controlled trial study in Mauritius
Ramdhonee-Dowlot, Karishma, S., Essau, Cecilia A. and Balloo, Kieran. 2019. "Effectiveness of the Super Skills for Life (SSL) programme in enhancing children’s emotional wellbeing, emotional regulation and cognitive control: a randomized controlled trial study in Mauritius." British Educational Research Association Annual Conference 2019 (BERA 2019). London, United Kingdom 10 - 12 Sep 2019
A systematic review of interventions targeting self-regulation in higher education
Balloo, Kieran, Panadero, Ernesto and Winstone, Naomi. 2020. "A systematic review of interventions targeting self-regulation in higher education." 2020 AERA Annual Meeting [Cancelled]. San Francisco, United States 17 - 21 Apr 2020 United States.
A crucial graduate attribute? Embedding feedback literacy into curricula across higher education
Balloo, Kieran, Winstone, Naomi E. and Carless, David. 2019. "A crucial graduate attribute? Embedding feedback literacy into curricula across higher education." Society for Research into Higher Education International Research (SRHE) Conference 2019 . Newport, United Kingdom 05 - 07 Dec 2018 United Kingdom.
‘What’s the point?’ Do annotations on students’ work promote self-regulation?
Derham, Cathrine, Balloo, Kieran, Norman, Maria and Winstone, Naomi E.. 2019. "‘What’s the point?’ Do annotations on students’ work promote self-regulation?" 18th Biennial Conference of the European Association for Research on Learning and Instruction (EARLI 2019). Aachen, Germany 12 - 16 Aug 2019 Germany.
Redefining scientific thinking for higher education: higher-order thinking, evidence-based reasoning and research skills
Murtonen, Mari and Balloo, Kieran. 2019. Redefining scientific thinking for higher education: higher-order thinking, evidence-based reasoning and research skills. Cham, Switzerland. Palgrave Macmillan.
Exploring ‘excellence’ in disciplinary contexts through student-staff partnerships
Williams, Sophie, Pricop, Malina, Heron, Marion, Balloo, Kieran and Barnett, Laura. 2021. "Exploring ‘excellence’ in disciplinary contexts through student-staff partnerships." Heron, Marion, Barnett, Laura and Balloo, Kieran (ed.) Exploring disciplinary teaching excellence in higher education: student-staff partnerships for research. Cham, Switzerland. Palgrave Macmillan. pp. 1-20
Exploring disciplinary teaching excellence in higher education: student-staff partnerships for research
Heron, Marion, Barnett, Laura and Balloo, Kieran. 2021. Exploring disciplinary teaching excellence in higher education: student-staff partnerships for research. Cham, Switzerland. Palgrave Macmillan.
Creating ‘excellent’ partnerships: reflections on practice
Barnett, Laura, Heron, Marion and Balloo, Kieran. 2021. "Creating ‘excellent’ partnerships: reflections on practice." Heron, Marion, Barnett, Laura and Balloo, Kieran (ed.) Exploring disciplinary teaching excellence in higher education: student-staff partnerships for research. Cham, Switzerland. Palgrave Macmillan. pp. 199-208
Students’ difficulties during research methods training acting as potential barriers to their development of scientific thinking
Balloo, Kieran. 2019. "Students’ difficulties during research methods training acting as potential barriers to their development of scientific thinking." Murtonen, Mari and Balloo, Kieran (ed.) Redefining scientific thinking for higher education. Cham, Switzerland. Palgrave Macmillan. pp. 107-137
Facilitating students’ proactive recipience of feedback with feedback portfolios
Balloo, Kieran and Vashakidze, Aka. 2020. "Facilitating students’ proactive recipience of feedback with feedback portfolios." Gravett, Karen, Yakovchuk, Nadya and Kinchin, Ian M. (ed.) Enhancing student-centred teaching in higher education: the landscape of student-staff research partnerships. Cham, Switzerland. Palgrave Macmillan. pp. 255-272
Individual Differences in Psychology Undergraduates’ Development of Research Methods Knowledge and Skills
Balloo, Kieran, Pauli, Regina and Worrell, Marcia. 2016. "Individual Differences in Psychology Undergraduates’ Development of Research Methods Knowledge and Skills." Procedia: Social and Behavioral Sciences. 217, pp. 790-800. https://doi.org/10.1016/j.sbspro.2016.02.147
Transparency Isn't Spoon-Feeding: How a Transformative Approach to the Use of Explicit Assessment Criteria Can Support Student Self-Regulation
Balloo, Kieran, Evans, Carol, Hughes, Annie, Zhu, Xiaotong and Winstone, Naomi. 2018. "Transparency Isn't Spoon-Feeding: How a Transformative Approach to the Use of Explicit Assessment Criteria Can Support Student Self-Regulation." Frontiers in Education. 3, pp. 1-11. https://doi.org/10.3389/feduc.2018.00069
Conceptions of Research Methods Learning Among Psychology Undergraduates: A Q Methodology Study
Balloo, Kieran, Pauli, Regina and Worrell, Marcia. 2018. "Conceptions of Research Methods Learning Among Psychology Undergraduates: A Q Methodology Study." Cognition and Instruction. 36 (4), pp. 279-296. https://doi.org/10.1080/07370008.2018.1494180
Effectiveness of the Super Skills for Life programme in enhancing the emotional wellbeing of children and adolescents in residential care institutions in a low- and middle-income country: A randomised waitlist-controlled trial
Ramdhonee-Dowlot, Karishma, Balloo, Kieran and Essau, Cecilia A.. 2021. "Effectiveness of the Super Skills for Life programme in enhancing the emotional wellbeing of children and adolescents in residential care institutions in a low- and middle-income country: A randomised waitlist-controlled trial." Journal of Affective Disorders. 278, pp. 327-338. https://doi.org/10.1016/j.jad.2020.09.053
Individual differences in research methods learning among undergraduate psychology students
Balloo, Kieran. 2016. Individual differences in research methods learning among undergraduate psychology students. PhD Thesis Doctor of Philosophy. University of Roehampton.
Differences in education and employment inequalities based on the social and cultural environments of universities
Hosein, Anesa, Balloo, Kieran, Byrom, Nicola and Essau, Cecilia A.. 2022. "Differences in education and employment inequalities based on the social and cultural environments of universities." Society for Research into Higher Education International Research (SRHE) Conference 2022. 05 - 09 Dec 2022 United Kingdom.
University Environment Classification, 2008-2012
Balloo, K. and Hosein, A.. 2022. University Environment Classification, 2008-2012. Toowoomba. UK Data Service. https://doi.org/10.5255/UKDA-SN-856056
Impacts of adolescent mental health, social identity, and university attendance on mental health during emerging adulthood: Intersectional multilevel analyses of a national cohort study
Balloo, Kieren, Hosein, Anesa, Byrom, Nicola and Essau, Cecilia. 2021. "Impacts of adolescent mental health, social identity, and university attendance on mental health during emerging adulthood: Intersectional multilevel analyses of a national cohort study." Institute of Mental Health Sciences Conference: Mental Health across the Life-course. 02 - 02 Jun 2021 United Kingdom.
Preface
Murtonen, Mari and Balloo, Kieran. 2019. "Preface." Redefining Scientific Thinking for Higher Education: Higher-Order Thinking, Evidence-Based Reasoning and Research Skills. Switzerland. Springer. pp. i-xxi
Undergraduates’ personal circumstances, expectations and reasons for attending university
Balloo, Kieran, Pauli, Regina and Worrell, Marcia. 2017. "Undergraduates’ personal circumstances, expectations and reasons for attending university." Studies in Higher Education. 42 (8), pp. 1373-1384. https://doi.org/10.1080/03075079.2015.1099623
Students as partners in departmental research participation: the approaches to learning project
Marzouq, Shems, McSweeney, Matthew S., Sinton-Hewitt, Morgan and Balloo, Kieran. 2014. "Students as partners in departmental research participation: the approaches to learning project." HEA STEM Annual Learning and Teaching Conference: Enhancing the STEM Student Journey (HEA 2014). Edinburgh, United Kingdom 30 Apr - 01 May 2014