PISA 2009 Technical Report -

Description

Foreword......Page 5 Programme for International Student Assessment: an Overview......Page 23 Participation......Page 25 Managing and implementing PISA......Page 26 Organisation of this report......Page 28 Test Design and Test Development......Page 29 Digital Reading Assessment (DRA)......Page 30 Paper-based assessment......Page 31 Development timeline......Page 33 The PISA 2009 reading literacy framework......Page 34 First phase of development......Page 35 National item submissions......Page 36 National review of items......Page 37 Preparation of dual (English and French) source versions......Page 38 Field trial selection......Page 39 Field trial design......Page 40 Field trial coder queries......Page 41 Main survey reading item selection......Page 42 Main survey mathematics items......Page 45 Despatch of main survey instruments......Page 46 Review of main survey item analyses......Page 47 The Development of the PISA Context Questionnaires......Page 49 The development of the PISA 2009 Questionnaire Framework......Page 50 Research areas in PISA 2009......Page 51 The field-trial of the PISA 2009 context questionnaires......Page 54 Student and School Questionnaires......Page 55 The implementation of the context questionnaires......Page 56 Sample Design......Page 59 Population coverage, and school and student participation rate standards......Page 60 Coverage of the PISA international target population......Page 61 School response rates......Page 62 Student response rates......Page 63 The sampling frame......Page 64 Stratification......Page 65 School sample selection......Page 68 Special school sampling situations......Page 70 Monitoring school sampling......Page 73 Student samples......Page 77 Definition of school......Page 78 Translation and Verification of the Test and Survey Material......Page 83 Development of source versions......Page 84 Double translation from two source languages......Page 85 Testing languages and translation/adaptation procedures......Page 86 International verification of the national versions......Page 88 Verification of test units......Page 89 Main survey verification......Page 90 Verification of questionnaires......Page 93 Final optical check of test booklets, questionnaire booklets and coding guides......Page 95 Verification of Digital Reading Assessment (DRA) units......Page 97 Summary of items deleted at the national level, due to translation, printing or layout errors......Page 98 Field Operations......Page 99 School Co-ordinators......Page 100 The selection of the school sample......Page 101 Preparation of test booklets, questionnaires and manuals......Page 102 Packaging and shipping materials......Page 103 Preparing for coding......Page 104 Logistics prior to coding......Page 106 Single coding design......Page 108 Multiple coding......Page 111 Managing the coding process......Page 113 Questionnaire coding......Page 114 The main survey review......Page 115 Quality Assurance......Page 117 Field trial and main survey review......Page 118 National Centre Quality Monitor (NCQM) visits......Page 119 Post final optical check......Page 120 Survey Weighting and the Calculation of Sampling Variance......Page 121 Survey weighting......Page 122 The school base weight trimming factor......Page 123 The within-school base weight......Page 124 The within school non-response adjustment......Page 127 The balanced repeated replication variance estimator......Page 128 Countries and economies where all students were selected for PISA......Page 130 Scaling PISA Cognitive Data......Page 131 The mixed coefficients multinomial logit model......Page 132 Combined model......Page 133 National calibrations......Page 134 National reports......Page 135 International calibration......Page 141 Student score generation......Page 142 Booklet effects......Page 143 Analysis of data with plausible values......Page 144 Developing common scales for the purposes of trends......Page 145 Uncertainty in the link......Page 146 Data Management Procedures......Page 149 Introduction......Page 150 Data entry quality control......Page 152 DRA data......Page 154 Review of the test and questionnaire data......Page 155 Next steps in preparing the international database......Page 156 Sampling Outcomes......Page 157 Design effects and effective sample sizes......Page 170 Design effects in PISA for performance variables......Page 173 Summary analyses of the design effect......Page 185 Sampling for the Digital Reading Assessment (DRA) component......Page 187 Scaling Outcomes......Page 189 International characteristics of the item pool......Page 190 Test targeting......Page 191 Domain inter-correlations......Page 196 National item deletions......Page 197 International scaling......Page 199 Generating student scale scores and reliability of the PISA scales......Page 200 Test length analysis......Page 201 Booklet effects......Page 205 Overview of the PISA cognitive reporting scales......Page 213 PISA literacy subscales......Page 214 Observations concerning the construction of the PISA overall literacy scales......Page 215 Framework development......Page 216 Testing time and item characteristics......Page 217 Reading......Page 231 Link error......Page 232 Coding Reliability Studies......Page 235 Consistency analyses......Page 236 International coder review......Page 241 Data Adjudication......Page 249 Implementing the standards – quality assurance......Page 250 Information available for adjudication......Page 251 Data adjudication process......Page 252 Digital Reading Assessment (DRA)......Page 253 Detailed country comments......Page 254 Proficiency Scale Construction......Page 259 Introduction ......Page 260 Stage 1: Identifying possible scales......Page 261 Stage 6: Revising and refining with main survey data......Page 262 Defining proficiency levels......Page 263 Building an item map for print reading......Page 265 Levels of print reading literacy......Page 267 Building an item map for digital reading......Page 274 Levels of digital reading literacy......Page 277 Interpreting the reading literacy levels......Page 278 Scaling Procedures and Construct Validation of Context Questionnaire Data......Page 281 Simple questionnaire indices......Page 282 School questionnaire indices......Page 284 Scaling procedures......Page 286 Describing questionnaire scale indices......Page 288 Student scale indices......Page 289 School questionnaire scale indices......Page 308 Parent questionnaire scale indices......Page 312 The index of economic, social and cultural status......Page 314 Digital Reading Assessment......Page 319 Translation......Page 320 School computer resources survey......Page 321 Test delivery system......Page 322 Online Coding System......Page 323 International Database......Page 327 Student files......Page 328 School file......Page 329 Records included in the database......Page 330 How are students and schools identified?......Page 331 Further information......Page 332 References......Page 335 Annexes......Page 337 Annex A ­– Main study item pool classification......Page 338 Annex B – Contrast coding used in conditioning......Page 346 Annex C – DESIgn EFFECT TABLES......Page 355 Annex D - Changes to core questionnaire items......Page 361 Annex E – Mapping of ISCED to years......Page 366 Annex F – National household possession items......Page 367 Annex G – PISA 2009 TECHNICAL STANDARDS......Page 369 Annex H – pisa Consortium, staff and consultants......Page 383 Annex I – SELECTION OF OECD PISA PUBLICATIONS......Page 386 Annex J – OECD countries included in standardisation of major PISA scales......Page 387