Hide keyboard shortcuts

Hot-keys on this page

r m x p   toggle line displays

j k   next/prev highlighted chunk

0   (zero) top of page

1   (one) first highlighted chunk

1"""Test scenario for reviews. 

2 

3## Domain 

4 

5* Users as in `conftest`, under *players* 

6* Clean slate, see `starters`. 

7* The user table 

8* The country table 

9* One contribution record 

10* One assessment record 

11* The assessment submitted and reviewers assigned. 

12* Two reviews, with review comments filled out. 

13 

14## Acts 

15 

16Filling out reviews. 

17 

18`test_reviewEntryViewAll` 

19: All users try to see the review entries, but only 

20 **expert** and **final** succeed for their own comments, and the power users 

21 succeed for all comments. 

22 Only when a review decision has been taken, the review comments become readable 

23 to the owner/editor of the contribution, the other reviewer, and the national 

24 coordinator. 

25 

26`test_decideOthers` 

27: All users except the reviewers try to make all review decisions. 

28 None of them succeeds. 

29 

30`test_reviewersDecide` 

31: **expert** tries to make all possible decisions and then revokes 

32 his last decision. 

33 After that, **final** tries the same, but fails, because 

34 **final** can only decide after **expert**. 

35 

36`test_expertDecides` 

37: **expert** tries to revoke the decision, but fails because the decision 

38 is already revoked. 

39 Then he Accepts. 

40 Then he Accepts again, but fails, because it is the same as the existing 

41 decision. 

42 

43`test_finalAccepts` 

44: **final** accepts. 

45 After that **expert** tries to take all review decisions, but fails, 

46 because the final decision has been taken. 

47 

48`test_modify` 

49: All users try to modify a field in the contribution, in the assessment, and in 

50 a review comment, but all fail, because everything is in a finished state. 

51 In particular, also a re-assignment of the reviewers to the assessment it attempted. 

52 

53`test_revokeDelay` 

54 There is a delay time during which the final decision can be revoked. 

55 We'll test what happens when the delay time is past. 

56 We do this by updating the `dateDecided` field under water, directly 

57 in Mongo. 

58 We shift the time back 23 hours 

59 **final** revokes and succeeds and then accepts again. 

60 We shift the time back 25 hours. 

61 **final** revokes and fails. 

62 We shift the time forward 25 hours again. 

63 

64`test_finalRevokes` 

65: **final** revokes his decision. 

66 After that **expert** tries to take all review decisions, and succeeds, 

67 because there is no final decision. 

68 

69`test_modify2` 

70: All users try to modify a field in the contribution, in the assessment, and in 

71 a review comment, and some succeed, because there is no finished state. 

72 

73 The assesssment is still submitted, so the contribution and assessment are still 

74 locked: nobody can modify them. 

75 

76 The reviewers can be reassigned by **office**. 

77 

78 The reviews can be modified. 

79 

80`test_finalRevise` 

81: We test revision and resubmission. The chain of events is: 

82 

83 * **expert** has to give an advisory decision first. We let hem accept 

84 the assessment; 

85 * we shift the submission date back 27 hours, 

86 so that the system does not consider the assessment resubmitted 

87 when we have shifted the review decisions back in time; 

88 * we shift the decisionDate back 26 hours, 

89 so that **expert** cannot revoke his decision right now; 

90 * **final** decides that revision is needed; 

91 * we shift the decisionData back 25 hours, 

92 so that **final** cannot revoke his decision under the current 

93 conditions; 

94 * **expert** tries to take all review decisions, but fails, 

95 because the assessment has not yet been resubmitted. 

96 

97`test_revise` 

98: Here we test the **owner**'s part in the revision: 

99 

100 * he checks the current stage of the assessment: "complete revised" 

101 * he sets the evidence of the first entry to an empty value 

102 * he checks the current stage of the assessment: "incomplete revised" 

103 * he sets the evidence of the first entry to a new empty value 

104 * he checks the current stage of the assessment: "complete revised" 

105 * he submits the assessment but fails 

106 * he resubmits the assessment but fails 

107 * he submits the assessment as revision and succeeds 

108 

109`test_newRound` 

110: Here we test the making of new review decisions. 

111 

112 * **final** tries to decide anything, but fails. 

113 * **expert** acceepts and succeeds. 

114 * **final** acceepts and succeeds. 

115""" 

116 

117import pytest 

118 

119import magic # noqa 

120from control.utils import pick as G, E 

121from conftest import USERS, POWER_USERS 

122from example import ( 

123 ACCEPT, 

124 ASSESS, 

125 COMMENTS, 

126 COMMENTS_E, 

127 COMMENTS_F, 

128 COMPLETE_REVISED, 

129 CONTRIB, 

130 CRITERIA_ENTRY, 

131 DATE_DECIDED, 

132 DATE_SUBMITTED, 

133 EVIDENCE, 

134 EVIDENCE1, 

135 EXPERT, 

136 FINAL, 

137 INCOMPLETE_REVISED, 

138 OFFICE, 

139 REJECT, 

140 REMARKS, 

141 REMARKS_E, 

142 REMARKS_F, 

143 RESUBMIT_ASSESSMENT, 

144 REVIEW, 

145 REVIEW_DECISION, 

146 REVIEW_ENTRY, 

147 REVIEWER_E, 

148 REVISE, 

149 REVOKE, 

150 SUBMIT_ASSESSMENT, 

151 SUBMIT_REVISED, 

152 TITLE, 

153 TITLE2, 

154 TITLE_A2, 

155 USER, 

156) 

157from helpers import ( 

158 findReviewEntries, 

159 forall, 

160 getItem, 

161 getReviewEntryId, 

162) 

163from starters import start 

164from subtest import ( 

165 assertFieldValue, 

166 assertModifyField, 

167 assertReviewDecisions, 

168 assertShiftDate, 

169 assertStage, 

170 assertStatus, 

171) 

172 

173startInfo = {} 

174 

175 

176@pytest.mark.usefixtures("db") 

177def test_start(clientOffice, clientOwner, clientExpert, clientFinal): 

178 startInfo.update( 

179 start( 

180 clientOffice=clientOffice, 

181 clientOwner=clientOwner, 

182 clientExpert=clientExpert, 

183 clientFinal=clientFinal, 

184 users=True, 

185 assessment=True, 

186 countries=True, 

187 review=True, 

188 ) 

189 ) 

190 

191 

192def test_reviewEntryView(clients): 

193 recordId = startInfo["recordId"] 

194 

195 cIds = recordId[CRITERIA_ENTRY] 

196 cIdFirst = cIds[0] 

197 reId = {} 

198 for user in {EXPERT, FINAL}: 

199 criteriaEntryInfo = getItem(clients[user], CRITERIA_ENTRY, cIdFirst) 

200 text = criteriaEntryInfo["text"] 

201 reviewEntries = findReviewEntries(text) 

202 reId[user] = reviewEntries[user][0] 

203 

204 def assertIt(cl, exp): 

205 user = cl.user 

206 for (kind, rId) in reId.items(): 

207 value = f"{kind}'s comment on criteria 1" 

208 expValue = ( 

209 None 

210 if exp is None 

211 else value 

212 if user in {EXPERT, FINAL} | POWER_USERS 

213 else None 

214 ) 

215 assertFieldValue((cl, REVIEW_ENTRY, rId), COMMENTS, expValue) 

216 

217 expect = {user: None for user in USERS} 

218 expect.update({user: True for user in {EXPERT, FINAL} | POWER_USERS}) 

219 forall(clients, expect, assertIt) 

220 

221 

222def test_decideOthers(clients): 

223 recordId = startInfo["recordId"] 

224 

225 reviewId = G(recordId, REVIEW) 

226 

227 def assertIt(cl, exp): 

228 user = cl.user 

229 for kind in [EXPERT, FINAL]: 

230 rId = G(reviewId, kind) 

231 expStatus = kind == user if exp else exp 

232 for decision in [REJECT, REVISE, ACCEPT, REVOKE]: 

233 decisionStr = G(G(REVIEW_DECISION, decision), kind) 

234 url = f"/api/task/{decisionStr}/{rId}" 

235 assertStatus(cl, url, expStatus) 

236 

237 expect = {user: False for user in USERS if user not in {EXPERT, FINAL}} 

238 forall(clients, expect, assertIt) 

239 

240 

241def test_reviewersDecide(clientsReviewer): 

242 recordId = startInfo["recordId"] 

243 reviewId = G(recordId, REVIEW) 

244 

245 assertReviewDecisions( 

246 clientsReviewer, 

247 reviewId, 

248 [EXPERT, FINAL], 

249 [REJECT, REVISE, ACCEPT, REVOKE], 

250 {EXPERT: True, FINAL: False}, 

251 ) 

252 

253 

254def test_expertDecides(clientsReviewer): 

255 recordId = startInfo["recordId"] 

256 reviewId = G(recordId, REVIEW) 

257 

258 assertReviewDecisions( 

259 clientsReviewer, reviewId, [EXPERT], [REVOKE], False, 

260 ) 

261 assertReviewDecisions( 

262 clientsReviewer, reviewId, [EXPERT], [ACCEPT], True, 

263 ) 

264 assertReviewDecisions( 

265 clientsReviewer, reviewId, [EXPERT], [ACCEPT], False, 

266 ) 

267 

268 

269def test_finalAccepts(clientsReviewer): 

270 recordId = startInfo["recordId"] 

271 reviewId = G(recordId, REVIEW) 

272 

273 assertReviewDecisions( 

274 clientsReviewer, reviewId, [FINAL], [ACCEPT], True, 

275 ) 

276 assertReviewDecisions( 

277 clientsReviewer, reviewId, [EXPERT], [REJECT, REVISE, ACCEPT, REVOKE], False, 

278 ) 

279 

280 

281def test_modify(clients): 

282 valueTables = startInfo["valueTables"] 

283 recordId = startInfo["recordId"] 

284 

285 users = valueTables[USER] 

286 eid = recordId[CONTRIB] 

287 aId = recordId[ASSESS] 

288 reviewId = recordId[REVIEW] 

289 rExpertId = reviewId[EXPERT] 

290 rFinalId = reviewId[FINAL] 

291 cIdFirst = recordId[CRITERIA_ENTRY][0] 

292 renId = getReviewEntryId(clients, cIdFirst, rExpertId, rFinalId) 

293 

294 def assertIt(cl, exp): 

295 assertModifyField(cl, CONTRIB, eid, TITLE, TITLE2, exp) 

296 assertModifyField(cl, ASSESS, aId, TITLE, TITLE_A2, exp) 

297 reviewerFId = G(users, FINAL) 

298 assertModifyField(cl, ASSESS, aId, REVIEWER_E, (reviewerFId, FINAL), exp) 

299 assertModifyField( 

300 cl, CRITERIA_ENTRY, cIdFirst, EVIDENCE, ([EVIDENCE1], EVIDENCE1), exp 

301 ) 

302 for (rId, remarks) in ((rExpertId, REMARKS_E), (rFinalId, REMARKS_F)): 

303 assertModifyField(cl, REVIEW, rId, REMARKS, ([remarks], remarks), exp) 

304 for (u, reid) in renId.items(): 

305 comments = COMMENTS_E if u == EXPERT else COMMENTS_F 

306 assertModifyField( 

307 cl, REVIEW_ENTRY, reid, COMMENTS, ([comments], comments), exp 

308 ) 

309 

310 expect = {user: False for user in USERS} 

311 forall(clients, expect, assertIt) 

312 

313 

314def test_revokeDelay(clientFinal, clientSystem): 

315 recordId = startInfo["recordId"] 

316 reviewId = G(recordId, REVIEW) 

317 

318 cFinal = {FINAL: clientFinal} 

319 rFinal = G(reviewId, FINAL) 

320 assertShiftDate(clientSystem, REVIEW, rFinal, DATE_DECIDED, -23) 

321 assertReviewDecisions( 

322 cFinal, reviewId, [FINAL], [REVOKE, ACCEPT], True, 

323 ) 

324 assertShiftDate(clientSystem, REVIEW, rFinal, DATE_DECIDED, -25) 

325 assertReviewDecisions( 

326 cFinal, reviewId, [FINAL], [REVOKE], False, 

327 ) 

328 assertShiftDate(clientSystem, REVIEW, rFinal, DATE_DECIDED, 25) 

329 

330 

331def test_finalRevokes(clientsReviewer): 

332 recordId = startInfo["recordId"] 

333 reviewId = G(recordId, REVIEW) 

334 

335 assertReviewDecisions( 

336 clientsReviewer, reviewId, [FINAL], [REVOKE], True, 

337 ) 

338 assertReviewDecisions( 

339 clientsReviewer, reviewId, [EXPERT], [REJECT, REVISE, ACCEPT, REVOKE], True, 

340 ) 

341 

342 

343def test_modify2(clients): 

344 valueTables = startInfo["valueTables"] 

345 recordId = startInfo["recordId"] 

346 

347 users = valueTables[USER] 

348 eid = recordId[CONTRIB] 

349 aId = recordId[ASSESS] 

350 reviewId = recordId[REVIEW] 

351 rExpertId = reviewId[EXPERT] 

352 rFinalId = reviewId[FINAL] 

353 cIdFirst = recordId[CRITERIA_ENTRY][0] 

354 renId = getReviewEntryId(clients, cIdFirst, rExpertId, rFinalId) 

355 

356 def assertIt(cl, exp): 

357 assertModifyField(cl, CONTRIB, eid, TITLE, TITLE2, exp[CONTRIB]) 

358 assertModifyField(cl, ASSESS, aId, TITLE, TITLE_A2, exp[ASSESS]) 

359 reviewerFId = G(users, FINAL) 

360 reviewerEId = G(users, EXPERT) 

361 assertModifyField( 

362 cl, ASSESS, aId, REVIEWER_E, (reviewerFId, FINAL), exp["assign"] 

363 ) 

364 assertModifyField( 

365 cl, ASSESS, aId, REVIEWER_E, (reviewerEId, EXPERT), exp["assign"] 

366 ) 

367 assertModifyField( 

368 cl, 

369 CRITERIA_ENTRY, 

370 cIdFirst, 

371 EVIDENCE, 

372 ([EVIDENCE1], EVIDENCE1), 

373 exp[CRITERIA_ENTRY], 

374 ) 

375 for (rId, remarks, kind) in ( 

376 (rExpertId, REMARKS_E, EXPERT), 

377 (rFinalId, REMARKS_F, FINAL), 

378 ): 

379 assertModifyField( 

380 cl, REVIEW, rId, REMARKS, ([remarks], remarks), exp[f"{REVIEW}_{kind}"] 

381 ) 

382 for (kind, reid) in renId.items(): 

383 comments = COMMENTS_E if kind == EXPERT else COMMENTS_F 

384 assertModifyField( 

385 cl, 

386 REVIEW_ENTRY, 

387 reid, 

388 COMMENTS, 

389 ([comments], comments), 

390 exp[f"{REVIEW_ENTRY}_{kind}"], 

391 ) 

392 

393 expectDefault = {user: False for user in USERS} 

394 expectPerKind = { 

395 CONTRIB: dict(expectDefault), 

396 ASSESS: dict(expectDefault), 

397 CRITERIA_ENTRY: dict(expectDefault), 

398 "assign": dict(expectDefault), 

399 f"{REVIEW}_{EXPERT}": dict(expectDefault), 

400 f"{REVIEW}_{FINAL}": dict(expectDefault), 

401 f"{REVIEW_ENTRY}_{EXPERT}": dict(expectDefault), 

402 f"{REVIEW_ENTRY}_{FINAL}": dict(expectDefault), 

403 } 

404 expectPerKind["assign"].update({user: True for user in {OFFICE}}) 

405 expectPerKind[f"{REVIEW}_{EXPERT}"].update( 

406 {user: True for user in {EXPERT} | POWER_USERS} 

407 ) 

408 expectPerKind[f"{REVIEW}_{FINAL}"].update( 

409 {user: True for user in {FINAL} | POWER_USERS} 

410 ) 

411 expectPerKind[f"{REVIEW_ENTRY}_{EXPERT}"].update( 

412 {user: True for user in {EXPERT} | POWER_USERS} 

413 ) 

414 expectPerKind[f"{REVIEW_ENTRY}_{FINAL}"].update( 

415 {user: True for user in {FINAL} | POWER_USERS} 

416 ) 

417 expect = {} 

418 for (kind, expectKind) in expectPerKind.items(): 

419 for (user, exp) in expectKind.items(): 

420 expect.setdefault(user, {})[kind] = exp 

421 

422 forall(clients, expect, assertIt) 

423 

424 

425def test_finalRevise(clientsReviewer, clientSystem): 

426 recordId = startInfo["recordId"] 

427 aId = G(recordId, ASSESS) 

428 reviewId = G(recordId, REVIEW) 

429 rExpert = G(reviewId, EXPERT) 

430 rFinal = G(reviewId, FINAL) 

431 

432 assertReviewDecisions(clientsReviewer, reviewId, [EXPERT], [ACCEPT], True) 

433 assertShiftDate(clientSystem, ASSESS, aId, DATE_SUBMITTED, -27) 

434 assertShiftDate(clientSystem, REVIEW, rExpert, DATE_DECIDED, -26) 

435 assertReviewDecisions(clientsReviewer, reviewId, [FINAL], [REVISE], True) 

436 assertShiftDate(clientSystem, REVIEW, rFinal, DATE_DECIDED, -25) 

437 assertReviewDecisions( 

438 clientsReviewer, reviewId, [EXPERT], [REJECT, REVISE, ACCEPT, REVOKE], False, 

439 ) 

440 

441 

442def test_Revise(clientOwner): 

443 recordId = startInfo["recordId"] 

444 aId = G(recordId, ASSESS) 

445 cIds = recordId[CRITERIA_ENTRY] 

446 cIdFirst = cIds[0] 

447 

448 assertStage(clientOwner, ASSESS, aId, COMPLETE_REVISED) 

449 assertModifyField( 

450 clientOwner, CRITERIA_ENTRY, cIdFirst, EVIDENCE, ([], E), True, 

451 ) 

452 assertStage(clientOwner, ASSESS, aId, INCOMPLETE_REVISED) 

453 theEvidence = [f"revised evidence for 1", "see the internet"] 

454 theEvidenceRep = ",".join(theEvidence) 

455 assertModifyField( 

456 clientOwner, 

457 CRITERIA_ENTRY, 

458 cIdFirst, 

459 EVIDENCE, 

460 (theEvidence, theEvidenceRep), 

461 True, 

462 ) 

463 assertStage(clientOwner, ASSESS, aId, COMPLETE_REVISED) 

464 assertStatus(clientOwner, f"/api/task/{SUBMIT_ASSESSMENT}/{aId}", False) 

465 assertStatus(clientOwner, f"/api/task/{RESUBMIT_ASSESSMENT}/{aId}", False) 

466 assertStatus(clientOwner, f"/api/task/{SUBMIT_REVISED}/{aId}", True) 

467 

468 

469def test_newRound(clientsReviewer): 

470 recordId = startInfo["recordId"] 

471 reviewId = G(recordId, REVIEW) 

472 

473 assertReviewDecisions( 

474 clientsReviewer, reviewId, [FINAL], [REJECT, ACCEPT, REVISE, REVOKE], False 

475 ) 

476 assertReviewDecisions(clientsReviewer, reviewId, [EXPERT], [ACCEPT], True) 

477 assertReviewDecisions(clientsReviewer, reviewId, [FINAL], [ACCEPT], True)