Step #4: Identity, describe, and rationalize your evaluation approach: Based on your learning in Modules 1 and 2, review your program theory and evaluation questions to identify an evaluation approach. Provide a description of how that approach will operate in your specific program context. Also, provide a rationale for why your approach is a good fit with your program context.
I have adjusted my program theory and questions based on feedback, to ensure the extent to which my questions can be answered, and because my focus seemed to be bridging either outcome or impact evaluation. Please note the changes indicated in the turquoise above; for comparison, you may contrast this text with the infographic at the top of this page (kept so you can see the trajectory of learning).
I adjusted the grade level of the students surveyed, for example, because I realized the likelihood of reaching all 600 students is very limited. However, I teach all English 10 classes but for two at our school. And English 10 is a required course for Grade 10 students. I could hypothetically (as I understand we are designing, not implementing) able to ask my students to voluntarily complete this survey. I would, again hypothetically, have to ensure two variables first: one, that these students were at our school two years ago, for the start of the program being evaluated; second, that they recall participating. My other option could be to survey the Grade 12 students, as I am responsible for an off-timetable course that is essentially a Graduation Transitions course. With two other teachers, I oversee all Grade 12 students. Again, I would hypothetically be able to communicate to all these students. The other variables would remain. To explain why I selected Grade 10, and not the 12’s, who would have more experience with PLT, I am keeping in mind the fact that this is an off-timetable course, and I could most effectively gather data from students when seen in-person.
Our case study with the logic model helped me to recognize some of the other problems in the nature of my questions. I was asking about levels of stress, to both students and staff. I stand by Herman et. al’s (1987) observation of how anecdotal evidence may be gathered as qualitative methods of research; however, I do see how questions 2, 3, and 4 (initially) were all anecdotal in this nature. I thought about the objectivity in measuring output, and the Yes/ No reflection approach in revising one’s questions – both of these points are from the Wilder Research (2009) article. With this, I determined that I did need to adjust my questions to allow for stronger output. For example, I attempted to reflect quantitative measures in the number of after-school hours teachers spent in tutorials, which would otherwise be conducted during PLT (Wilder Research, 2009).
With these questions, it does seem that I am looking at an impact evaluation. As Gertler et. al (2011a) note, I would be looking at a descriptive study with recognition of normative values and cause-and-effect questioning practices. They note (Gertler et. al, 2011a), as well, that impact evaluation may be prospective or retrospective, and this is the gray area for me: we did run PLT, but have not operated it in the last 1.5 years – it may be argued, then, with the intention of possibly running PLT again, that this is a prospective design; however, I would indicate that based on the questions designed, I am focusing on retrospective inquiry.
Thus, this program evaluation will reflect impact evaluation with a focus on “intention-to-treat” estimate (ITT) (Gertler et. al, 2011b). Gertler et. al (2011b) define this as a study including those who do and do not participate in the program; they contrast this with “treatment-on-the-treated” (TOT), an evaluation that would look just at the participants (p.39). This is tricky because all students are required to be in a physical classroom for PLT, but they cannot be required to necessarily engage or do anything, as this is the whole purpose of PLT is to allow students agency and choice in their learning. That is, as a teacher I do not punish those who choose to sit in a study block room and draw in their sketchbook. To be most accurate to how PLT has been implemented, and the underpinning philosophy of the program, I feel that the ITT method is most suitable.
Gertler, P. et. al. (2011a). Why Evaluate. In Impact evaluation in practice (pp.3-19). World Bank. https://openknowledge-worldbank-org.proxy.queensu.ca/bitstream/handle/10986/2550/599980PUB0ID181BLIC1009780821385418.pdf?sequence=1&isAllowed=y
Gertler, P. et. al. (2011b). Causal Inference and Counterfactuals. In Impact evaluation in practice (pp.33-47). World Bank. https://openknowledge-worldbank-org.proxy.queensu.ca/bitstream/handle/10986/2550/599980PUB0ID181BLIC1009780821385418.pdf?sequence=1&isAllowed=y
Wilder Research. (2009). Program theory and logic models, 1-19 http://www.evaluatod.org/assets/resources/evaluation- guides/logicmodel-8-09.pdf
Comments
Post a Comment