Evaluation report 2015 by Lou McGill Consultancy Ltd.A challenge for us as evaluators was that we were not there to identify a baseline at the beginning of the project so when we examined changing practices many of these had already happened or started to happen. So we took a retrospective view looking at how they changed, why they changed, and what didn’t change by utilising historical accounts, multiplicity of voices (many stakeholders), and by identifying barriers and tensions, evolving transformations, enablers, emerging benefits.
Overarching questions:
- Is this an appropriate model of open education (one of them) for Coventry University to adopt at a larger scale? (transferable/sustainable)
- What is the return on investment (ROI) (benefits)
- How far can the model work without charismatic leaders?
- How far the wider community engages with an open philosophy? – (cultural and traditional barriers)
- What is the impact on various stakeholders? (benefits and challenges)
- How does it change pedagogy and curriculum design?
Stakeholders
AT – academic team – Impact on academic practice
ST – support teams
RS – registered students – impact on learning, employability, professional networking
OS – open students
OP – open professionals
IM – institutional senior managers (dept/faculty level, strategic level, operational level)
WC – wider HE community (UK and global) Jisc, other HEIs – dissemination
Evaluation questions
These questions can also be framed in a different way which identifies what different stakeholders want to know and also links them to the main Activity Theory Areas. The table on the wiki Stakeholder page contains more specific questions that we have aimed to answer.
| Focus | Evaluation Questions | Identifying | Stakeholders to ask | Measure of success(evidence of) |
| Roles and skills | How do roles of academic and support staff change? | perceptions, practice change, cultural change, new skills, expectations | AT, ST | practice change,Increased confidence, new skills |
| What factors enable staff to change roles? | support needs, senior management support | AT, ST, IM | effective support mechanisms | |
| What are the barriers to this approach? | perceptions, fears, barriers | AT, ST | change in perceptions,lessening of fears,
practice change |
|
| How does the student role change? | perceptions, practice change, new skills, expectations | RS, OS, AT | increased input to coursedesign,
creation of course content, improved retention, improved progression, new professional skills and networks, peer support and feedback |
|
| What new roles in learning and teaching emerge for open professionals | new skills, expectations, perceptions, stakeholder engagement | OP, AT | input to coursedesign,
creation of course content, interaction with students, provide student feedback |
|
| What kind of support structures are needed for staff, registered and open students, open professionals? | stakeholder engagement, support needs | AT, ST, RS, OS, OP, IM | effective support mechanisms,digital literacy support | |
| Institutional context (Conventions, guidelines and rules regulating activities in the system) | What are the resource implications? | costs, staffing, time allocation, operational issues, technologies | AT, IM, ST | Balance of costs and benefits,increased enrolments of paid students,
input from open students and professionals (eg peer review) |
| How does this model relate to existing institutional strategies and policies? Are adaptations required or new ones needed? | strategy and policy | AT, ST, IM | adaptations to strategies/policies,new strategies/policies
institution-wide commitment |
|
| What are the issues around ownership? | legal issues, ownership and control, Copyright issues, licencing approaches | AT, ST, RS, OS, OP, IM | increased understanding of legal aspects,effective institutional mechanisms,
clear guidelines, changes to policy |
|
| Do existing course validation and quality mechanisms need to be changed? | quality issues, assessment needs, curriculum design issues | AT, IM | integration into existing mechanisms,institutional buy-in | |
| How do we brand and market the course? | branding issues, marketing methods, institutional reputation | AT, IM, RS, OS, OP | enhanced institutional reputation,enhanced individual academic reputation,
increased enrolments of paying students, increased involvement of open students and open professionals, enhanced student employability, improved alumni relationships |
|
| Are new operational or organisational structures needed to support this model | operational issues, open student issues | AT, ST, IM | wider institutional mechanisms support this approach,open content is effectively managed in both short and longer term, networks are sustained and maintained | |
| Tools: the artifacts (or concepts) used by actors in the system(model/approach) | Which technologies are most appropriate for this approach? | institutional technologies, social networking technologies, accessibility, discoverability, archiving and management | AT, ST, RS, OS, OP | effective technologies highlighted,understanding of appropriate technologies for different functions/activities, effective integration of institutional and open technologies |
| Is the model sustainable? | sustainability, support needs, operational issues, content management, community support needs | IM, AT, ST, OP | integration of approach into mainstream practice,changes to strategy/policy/operational mechanisms,
strategic manager buy-in, implementation with other subject disciplines |
|
| Is the model transferable? | subject discipline aspects, institutional technologies, social networking technologies, operational issues | AT, ST, IM | adoption of model or approach by other institutions or departments,additional/new partners | |
| How are the outputs managed/maintained? | content management, community support needs, technologies, operational issues, ownership and control, accessibility, discoverability, archiving and management | AT, ST, IM, OS, RS, OP | effective archiving, access and use within and outside the institution | |
| Community: social context; all actors involved in the activity system | How do we engage stakeholders in this open approach? | motivations, benefits, barriers, support needs | AT, IM, RS, OS, OP, OO | increased participation, enquiries from inside and outside the institution,more OPs and OSs,
increased enrolment |
| What is the impact on different stakeholders? | student experience, student retention, student enrollment, student progression, student attainment, open student experience, open professionals experience, staff experience, benefits, course registrations, digital literacies, professional visibility, online identities. digital storytelling, professional networking opportunities, professional development, curriculum design issues, independent learning, online learning communities | AT, IM, RS, OS, OP, OO | increased skills,increased awareness,
changed perceptions, changed practice, enhanced academic and institutional reputation, press and publicity, increased professional networking, professional opportunities for students, job offers for staff students, |
|
| What new relationships have emerged? How have existing relationships changed? | relationships between stakeholders (institution – staff, institution – students, institution – open professionals, institution – open students, institution – other organisations, staff – students, staff – open professionals, academic staff – support staff, etc, etc,), professional networking opportunities | AT, IM, RS, OS, OP, OO | new relationships,changed relationships,
increased professional networks, new partnerships, |
Move forward to Drivers