Evaluation activities

A challenge for us as evaluators was that we were not there to identify a baseline at the beginning of the project so when we examined changing practices many of these had already happened or started to happen. So we took a retrospective view looking at how they changed, why they changed, and what didn’t change by utilising historical accounts, multiplicity of voices (many stakeholders), and by identifying barriers and tensions, evolving transformations, enablers, emerging benefits.

Overarching questions:

  • Is this an appropriate model of open education (one of them) for Coventry University to adopt at a larger scale? (transferable/sustainable)
  • What is the return on investment (ROI) (benefits)
  • How far can the model work without charismatic leaders?
  • How far the wider community engages with an open philosophy? – (cultural and traditional barriers)
  • What is the impact on various stakeholders? (benefits and challenges)
  • How does it change pedagogy and curriculum design?


AT – academic team – Impact on academic practice

ST – support teams

RS – registered students – impact on learning, employability, professional networking

OS – open students

OP – open professionals

IM – institutional senior managers (dept/faculty level, strategic level, operational level)

WC – wider HE community (UK and global) Jisc, other HEIs – dissemination

Evaluation questions

These questions can also be framed in a different way which identifies what different stakeholders want to know and also links them to the main Activity Theory Areas. The table on the wiki Stakeholder page contains more specific questions that we have aimed to answer.

Focus Evaluation Questions Identifying Stakeholders to ask Measure of success(evidence of)
Roles and skills How do roles of academic and support staff change? perceptions, practice change, cultural change, new skills, expectations AT, ST practice change,Increased confidence, new skills
What factors enable staff to change roles? support needs, senior management support AT, ST, IM effective support mechanisms
What are the barriers to this approach? perceptions, fears, barriers AT, ST change in perceptions,lessening of fears,

practice change

How does the student role change? perceptions, practice change, new skills, expectations RS, OS, AT increased input to coursedesign,

creation of course content,

improved retention,

improved progression,

new professional skills and networks,

peer support and feedback

What new roles in learning and teaching emerge for open professionals new skills, expectations, perceptions, stakeholder engagement OP, AT input to coursedesign,

creation of course content,

interaction with students,

provide student feedback

What kind of support structures are needed for staff, registered and open students, open professionals? stakeholder engagement, support needs AT, ST, RS, OS, OP, IM effective support mechanisms,digital literacy support
Institutional context (Conventions, guidelines and rules regulating activities in the system) What are the resource implications? costs, staffing, time allocation, operational issues, technologies AT, IM, ST Balance of costs and benefits,increased enrolments of paid students,

input from open students and professionals (eg peer review)

How does this model relate to existing institutional strategies and policies? Are adaptations required or new ones needed? strategy and policy AT, ST, IM adaptations to strategies/policies,new strategies/policies

institution-wide commitment

What are the issues around ownership? legal issues, ownership and control, Copyright issues, licencing approaches AT, ST, RS, OS, OP, IM increased understanding of legal aspects,effective institutional mechanisms,

clear guidelines,

changes to policy

Do existing course validation and quality mechanisms need to be changed? quality issues, assessment needs, curriculum design issues AT, IM integration into existing mechanisms,institutional buy-in
How do we brand and market the course? branding issues, marketing methods, institutional reputation AT, IM, RS, OS, OP enhanced institutional reputation,enhanced individual academic reputation,

increased enrolments of paying students,

increased involvement of open students and open professionals,

enhanced student employability,

improved alumni relationships

Are new operational or organisational structures needed to support this model operational issues, open student issues AT, ST, IM wider institutional mechanisms support this approach,open content is effectively managed in both short and longer term, networks are sustained and maintained
Tools: the artifacts (or concepts) used by actors in the system(model/approach) Which technologies are most appropriate for this approach? institutional technologies, social networking technologies, accessibility, discoverability, archiving and management AT, ST, RS, OS, OP effective technologies highlighted,understanding of appropriate technologies for different functions/activities, effective integration of institutional and open technologies
Is the model sustainable? sustainability, support needs, operational issues, content management, community support needs IM, AT, ST, OP integration of approach into mainstream practice,changes to strategy/policy/operational mechanisms,

strategic manager buy-in,

implementation with other subject disciplines

Is the model transferable? subject discipline aspects, institutional technologies, social networking technologies, operational issues AT, ST, IM adoption of model or approach by other institutions or departments,additional/new  partners
How are the outputs managed/maintained? content management, community support needs, technologies, operational issues, ownership and control, accessibility, discoverability, archiving and management AT, ST, IM, OS, RS, OP effective archiving, access and use within and outside the institution
Community: social context; all actors involved in the activity system How do we engage stakeholders in this open approach? motivations, benefits, barriers, support needs AT, IM, RS, OS, OP, OO increased participation, enquiries from inside and outside the institution,more OPs and OSs,

increased enrolment

What is the impact on different stakeholders? student experience, student retention, student enrollment, student progression, student attainment, open student experience, open professionals experience, staff experience, benefits, course registrations, digital literacies, professional visibility, online identities. digital storytelling, professional networking opportunities, professional development, curriculum design issues, independent learning, online learning communities AT, IM, RS, OS, OP, OO increased skills,increased awareness,

changed perceptions,

changed practice,

enhanced academic and institutional reputation,

press and publicity,

increased professional networking,

professional opportunities for students,

job offers for staff students,

What new relationships have emerged? How have existing relationships changed? relationships between stakeholders  (institution – staff, institution – students, institution – open professionals, institution – open students, institution – other organisations, staff – students, staff – open professionals, academic staff – support staff, etc, etc,), professional networking opportunities AT, IM, RS, OS, OP, OO new relationships,changed relationships,

increased professional networks,

new partnerships,

Move forward to Drivers