Evaluation of the Redesign of the Learning Experience (RoLEx)

Appendix C

Birmingham City University

RoLEx Project Board – 6 July 2009

Evaluation of the Redesign of the Learning Experience - (RoLEx)

1. Aim

The purpose of this paper is to provide members with a report of the evaluation of the RoLEx process as conducted by the RoLEx Evaluation Group (REG)

2. Executive Summary

This evaluation report has as its scope the operation of the RoLEx process in its first iteration, which has taken place during academic year 2008-2009.

A RoLEx Evaluation Group (REG) was convened early in 2009 following extensive discussions between the Lead External Adviser, the RoLEx Project Manager and the Lead Academic Consultant. The Group comprised three External advisers chosen for their expertise in Quality Enhancement together with three internal staff (see Annex 1 for REG membership).

A paper concerning RoLEx evaluation and the work of REG was presented to and agreed at Senate in February 2009.

External members of REG were briefed well in advance of the main evaluation week, which took place from June 15-19 2009 at Birmingham City University. REG members were provided with access to all documentation relating to the Streamlined Re-approval Process (STRAP) in advance of this week.

This report was produced at the end of the evaluation week at the university in June 2009. It has been shaped by the three external advisers working closely with the internal members. Their sources of information comprised:

• STRAP documentation, notably Explanatory Statements for programme teams and notes from the deliberations of STRAP panels;

• Seven meetings with staff and student groups during the June 2009 evaluation week, each led by one of the external advisers and internally clerked;

• Attendance at the final meetings of STRAP panels in June

• Discussions building on the reflections of the RoLEx Project Manager and the Lead Academic Consultant to the project.

The report provides:

• Detailed findings grouped under ten themes

• A number of Recommendations (19) for the University to consider when implementing subsequent iterations of RoLEx.

The Evaluation Group concludes that Birmingham City University, through the commitment of its staff, has delivered significant and positive change by way of response to the challenge of the RoLEx initiative. The Group notes that though some anticipated progress in bringing forward change remains aspirational, this is to be expected for a project of this size.

The Group concludes that further enhancement of the students' learning experience is possible and desirable. It is the Group's view that progress on such an agenda will only be achievable though the creation of a culture in which student learning is at the centre of academic activities; but the Group has found cause, during their review, to be optimistic about the capability and willingness of Birmingham City University, through its staff, to rise to the challenge and embrace the cultural change advocated in this report.

3. Introduction

The purpose of this RoLEx Evaluation Report is to contribute to Birmingham City University’s understanding of what has worked well about RoLEx so far, and where changes and improvements would help the processes of enhancing the students’ learning experiences in the future. Annex 1 A contains details of the Evaluation Group membership.

The findings emerged from:

• Six group interviews with staff and students, held between Tuesday 16 June 2009 and Thursday 18 June 2009 (a list of the constituency of these groups is attached as Annex 2)

• Observations at each of the final University Streamlined Re-approval Process Panels, Tuesday 16 June and Thursday 18 June 2009.

• A meeting with the Vice-Chancellor, Wednesday 17 June 2009.

• Prior study of the course programme teams’ RoLEx documentation delivered to the Faculty Approval Panels.

The intention of this report is to assist the University in making the most of and learning from the changes already implemented. It also considers how to facilitate desirable changes for students and staff and address weaknesses in the future. Explicit evaluation of the actual programme revisions is outside our remit.

4. The aims of RoLEx as understood by the external evaluators

Before the evaluation began, the external evaluators were extensively briefed on the rationale and descriptions of the RoLEx process.

RoLEx is the title of the arrangements for enhancement of Birmingham City University’s academic programmes, with particular attention given to the students’ learning experiences, commencing with undergraduate programmes for 2009/2010. The external evaluators were told that these developments would be a means to implement the University Learning and Teaching Strategy (2007), which was developed through extensive discussion across the university, and identifies seven goals:

1. Facilitate a smooth transition to and through Birmingham City University for all students. 2. Gain a high level of student engagement with the learning process 3. Offer flexible approaches to learning 4. Meet the needs of a diverse student group 5. Develop highly employable students who are aware of their responsibilities to their profession and to society 6. Develop and support staff to achieve high academic and professional standards in learning and teaching and recognise excellence 7. Provide the most appropriate and effective learning environment

In November 2007 Senate decided that the core unit size in undergraduate programmes would be increased from 12 credits to 15 credits, to bring the institution more in line with sector norms. The decision to make that structural change was perceived as providing an opportunity for enhancement of students’ learning experiences throughout the university.

5. Implementation of RoLEx

During our evaluation work the Group noted the following positive aspects concerning implementation of RoLEx:

• The involvement of the Vice-Chancellor, who chairs the University’s RoLEx Project Board;

• The secondment of staff to support the process and its evaluation;

• The funding from the Centre for Enhancement of Learning and Teaching to enable special initiatives in faculties;

• The entrusting to faculties of responsibilities for overseeing the process and assuring the quality and standards of the outcomes;

• The establishment of Faculty RoLEx Project Boards to be responsible for implementation of the initiative in their faculties;

• The provision of support for staff both from the Centre for Enhancement of Learning and Teaching (CELT) and within faculties;

• The creation of processes designed to confirm the rigour of the (streamlined) process of re-approval (STRAP), based upon samples within faculties;

• The appointment and funding of external consultants to take part in an independent evaluation of the first year of operation.

6. Themes identified in our evaluation of RoLEx

These ten themes were frequently mentioned in staff meetings and the documentation available to us:

a. Commitment to enhancement b. Leadership, drivers for change and ability to influence c. Communication d. Pressure of time, timing and institutional circumstances e. The relationship between quality enhancement and quality assurance f. The relationship between reality and the documentation g. Variability and flexibility h. Student engagement i. Employer engagement j. Staff development and support

7. Findings grouped by theme

a. Commitment to enhancement

• Generally speaking, staff reported themselves to be enthusiastic and positive about the intentions of RoLEx. They welcome the opportunity to enhance students’ learning experiences in accordance with the Learning and Teaching Strategy.

• They believe that their redesigned programmes will have a positive impact on the staff and the learning experiences of students.

• Many staff have worked exceedingly hard, occasionally in difficult circumstances, to improve their programmes and complete the required paperwork within the limited timeframe.

b. Leadership, drivers for change and ability to influence

• We noted the evident commitment of senior University staff to RoLEx.

• There was much leadership in faculties from the RoLEx Project Boards.

• There was often a strong and useful influence from Academic Registry on what had to be done for STRAP and how it should be done.

• Programme Leaders and Module leaders took responsibility for those parts of the process that were assigned to them.

• Largely we met staff who believed in the ethos of RoLEx. However, some groups of staff linked RoLEx to the stated University strategy of increasing financial turnover and assumed that there was an implicit agenda of cost savings, despite there having been many reassurances to the contrary when RoLEx was launched.

• We met some students and staff who were committed to the initiative but did not feel they had the agency to take appropriate decisions to participate fully in the realisation of RoLEx. This had arisen through being informed about RoLEx in contexts which did not encourage participation, or from an expectation that the culture was one of a high level of prescription.

c. Communication

There was evidence of effective communication found in such forms as:

• All sources showed considerable evidence of effective university wide communication with staff in that there was widespread understanding that the primary intention of, and potential within, RoLEx was to enhance students’ learning experiences.

• Information concerning the range of support available was provided for staff at the start of the initiative and was perceived to be clear, helpful and sufficient. CELT ran effective workshops which helped staff to see what was required and how they could implement it. Some interviewees stated that the RoLEx Moodle site provided was seen to be very helpful.

• Faculty Project Boards were perceived as effective sources of support in cases where staff felt they could ask their questions, and receive useful answers.

• Examples were provided of senior staff across faculties communicating effectively with each other and sharing solutions, often informally as well as formally.

Where there was a lack of effective communication, it was manifested in the following ways:

• Overall, despite good communication identified in some faculties, there was a tendency towards one way downwards communication.

• There was evidence in some cases that feedback loops did not have timely closure to respond to staff members’ misunderstandings and genuine problems.

• We heard from Faculty Project Board chairs who had wanted a higher level decision, but had difficulty finding out who could provide it.

• As RoLEx progressed into the day-to-day implementation phase, there was evidence that communication practices were less effective. Many staff reported feeling increasing anxiety due to absence of clarity in information, inadequate or contradictory answers to queries, and late injunctions to undertake essential activities (for example, the delayed provision of templates).

d. Pressure of time, timing and institutional circumstances

• We encountered no evidence of programmes which had not been completed in time. Staff commitment to the possibilities of RoLEx appeared to be such that they had overcome time pressures and inconvenient timing to undertake the consultations, engage in planning and complete the documentation.

• We noted contexts where staff had encountered substantial institutional change but had been very active in engaging with the possibilities of RoLEx.

• Staff spoke extensively about the difficulty of engaging in RoLEx in the time available to them. Many said they could see the potential of RoLEx but hadn’t been able to realise it fully because of the shortness of time or pressure of other commitments.

• Some staff reported a clash between the demands of RoLEx and other priorities: e.g. student placement timing, professional body standards changing the following year, and complete programme reapproval having taken place the previous year.

• There was a feeling that the choice of year for RoLEx was particularly inappropriate due to the faculty restructuring, revision of SUAR, and the number of vacant key posts.

e. The relationship between quality enhancement and quality assurance

• Many of the staff with whom we met spoke of quality assurance and activity involved with enhancement as different processes, each with their own requirements and expectations.

• We found little to persuade us that the complementary nature of these activities was understood and appreciated by staff.

f. The relationship between reality and the documentation

• There were instances in the programme documentation which showed that teams had integrated their vision for student learning fully into the descriptions contained in the templates and the explanatory statement.

• We also found some discrepancies where the vision had not successfully translated into formal documentation. Some staff described finding the first stage of RoLEx as exciting and full of opportunities, while in the later stage their sense of opportunity narrowed down to it being just a paper exercise.

• It appeared that excessive paperwork was generated in some cases, for no apparent need, which staff described as an over-compensatory response to a perceived lack of confidence in meeting the demands of the University’s QA processes.

• Some staff reported finding some of the processes and language unfamiliar and cumbersome (for example, the language and process of ‘mapping’ and constructing matrices). They perceived these as reiterating action already undertaken and already evident elsewhere within the documentation.

g. Variability and flexibility

• Teams appreciated the opportunity to be creative in how they addressed RoLEx.

• Some programme teams produced documentation in refreshingly helpful formats which conveyed clearly the information required in ways appropriate to the programme and showed full engagement with the vision and principles of RoLEx.

• However, the documentation provided to STRAP panels was variable in coverage. In some cases it did not respond to all the headings in the template provided and thus came across as incomplete.

• Some staff experienced the latitude offered by RoLEx (at least initially) as an absence of direction and prescription, which left them feeling unsure and/or unable to act in the resultant vacuum.

• In some programmes there had been considerable flexibility in the students’ experiences of learning available prior to the RoLEx initiative and this was built upon. In other cases, new practice in relation to flexibility was introduced.

• Students were not necessarily sufficiently prepared in order to take full advantage of the flexibility available to them through RoLEx.

h. Student engagement

• Many students who participated in RoLEx supported it, believing that it could enhance their learning experiences.

• The students in the interview group reported that a large number of students did not know about RoLEx.

• Variability in the students’ experience of consultation and involvement was noted. While the documentation presented to us indicated extensive and effective student engagement, the evidence of this was less strong from the student interview group in terms of both quality and quantity of engagement.

• There has often been a focus on informing students (for example, documents sent by e-mail) rather than involving them (for example, through workshops with their course coordinators and staff team). Some students seemed to have been satisfied with very limited involvement which may have accorded with the way in which RoLEx had been framed for them.

• There was some evidence of student anxiety about RoLEx; for example it was reported to us that some students had cancelled their sandwich placements in order not to be affected by RoLEx.

i. Employer engagement

• Most programme teams reported engagement with employers. Engagement was particularly well detailed where existing (pre-RoLEx) relationships were firmly established.

• Most consultation appeared limited in scope, reported as mainly confined to the sharing of plans and the receiving of feedback. However, some co-creation of programme design was apparent.

• Where co-creation of programme designs with employers was evident, this was often reported as being mainly located in the placement element of the programme.

• Some staff reported that, had they had more time, they would have liked to consult more thoroughly with employers.

• Employers were also used as a route to enable engagement with other stakeholders such as clients and service users. Though this is a statutory requirement for some programmes, it nonetheless attracted the interest of staff from other Faculties through STRAP.

j. Staff development and support

• We encountered many examples of effective and valued staff development through a range of usually collegial activities such as away days, seminars, and network lunches.

• Interview groups were largely positive about the first phase of RoLEx, where they had engaged in supportive redesign processes within their subject areas, benefiting from useful dialogue with colleagues in other disciplines.

• Many of the new module plans and explanatory statements contained descriptions of learning and teaching approaches which appeared to be new developments for the programme teams and was therefore taken as an indication of staff development and support.

• Staff who had participated in the PG Cert Learning and Teaching reported to have found it relevant and useful to the task at hand.

• Some staff reported a need for more staff development and support than was perceived to be available.

8. Conclusions and recommendations

Fully achieving the seven goals of the Learning and Teaching Strategy is a laudable and ambitious aspiration which will not be achieved without continued commitment. Undergraduate RoLEx is the first of a series of steps in this direction. Further progress towards achieving those goals would be facilitated by cultural change in Birmingham City University’s approach to curriculum design and delivery. Students need to be allowed, encouraged and empowered to take responsibility for co-creating their own learning processes. This would entail aims and learning outcomes in which self-directed and self-managed learning and development are central. Staff, too, will need to be encouraged and empowered to engage in design and delivery of flexible curricula which are tailored towards identifying and meeting the learning aims of the individual student as well as the needs of their prospective employers. This will require a change in staff culture from the delivery of academic content to facilitation of students’ learning. Action should be taken to develop a culture in which students regard themselves as active and responsible partners in the learning process rather than as customers.

From an organisational perspective we believe that Birmingham City University, in order to succeed in its aims, will have to establish processes and practices to recognise and nurture pedagogical innovation and excellence and encourage these to spread across course teams, departments, and faculties. It will be important to review current conventions about the roles and duties of academic staff to prioritise the greater focus on enhancement.

The University should ensure that matters of assurance and compliance are regarded and handled as complementary to the promotion of educational enhancement and innovation, rather than being regarded as distinct and dominant.

We further believe that it is only by continuing the journey towards creating a learning organisation that Birmingham City University will be able to develop and transform its teaching and learning activities and at the same time interweave matters of enhancement and assurance.

Specific Recommendations

For involving students

We suggest that it is very important for achieving the University’s goals that students are involved as co-creators in their curricula, and have the possibility of influencing how their studies link meaningfully to their personal development including employment plans. Recommendations to support this process are provided below:

1) Ensure that students are more involved in the next iterations of RoLEx. For many of the programmes across the Faculties, that means moving forwards from getting students’ opinions as an input to staff’s own redesign project to involving students as partners in the co-creation of the learning experience/curriculum.

2) Consider how to enable students to contribute to the formulation and communication of central processes and underlying ideas, for example by using language and forms of communication that appeal to students.

3) Arrange interactive situations such as workshops to draw together students’ learning experiences and teachers’ teaching experiences. Consider the use of neutral and experienced facilitators who can help course programme teams and students to undertake collaborative curriculum development.

4) Arrange that students’ feedback about their learning experiences is considered regularly and frequently, so that development becomes an iterative process where students’ views are always incorporated.

5) Promote activities in which students are encouraged and supported by staff to become action researchers of their own learning processes.

For involving external parties

The University will be assisted in its goals by moving to a deeper engagement with employers, alumni, external examiners, Professional and Statutory Regulatory Bodies, clients, service users and other stakeholders. Recommendations to support this process are provided below:

6) Enable external parties to play a much more significant role in the planning and delivery of the curriculum.

7) Establish transparent processes and use accessible language to draw external parties deeper into consultations about the effect of innovations in curriculum design and to obtain their advice on the value of the learning experiences of students in relation to their employability.

For involving staff

We recognise the enthusiasm with which many staff have engaged with RoLEx this year. Recommendations for building on the existing involvement of staff to encourage them to continue to support this process are provided below:

8) Ensure that staff throughout the University are involved in the next iterations of initiatives to enhance the quality of students’ learning experiences. Avoid such developments being exercises centred upon the activities of only responsible course coordinators and other select groups of staff.

9) Use the next iterations of RoLEx as a vehicle to promote a culture amongst the wider staff population where enhancement of students’ learning experiences is placed at the centre.

10) Use neutral and experienced facilitators who can help staff including managers, in their move towards an enhancement culture, where the focus is on the students’ learning experience.

For staff development

Recommendations are provided below to support staff in deeper engagement in future iterations of RoLEx:

11) Involve staff in planning their own continuing professional development (CPD) including and beyond the PG Cert, to meet the demands which they perceive to arise from RoLEx initiatives.

12) Consider offering a mentoring programme where champions in faculties can help other staff members to find their way through curriculum development projects, supporting their CPD in the process.

13) Help staff establish and maintain support networks to share information and good practice and support each other through change.

For management and implementation issues

In order for the University to manage and implement RoLEx processes in future, we make the following recommendations:

14) In future iterations of course redesign, avoid being caught up in a hierarchical structure where the people involved are, or feel, micromanaged due to rules and regulations surrounding the process of change.

15) Support change through staff away days, facilitated to ensure all staff have a voice, and include students as active participants.

16) Consider populating project boards and project teams according to personal competences, drive, and inspiration in relation to the merits of the task at hand, and not according to role.

17) Consider locating and using local champions who can step into the redesign project and facilitate and inspire (e.g. current and former members of “Learning and Teaching Task Groups”, University Learning and Teaching Fellows, National Teaching Fellows).

18) Develop adequate monitoring and feedback mechanisms to ensure that timely curricular revisions can be made to benefit current students.

19) Consider developing methods and models for learner-centred higher education that staff and students involved can learn from. Models of systematising innovation in learning and teaching have flourished in some areas of the university and the university should seek to replicate these successes across all faculties.

9. Concluding Remarks

Based on our evaluation, it is our belief that Birmingham City University has taken a first important step towards creating a culture in which student learning is at the centre of academic activities. There has been considerable commitment in the University community to the aims and potential of RoLEx and a substantial amount of curriculum enhancement has occurred as a result, which will no doubt improve the experiences of students and staff. Some initial difficulties have been met in the RoLEx-process which must be expected in a project of this size. Nevertheless, we are confident that the experience gained from the first iteration of undergraduate RoLEx will benefit the University in its future redesign of students’ learning experiences.

10. Acknowledgments

The three external evaluators wish to express sincere appreciation of the warm welcome they experienced in all their activities, of the frank and open nature of the many searching dialogues which they initiated, and of the thoughtful hospitality shown them by the University during their stay in Birmingham.

11. Conclusion

Members are asked to consider the recommendations in this report.

Stuart Brand

Director of Learning and Teaching

Annex 1 to Appendix C

Membership of RoLEx Evaluation Group

Professor John Cowan (Lead External Adviser)

Formerly Director, Open University in Scotland and Professor of Engineering Education, Heriot-Watt University. Emeritus Professor of Learning Development in the Open University.

Professor Claus Nygaard

Professor of Management Education with strategic focus on institutional research at Copenhagen Business School Learning Lab

Dr. Shân Wareing

Dean of Learning & Teaching Development and Head of the Centre for Learning & Teaching in Art & Design at the University of the Arts London

Professor Stuart Brand

Director of Learning and Teaching, Birmingham City University and RoLEx Project Manager

Paul Bartholomew

Senior Academic in Learning and Teaching, Birmingham City University and Lead Academic Consultant, RoLEx Project

Rebecca Freeman

Research Assistant, Centre for Learning Partnerships (CETL) at Birmingham City University and Co-Co-ordinator CETL National Student Network

Annex 2 to Appendix C

Constituency of meetings held by RoLEx Evaluation Group (REG) during evaluation week: June 15 – 19 2009

During the evaluation week a series of five meetings with groups of staff involved in RoLEx were held. The range of staff involved included Chairs of Faculty RoLEx Project Boards, Senior Learning and Teaching Fellows, staff from the Centre for Enhancement of Learning and Teaching (CELT), Programme directors, ‘grass roots’ teaching staff and others with significant roles in the RoLEx project. These meetings involved a total of 29 academic staff and all faculties were represented.

A further meeting was held with students: 11 undergraduate students, again drawn from all faculties, attended. There were part-time and full-time students in this group.

The above six meetings were each chaired by one of the External Advisers. Notes were taken by an internal member of REG.

In addition two of the external advisers met with the Vice-Chancellor to discuss his perspectives on the RoLEx project.