Research Project Information Form – 2_RPIS OLTExt_13April_USC
Workshopping the mapping process with staff was important in getting initial engagement, particularly as the mapping process in some cases led to collaborative problem solving around curriculum and assessment structure. It was important to create a critical space where academic staff were able to discuss the relationship between the content and the attributes. Along with this buy-in amongst academic staff maps are more likely to reflect the program and unit content when the staff involved in delivering the context are involved.
At UTS Business School the emphasis on a participatory process involved sitting down with subject coordinators and having them work through how the graduate attributes and program learning objectives fit into their subject. Using the Subject Overview Spreadsheet (SOS), subject coordinators collaborated in not only the mapping of attributes across the program, but identifying and resolving issues around the distribution and gaps in the curriculum. While the teaching and learning team facilitated the process and did some of the early work of entering details into SOS to hand back to the subject coordinators, the process centred on the involvement of academic staff.
At the Australian School of Business at UNSW, initial work on mapping was done through workshops where unit coordinators in program/discipline teams were asked informally to indicate which graduate attributes were involved in their assessment tasks. Using Post-it notes, they were asked to map out the distribution of the attributes across assessment tasks through a program or major, from which a number of gaps and overlaps were identified and discussed. The resulting maps from this exercise were developed by the ASB Teaching and Learning team, and then presented back to the program directors and unit coordinators, who were then responsible for any changes.
Griffith Business School held a number of workshops off-campus, where staff worked through charting the learning goals over the course of the program on butcher’s paper. This included unit and program coordinators, heads of departments and the dean of learning and teaching. The process of refining the map was continuous, primarily taking place by email, but with additional yearly workshops to go over the process again to make sure the mapping reflects the way the unit is being delivered.
SCU engage in participatory mapping by email, sending out a spreadsheet with the attributes, which lecturers fill in for their individual units. The collaboration and negotiation occurs at the level of discipline groups who share out the assessment of the required attributes across the degree or major. A process of reflecting on the coverage of graduate attributes at the end of the semester also feeds into this.
Development of the law program at Edith Cowan University began with the embedding of new university level attributes, along with the threshold learning outcomes into the program. This came about as part of a collaborative process of workshopping the TLOs and capturing current practices in order to identify strengths and gaps. This then lead into a redesign of unit and learning tasks guided by the gaps identified.
The process of manually working through how programs and units align with attributes was often talked about as being important in getting subject coordinators to take a program level view of attributes. Significant engagement seems to start from the moment that subject coordinators could see the bigger picture of the program and how their unit fits with the development and assessment of attributes. This often created the opportunity for collaborative problem solving in addressing gaps, or in identifying assessments in the program that reflect the attributes for measurement. Most of the participants described a very similar process of teaching and learning staff presenting the graduate attributes to unit and program coordinators and having them map the attributes throughout the course or unit. This was typically done with the use of specialist software, spreadsheets, butchers paper, or post-it notes.
At UTS Business School, specially developed software (Subject Overview Spreadsheet) was used to present how unit level assessments fit in at a program and faculty level, using program and subject coordinator’s own knowledge of the program. The presentation of this information through SOS made gaps and overlaps over the course of the program clear, and also identified how particular assessment types (multiple choice, essays, case-studies) were distributed over the course. Being able to present all this information seemed to be important in fostering a program-wide view.
An ongoing process of review takes place at the Faculty of Business and Economics at the University of Melbourne where program directors are asked to code all unit objectives against program learning goals and outcomes. A filemaker pro database is used to present how the program learning goals are distributed over the units. The mapping is updated every year, which feeds into a program review every five years.
In the QUT Business School all programs map to five key learning goals. For majors within programs, goals are adapted to reflect discipline needs. Mapping was initially done in the core units, and then discipline staff were given the task of building on the core units and showing the sequential development of program goals across the units within the major. This required taking a view of the program as a whole and observing how units fit into the program in relation to attributes.
In the law school at Charles Darwin University the degree was developed collaboratively with staff. While it was a complex and time consuming task, it was thought to be formative for the teaching team as everyone was able to get a program level view and ownership of the program as a whole.
Mapping the development of attributes over the program helps to encourage a more systematic discussion about curriculum structure beyond assessment near the end of the program. Mapping the progression towards the graduate level of attributes created opportunities to consider how to improve development through changing the substance or structure of a program. This also meant that there was multiple sets of data allowing some analysis of the progression as well as assuring the learning at the end of the program. This additional data creates the possibility for very detailed changes to program, courses, and assessments that have been shown to be effective in developing a particular attribute.
As SCU map by course rather than individual assessment items, lecturers are asked to indicate the extent to which each attribute is covered in their course: 1) content only; 2) attribute is assed in the course; 3) cannot pass the course without demonstrating the skill. The map of the program provides a broad sense of where the attribute is developed and assessed.
CQU build levels of attributes into their process, along individual assessment items. Learning outcomes and assessments are rated in terms of the level of the graduate attribute demonstrated: 1) introductory; 2) intermediate; 3) graduate. These levels are used to show that the relevant attribute has been developed over the course of the program.
Mapping at UTS BUSINESS SCHOOL involves working out and communicating to students where the program learning objectives are introduced, developed and assessed. This allows for more detailed adjustment when it comes to closing the loop, but also is designed to promote reflection amongst students on the progression towards graduate level.
In the School of Law at Flinders, the law program was redeveloped over a series of meetings where there was discussions about the sequence of skills developed in the program and where performance of the skills at an introductory, intermediate and exit level are assessed.
Mapping at the assessment level allowed for a greater level of clarity and detail in understanding the distribution of graduate attributes over the program. Where this occurs staff were able to very clearly indicate what activities were undertaken to develop and assess the attributes, unpacking the black-box of courses to allow for reflection and improvement at this very detailed level. There was also the opportunity to communicate to students the relationship between the assessment task and the graduate outcomes, and in some cases students were asked to reflect on the last time they undertook work related to that attribute. For universities with embedded assurance of learning measurement, this was also useful in being able to draw on the mark s for the parts of an assessment related to the graduate attribute.
At UTS Business School there is a descriptor for every course that contains all the learning objectives in the UTS handbook. For each of the assessments in that course the course outline will show which program learning objectives are being addressed as well as subjective learning objectives. These maps are available to students so they can see of particular skills are developed over the course of the unit. Students are also prompted to look back at the last time they addressed that particular outcome in their program.
In the QUT Business School assessments are marked against criteria that are linked to a program learning goal with the link made evident in ReView. The program learning goals encompass all of QUT’s graduate attributes. This means the mapping shows how each assessment relates to the graduate attributes.
The process of mapping at CQU involved every unit coordinator mapping each of their learning outcomes and assessment items to the graduate attributes, allowing gaps in the program to be identified.
A number of universities talked specifically about having students aware of and reflecting on the graduate attributes. This was seen as an important part of getting students to get a sense of progression over the course of the program so they could reflect on and engage in their own learning. The next step beyond this was to turn student engagement in thinking about graduate attributes into a portfolio of achievements towards particular attributes.
A fairly unique system around graduate attribute mapping was used at Curtin’s Distance Education School. Graduate attributes are mapped against an employability skills framework. Students are asked to record their past extracurricular learning and previous studies, which are then combined with their current studies to produce a Career Point Index in line with the graduate attributes. Opportunities are then delivered in line with building up aspects of students’ Career Point Index through extracurricular learning activities. Students are encouraged from early in their program to start planning and developing their Career Point Index aligned to their desired career path.
The e-portfolio in the Bachelor of Business at UTS BUSINESS SCHOOL, is built around the graduate attributes. Students are prompted to find examples of how they have demonstrated each of the attributes through their course work and extracurricular experiences. This served to not only highlight the skills and attributes they had developed over the course of their studies for themselves and future employers, but helped to identify areas for further development.
The School of Law at UTS received a grant to develop a website that explains to students the importance of the graduate attributes contextualised to careers in law. It includes an explanation of different levels of performance of these attributes, how they link to legal practice, and a number of checklists for students to consider where they are in the development of the skills. Of particular note is the inclusion of videos of legal practitioners explaining the importance of the attributes to their work
Each semester, a meeting is held for a formal handing over of the data related to each major in the program (AoL, ICR, and LEX data). Each discipline group sits together to consider the data through very robust discussion of the data and its meaning. The visual presentation of the data in graphs and charts and turning this handover of data into an event was talked about as being important to fostering engagement. One of the most important factors that proved to be significant in engaging staff was the depth of view of the data that was available: a macro view of how students performed against each learning goal in the major, but also a micro view of how students performed against every criterion in every assessment item in the major. This meant that staff were able to make very specific analyses and identify those aspects of assessment and the learning goals most requiring improvement. This facilitated staff being able to readily identify direct and tangible improvements, and enabled staff to be very engaged in the process. A further very important factor was the insistence on engaging discipline groups, enabling them to own and take responsibility for the integrity of their discipline majors.
QUT Business School’s implementation was initially driven by a university wide policy change to criterion referenced assessment (CRA). Assessment champions were identified in each discipline school to guide the implementation of CRA. Having prior knowledge of CRA made the transition to mapping learning goals and aligning assessment criteria with learning goals a much easier transition. These assessment champions worked with chairs of discipline school teaching and learning committees and together formed a critical mass locally to support the discipline leaders in the mapping of learning goals and in influencing the attitudes and behaviours of colleagues towards the cultural change.
The undergraduate and postgraduate program coordinators work with the discipline leaders for each major in their program. The discipline leader was responsible for coordinating the mapping activities for their major and is now responsible each semester for engaging the discipline team in analysing and reporting student performance against learning outcomes and, where warranted, identifying improvements.
Delegating leadership responsibilities to key people who were able to influence colleagues created buy-in and eased the transition through interpersonal influence.