- Philosophy: Participants emphasised the ability to identify and address gaps, and to better elucidate the process of their programs.
- Motivations: Accreditation, student expectations about the acquisition of skills, and the importance of graduate attributes to the profession.
- Mapping: Most of the schools with developed programs engaged in some kind of mapping, although often it reflected an out of date template. Newer schools were actively engaged in developing their programs through curriculum mapping.
- Data Collection: Two of the smaller but established schools had data collection processes in place; one with consistent criteria for graduate attributes and another without.
- Closing the Loop: Closing the loop primarily involved student feedback information, although one school reported that data collection fed into course review processes.
- Challenges: The most common challenge was how to manage assurance processes without adding to academic workloads. There were also some challenges in fitting the rigours of a pharmacy program into the university’s restrictions around contact hours.
- Solutions: External advisory panels with staff and students looking at assurance, flexible processes that can work around the preferred practice of teams, and having staff with experience in curriculum design.
Philosophy for Assuring Learning
The participants emphasised the concrete things a systematic approach to curriculum was able to do, such as identify and address gaps. Overall the described the importance of elucidating the process by which their programs go about producing well rounded, practice ready, and employable graduates. This suggests that the primary philosophy in place is based around accountability to the profession/industry and to students undertaking the programs.
Motivations for Assuring Learning
In terms of motivation for assurance of learning, participants talked about accreditation, student expectations about the skills they will develop, and that the graduate attributes represented the skills required for success in the profession. The participants suggested that accreditation prompted the push for a systematic approach to graduate attributes, but in concert with other motivators. There was an acknowledged responsibility to ensure that the program met student needs and that they had the chance to develop the skills to be practice ready. Beyond this was a more general commitment to produce competent practitioners, with a sense that assuring graduate attributes was central to this. Marketing and program development were also mentioned as motivators.
The participants talked about the relationship between the different levels of graduate attributes across the university and faculty, primarily that pharmacy program level requirements fit in quite well with university graduate attributes. Most of the schools said that the university graduate attributes were not mapped directly into the program, rather that the program outcomes fulfilled Australian Pharmacy Council (APC) requirements, in a way that contextualised the attributes to the pharmacy discipline. One school suggested that there was some difficulty in keeping to the generic outcomes required by the university, particularly in the context of trying to fit these into the limited course time available with students.
There was a brief discussion about how the program level learning outcomes had come about. One school talked about a course review process, where through a school level discussion they blended APC professional standards, university level outcomes, and some program specific outcomes they felt were important. Another school engaged in more of a ‘brain dump’ of program learning outcomes from the teaching team, which the team has continued to work on refining and reducing. In a more systematic approach, another school worked on a number of long-term research projects looking at graduate attributes and employability in the discipline, and student, staff, and employers’ perceptions of particular graduate attributes. This ongoing work fed into continuing review of the graduate attributes in the program.
For some of the schools, the task of mapping was ongoing as they were engaged in seeking accreditation, or maintaining provisional APC accreditation. This process involved starting with the planned activities and then working out the spread of graduate attributes in those activities with the teaching staff. The people leading these processes across these two schools worked in small cohesive teams, and viewed the challenge of developing the program from scratch as a positive one. Among the smaller, but established pharmacy schools, there was comprehensive mapping at program, unit, and assessment level. One school relied on capstones to address graduate attributes that were kept separate from the pharmacy content. For another school while a rough curriculum map existed, they were engaged in the process of working with course coordinators to ensure the map reflected the actual curriculum. In one of the more established schools, there was a kind of broad map, but there was a lack of formal process other than teaching feedback to ensure that graduate attributes were developed over the progression of the program.
In terms of the tools used for the mapping, most schools used generic software (i.e. Excel or Access), with one school using software developed by the university teaching and learning team that had been developed from an ALTC grant.
As many of the schools included were in the early stages of accreditation and were still setting up their programs, most did not have any graduate attribute data collection in place. In particular the more established schools also did not have any data collection in place, referring only to student satisfaction and outcomes surveys.
Two of the smaller schools reported having data collection processes in place. One school had specific assessments that are mapped to assure graduate attributes. These involve the collection of whole of assessment marks, part-marks for some criteria (e.g. communication skills), and combined with student experience surveys. No specific tool is used to collect and mange this data. Another school had a system in place to collect part-marks for specific criteria related to learning outcomes, yet these are marked with no real consistent criteria, limiting the usefulness of the data for program improvement.
Closing the Loop
Beyond the discussion and review of student satisfaction information, there were limited formal processes for closing the loop with student outcome data. One of the schools that had a well developed data collection process had a review of the data as part of a cycle of improvement, with refinements to learning outcomes sometimes resulting.
The participants spoke about a variety of challenges that tended to differ based on how developed the assurance processes were at the school. The most common challenge was how to manage processes without adding to academics’ workloads. Another common challenge was working around university wide directives that make it difficult to properly assure learning. Pharmacy programs are quite intensive, and the participant reported difficulty in complying with the university requirements for their graduate attributes, as well as restrictions around the number of assessments and contact hours. Some of the university’s directives were thought to clash with the rigorous requirements of a pharmacy program. Working with the university’s requirements also added another set of paperwork on top of the reporting required for APC accreditation.
Other challenges involved how to properly assure programs that were primarily delivered externally, the lack of guidance from the Australian Pharmacy Council on the mapping of graduate attributes, and the challenges of mapping and defining the program learning outcomes. Participants also found many of the existing tools for mapping not suited their purposes.
Some of the broader challenges discussed were how to include students and academics as key stakeholders in the process, how to assess and assure graduate attributes beyond summative assessment, and how to work with a program structure that followed a generic template that doesn’t consider the requirements of a pharmacy program.
The participants reported staff engagement to be particularly strong among small teaching groups, and where the program was being set up, yet they reported inevitable challenges in getting academics’ time, whose priority is research.
Solutions to Challenges
Participants described how they and their school had dealt with some of the challenges they described. Engagement with students on the wording and processes around graduate attributes had been attempted through external advisory panels with students. One participant felt that they had been able to get staff engagement through flexibility in the processes, to fit around the preferred practice of teams. Online resources had been used to solve some challenges in the assessment of group-work among external students. Having a program coordinator with experience in teaching and learning, and curriculum design was also thought to be an important part of a sound strategy to assure learning. Having links to other disciplines (speech pathology in particular) was suggested as helpful in creating a shared sense of what a sound process looks like. In a regional setting, having a strong connection to pharmacists in the community helped to emphasise that the skills being built into the programs were important for graduates in this discipline to develop. Amongst the small teams, they reported engagement being much easier, particularly when they were in the process of developing a program from scratch; there was an ability to collaboratively discuss and define the vision of the program.
In terms of the sustainability of the process, while the participants agreed that sustainability was important, they weren’t sure how to go about it, particularly in small schools where processes relied on the rapport of small teams. One participant suggested that they were aiming to make assurance part of the regular process of program development, delivery and review. They thought they would get this through embedding reflection into academics’ normal processes of review, and get cultural acceptance of the value of the process.
- In terms of the philosophy of their approach, participants primarily identified accountability to students and other external stakeholders in order to protect the integrity and reputation of the school. Curriculum improvement was also mentioned by some participants.
- Accountability also figured prominently in the motivators for assurance of learning, again related to the reputation of the school, but also accreditation requirements.
- From the 17 law schools represented 14 (82%) had some formal mapping process in place to structure the development of skills and assure learning at particular points in the program. The approach to mapping involved either: (a) individuals or teaching and learning groups undertaking the process; (b) teaching and learning staff engaging the faculty in the process; or (c) the work being delegated out to unit and program coordinators. Mapping tools were generally not used.
- Data collection was relatively uncommon amongst the schools included with only two schools (12%) undertaking the collection of student learning outcome data. Participants felt that their assurance of learning came down to the identification of assessments that represented a good test of a particular skill.
- For schools that collected data, closing the loop involved a committee with broad representation across the faculty. Participants without assurance data collection also engaged in improvement processes, using overall student performance in particular units and assessments.
- In terms of the challenges, participants talked about: integrating the law threshold learning outcomes into their existing processes, dealing with regulatory arrangements that don’t have teaching and learning considerations at the centre, the teaching of skills, time constraints, student awareness and engagement with attributes, the different contexts for regional/distance universities, and unrealistic expectations of employers. The main challenge identified was staff engagement, which participants felt may be particularly difficult in the law context. Participants talked about resisence from (sometimes) small groups of staff that were resistant to any discussion about teaching and learning.
- The strategies used to resolve some of these challenges centred around professional development and support for staff, demonstrating organisational commitment to assuring learning through leadership, and communication strategies alongside collaborative processes.
1. Philosophy of Assurance of Learning
Three main themes emerged as the philosophy behind the schools’ approach to assurance of learning: accountability to students, external accountability, and assurance as a framework for curriculum improvement. Seven participants talked about the responsibility to be accountable to students about the skills and qualities they will graduate with, and the importance of those skills for their careers. Six participants mentioned that accountability to external stakeholders was central to the school’s approach, that the university had a prerogative to deliver on the industry’s expectations of graduates in order to be competitive, and to avoid misrepresentation. When asked about the philosophy of the approach, four of the respondents talked about assurance as enabling curriculum improvement processes. Specifically that it enabled improvement through: the articulation of end of degree goals, the embedding of skills into discipline knowledge, and the alignment of programs along learning outcomes.
2. Motivators for Assurance of Learning
Accountability also featured prominently in the motivators discussed by the participants. Five participants talked about the approach being driven by a need to assure that students have the skills expected of graduates. These participants talked about the reputation of the faculty as central to assurance, specifically in the quality of their graduates in the legal profession. It was also noted that many law graduates have careers outside of legal practice, so checking skills was an important process on top of what occurred for accreditation. Accreditation was identified as a motivator by five participants in the sense that it was important for continued accreditation that students achieved the required graduate skills in addition to the subject content. This linked into the discussion about the adoption of the threshold learning outcomes for law.
In terms of motivators, five participants talked about the broad changes in thinking brought about by the threshold learning outcomes for law. Almost all participants indicated that these outcomes had been adopted into their curriculum, or mapped to existing student learning outcomes. Doing this was seen as heading off future demands by TEQSA or accreditation bodies, along with being able to being able to focus curriculum review processes on skills embedded in discipline specific content.
Four participants indicated that assurance and a learning outcomes agenda was being driven by university level policy or procedural requirements to build in graduate attribute assurance. Only two participants indicated that teaching and curriculum improvement were motivators for assurance processes in law programs.
3. Approaches to Curriculum Mapping
Development of Attributes and Learning Outcomes
The participants described the relationship between the different layers of graduate attributes the programs were responsible for developing in students, and the process that led to outcomes being developed and adopted in the school. Primarily they described a situation where the university wide graduate attributes had been developed into law specific learning outcomes. The development of these outcomes usually occurred through a process of consultation with the faculty, either mapped to or related to the law threshold learning outcomes.
Communication of Attributes
While graduate attributes were commonly communicated through the usual channels of enrolment guides and online course reference material, many of the participants acknowledged that students are not particularly engaged in thinking about the graduate attributes. There was an expectation that academics were communicating learning outcomes in the units and prior to assessments, but not a clear sense of how effective they were in engaging students in thinking about them. One of the participants talked about a detailed website that had been developed to raise student awareness of the graduate attributes in the context of the law program, which had been a successful tool in engaging students in reflecting on their learning.
|Schools with Mapping||Schools with no Mapping||No Response to Question|
Most of the participants (82%) reported engaging in some kind of mapping process, identifying where either the graduate attributes or program learning outcomes were developed in particular units. For the two participants that reported that there wasn’t any mapping of the graduate attributes to their courses, there was a reliance on individual academics’ informal understanding of the attributes and which units were associated with their development.
The participants talked about the approaches taken to mapping. Five participants described a process that was undertaken and managed by individuals or workgroups independent of the teaching of the course, primarily curriculum coordinators or teaching and learning committees. This approach tends to involve some consultation with the faculty on the content and structure of their programs, but primarily the work was managed in such a way to minimise the amount of time academics need to spend in complying with these processes. Participants talked about the allocation of graduate attributes to particular units, as opposed to the kinds of collaborative discussions about embedding skills that occurred in the next group of participants.
The second group talked about a long process of engaging staff in discussions about program structure and the distribution of graduate attributes in units. Four participants described a process of robust and critical discussion on the best way to align the curriculum, identifying points for the introduction, development, and assurance of particular skills through assessments. An initial phase of working with the faculty was usually followed with the production of a formal curriculum map for comment and feedback from various teaching and learning groups in the university.
The final group approached mapping as a task to distribute out to unit and program coordinators, with the mapping done primarily by email. The participants talked about some of the problems with this approach, primarily that there was no clear sense of how particular assessments develop skills and coordinators tended to indicate that their units developed all skills.
|Schools with Unit Level
|Schools with Assessment &
Unit Level Mapping
Only three participants reported mapping at the unit level, otherwise mapping focused on the role of the assessment in developing skills or in checking that students had the required level of a particular competency, serving as a ‘hurdle assessment’.
A number of participants talked about emphasising the embedding of graduate attributes into the law content set out by the Priestly eleven. Some participants indicated that skills tended to be separate to content and delivered through skills oriented units that form capstones for specific skills that are more indirectly developed through the content oriented units.
For the most part the participants indicated that no particular tools were used, just a piece of paper or a generic excel spreadsheet. Three participants indicated that excel based templates were used such as the Subject Overview Spreadsheet (SOS). A number of participants had indicated they had used particular mapping tools for a time, but that they had fallen into disuse as they weren’t suited to the task.
4. Approaches to Data Collection
|Schools with the Collection of Student Learning Outcome Data||Schools with Student Learning Outcome Data Collection||No Response|
Data collection was relatively uncommon among the participants’ law programs. When asked about how they assure graduate attributes in the program, participants talked about the role of mapping in identifying assessments and units that represented a good test of a particular attribute. Some participants talked about the use of student feedback surveys as a source of data on their achievement against graduate attributes. One participant indicated that while there were no specific barriers to collecting assurance data, this was unlikely to occur until it was a requirement of the regulatory authority.
The two participants from schools that collected data on the student achievement of learning outcomes had the embedded measurement of learning outcomes in assessments. This meant that the rubrics based on the learning outcomes were part of the assessment criteria, were communicated to students, and were marked by the academics themselves (as opposed to an external skills assessor). One of these participants talked about different systems of data collection for skills as opposed to knowledge; that skills sit alongside the degree and are measured in skills units, while data collection on knowledge oriented learning criteria were collected across a number of assessments, using part marks for that specific criteria.
5. Closing the Loop
The two participants that indicated assurance data was collected talked through the review processes that took place regularly within the school. This involved the Associate Dean (Teaching & Learning) and a sub-committee of the university teaching and learning committee that have a strong background in legal education. Changes are then discussed with the relevant unit coordinators. This review process occurs in 3-5 year cycles, although ad-hoc reviews occur that deal with minor changes each semester.
Participants that did not have student learning outcome data collection also discussed some processes of closing the loop, using overall student performance in particular units and assessments as the measure of student learning. This review processes primarily occurred as part of the course accreditation process.
6. Challenges & Solutions
Challenges of Assuring Learning
Law Threshold Learning Outcomes
The challenge of incorporating the law learning threshold outcomes into the curriculum featured prominently in the discussion with participants. One participant said that while academics were often dubious of the value of the learning outcomes linked to university level graduate attributes, there was a strong acceptance of the value of the TLOs as they had endorsed by the Council of Australian Law Deans. Some participants did mention improving staff awareness of the outcomes as a challenge. That said, some participants thought the outcomes could be challenging to use as the basis for measurement, particularly as they would likely be evaluated externally, which would complicate their interpretation.
As mentioned above, almost all participants indicated that their curriculums had incorporated the threshold learning outcomes into the curriculum either directly, or by linking them to graduate attributes that were built into the programs. Some participants indicated that linking the threshold learning outcomes to the faculty or university graduate attributes was a difficult exercise despite containing similar criteria.
Participants were critical of some of the regulation associated with accreditation processes, with time and effort spent on processes that they felt didn’t enhance teaching and learning. There was a sense that regulation in this area loses sight of the goals of legal education, and that accreditation authorities had a limited knowledge of pedagogical research. Alongside this was a sense of uncertainty about the regulatory arrangements, primarily to do with TEQSA, but also around professional accreditation standards.
The Teaching of Skills
Designing learning activities and assessments that develop and/or measure skills presented an ongoing challenge in the schools at a number of levels. While the primary challenge was engaging staff with these challenges, these processes present considerable challenges in and of themselves.
While most participants saw the value of the graduate attributes as they were expressed in the learning outcomes for their law programs, others had a number of criticisms of these attributes. Participants reported problems where the attributes were too numerous, not properly adapted for the legal context or aspirational rather than concrete. Proper articulation of the attributes was also reported as a challenge by some participants, particularly vague attributes tended to result in lengthy debate about their meaning amongst the faculty.
In terms of the attributes being incorporated into units, some participants reported challenges with the coordination of outcomes across programs. This included improving the level of information about what outcomes are associated with particular units, ensuring that units aren’t substantially changed, and ensuring that skills are actually represented in units. Teaching and learning teams often had to engage in difficult discussion about what skills were represented in particular units and how they were actually being developed. A constant challenge in the schools was embedding the teaching of skills within law content areas enshrined by the Priestly Eleven.
The direct teaching of attributes presented faculty wide challenges in developing effective assessments that added to and assessed these skills in a way that was connected to content. A subset of these challenges was the language of rubrics that clearly describe levels of achievement against skills in the context of assessments. Participants in schools that had plans to begin collecting assurance data talked about the challenges of moving to assuring outcomes with data collection. The difficulties discussed were primarily in coordinating the process; participants felt that graduates currently do come out with the skills, just that it’s currently not documented and evidenced.
Time constraints presented a problem in two senses: getting space in the program for skills, and for academic’s time. While some participants were engaged in embedding skills in content, others saw skills as a separate area of learning that competed and butted up against content learning. The difficulties in getting academics’ time for teaching and learning related tasks were a common experience amongst the ADTLs. There was a sense that despite many universities attempting to build quality teaching into the promotions process, research was still what was valued institutionally.
Student Engagement of Attributes
While participants reported having strategies in place to build student awareness of graduate attributes, it was also often acknowledged that it was unclear if students thought about these skills. One participant thought that students questioned the meaningfulness of graduate attributes; focus groups with students had found that they thought the graduate attributes were more marketing than a teaching and learning principle. Another participant suggested that while what it means to be a university graduate was well established in some communities, broadening participation in higher education meant that the graduate attributes needed to be communicated better.
Different Educational Contexts
Some of the different contexts universities operated in presented challenges to assuring learning. Some of the regional universities complained of inflexible general models applied by government and professional accreditation bodies that were impractical in their context. Likewise units delivered externally presented challenges for assuring that students develop particular skills. A number of participants talked about getting an ever more diverse student body up to the required standards. Generally, having a large number of electives in a degree complicated efforts to structure skill development over the program.
Industry Expectations of Universities
Participants talked about the weighing up of industry expectations and the consultation done in developing learning outcomes. While recognising the importance of connecting the program to industry requirements there was also a sense that these discussions were very one-way. The university was often engaged in attempting to meet employers’ demands for graduates that they be ‘earners’ from day one, which may be unrealistic. Moreover the development of curriculum to attempt to meet these standards may in fact be setting students up to fail.
Staff engagement is the primary challenge participants talked about, particularly where a participatory and collaborative approach to assurance processes had been attempted. While some more general challenges apply in engaging staff, the participants felt there were some that were particularly pronounced in the law discipline. Efforts to engage staff in a collaborative approach may be particularly difficult in law where debates about the meaning of words can be particularly ferocious. A participant also pointed out that lawyers have a strong sense of procedural fairness, meaning that conducting a process that claims to be collaborative will have to live up to the meaning of the word.
Participants talked in some detail about the challenges of converting legal academics, who are content area experts into the view that they are also legal educators. There was a sense that while some staff are engaged in thinking about teaching and learning practices, others are resistant and sceptical of efforts to move to a coordinated program. This split seemed to be associated with the tensions between staff engaged in active legal practice, and staff who identified as law academics and educators This view seemed to be linked to traditional ideas of legal education; education as a kind of knowledge transfer and modelling behaviour. Participants talked about a total ignorance of teaching and learning practice among some staff, and an active resistance to engaging in the teaching and learning discourse or any kind of professional development. There were additional challenges in staff teach things they don’t agree with and don’t feel confident in assessing. While staff were dedicated to teaching and learning, participants felt it was a very slow process engaging people and convincing them that there was pedagogical value in assuring student learning outcomes. Staff were also fatigued from the pace of change and constant reforms in the sector, requirements to assure student learning outcomes presented as another well-meaning reform.
Part of the challenge of engaging staff in this work is the lack of institutional recognition and reward. One participant pointed out that assurance is not the kind of thing people come into academic life to do, there’s a need for motivated, capable, and talented people, yet a lack of incentives for academics to spend time on this. Despite teaching figuring into promotions alongside research in theory, there is a sense that research is actually what is valued institutionally. Also challenging in engaging staff is the long term nature of efforts to change attitudes and practices, yet staff in teaching and learning roles tend to be in 2-3 year cycles meaning that engagement and rapport can be lost as people move around. Parts of the faculty are also quite mobile meaning that champions can be lost as they move to other positions and other universities.
The challenges described above have been addressed in variety of ways, primarily through attempting to apply a collaborative approach while demonstrating institutional support for the process.
Professional Development and Support
Developing the capacity of staff to engage in assurance processes, and to be able to teach and assess graduate attributes was an important part of fostering engagement. Participants talked about informal sessions delivered by staff from teaching research centres, and from people within the faculty that acted as champions for assurance. Long term professional development and information strategies around the graduate attributes were offered by many participants as the way to effective engagement. Alongside these sessions was the support in place to help staff with the administrative demands of assurance work, particularly mapping, developing and using rubrics, and the teaching of generic skills. Staff also developed their own communities of practice in order to build their own collaborative capacity to engaging in assurance and to teach and assess generic attributes.
Demonstrated leadership was an essential part of engaging staff, showing that assurance initiatives have high level institutional support and that the efforts of staff are valued. Leadership at the head of school and associate dean level was important, but also leadership through dedicated champions at the uni and program coordinator level. Having dedicated staff able to communicate the value and benefits of assurance processes from their own experience were particularly effective in fostering engagement.
Communication and Collaboration
An essential part of developing engagement with the faculty was the development of rapport and relationships with staff, which primarily came through collaborations with staff on parts of the process. Reinforcing the usefulness of a coordinated program approach in aligning the curriculum and understanding the existing competencies of students, can also be effective in selling engagement in assurance. Alongside effective communication is meaningful consultation and collaboration with staff to raise awareness of what is occurring and what, and to bring staff along for all parts of the process. Consultation and collaboration is particularly important where there are changes in the units that come from assurance processes.
Part of the approach to communication was the work in raising awareness of graduate attributes among both staff and students. One participant talked about a web-based interactive tool that had been developed to raise student awareness of the graduate attributes developed and their relationship to professional legal practice. Another participant talked about the inclusion of questions about the graduate attributes on student experience surveys of particular units. The inclusion of graduate attributes in unit and program reports also helped to promote discussion and awareness.
Some of the participants felt the best way to engage academics in assurance processes was to demonstrate the curriculum improvements that resulted. One participant talked about the importance of constant updates about what was happening, particularly changes that occurred as a result of reviews. Explaining that teaching teams were subject to many different demands related to curriculum review, another participant thought it was important to immerse academics in one demand at a time so there is a clear connection and awareness of the effort involved and the benefits derived.
- Philosophy: Clarifying the relationship between the program and the development of graduate attributes, assuring learning has occurred, and improvements to curriculum;
- Motivations: Compliance with Engineers Australian accreditation requirements, providing the industry with competent graduates and improvements to teaching and learning;
- Mapping: Programs were often mapped to both the EA requirements and the university level attributes, while some schools used the EA requirements to demonstrate the development of the GAs. While all the participants had curriculum maps they varied in terms of if they were mapped at the assessment or unit level;
- Data Collection: Half of the schools included were engaged in collecting assurance data. One school used full assessment marks, while two others used part-marks related to unit learning outcome criteria;
- Closing the Loop: One of the schools collecting data had a well developed formal process for closing the loop. Another undertook data collection primarily for accreditation purposes and had limited feedback to curriculum from this data;
- Challenges: Communication and engagement, the use of technology, the sense that staff did not have the skill set to teach these skills, preparing for TEQSA and the AQF, and the fact that students could pass the unit and not develop the GAs without proper assurance. Staff engagement as a whole had not been a particular problem, but to suddenly involve staff at the point of mapping without previous engagement could be challenging;
- Solutions: Professional development and community, and the proper resourcing to build capacity.
Primarily the philosophy around assurance was oriented toward clarifying the relationship between the program and the development of attributes and learning outcomes. It was thought that it was important to clarify this for both staff and students. The point was made that the difference between a technical education and a university education is the development of personal and professional qualities, in addition to the requisite skills required by the industry.
Also mentioned was the need to assure learning, particularly as a means of comparing the standards of different institutions in this discipline, improvement to the design and delivery of courses, and to plan the development of graduate attributes in programs.
Primarily assurance processes were motivated by compliance and accreditation to external bodies, namely Engineers Australia and TEQSA. The point was made that there is no degree without the accreditation, particularly in a discipline like engineering. Also mentioned was compliance with requirements within the universities; a number of institutions were engaged in comprehensive curriculum reform with standards drive at the university level. Concern for student’s careers and the ability to provide industry with skilled and competent graduates were also mentioned.
Assurance was also motivated by improvements to teaching and learning, and by enhancing student understanding of attributes. One university mentioned professional recognition, to be able to assure standards to the industry and employers about the quality of graduates. Another specifically talked about the Engineers Australia Changing the Culture report as a motivator.
All the universities were fairly similar in that their programs were mapped to both the Engineering Australia outcomes, and to their university level graduate attributes. On the university side of things, engineering programs all worked with a set of faculty or school level graduate attributes, reinterpreted from the university level attributes. The connection between the Engineering Australia competencies and the university graduate attributes tended to not be problematic. Participants reported working around the EA requirements and using these to demonstrate the development of graduate attributes, although a number of participants reported that in their programs these were mapped separately. Where there were graduate attributes that were not addressed by the EA standards.
The subject learning outcomes are primarily determined by a mix of the Engineers Australia standards, and an interpretation of what the university graduate attributes mean in the context of the discipline. A participant described a process of constructive alignment, through workshops with course coordinators to develop learning outcomes alongside the existing course profiles. This process has since been exported across the university. One university indicated their subject learning outcomes directly reflected the competencies for the three levels of registration set by EA, complimented with generic skills, and discipline specific competencies related to the major; there was in effect a line between these different types of competencies. One of the participants described the level of autonomy the subject coordinators have and that the content of subjects can drift, particularly in relation to the university level attributes. There is an understanding that these attributes are fulfilled by the engineering threshold outcomes that are embedded in the program.
Mapping forms a part of retaining accreditation, yet some universities went beyond this to incorporate non-EA outcomes in maps, and to have more detailed mapping processes in place in order to understand the curriculum. Some of the programs reported a well developed mapping process of working with academics to connect learning outcomes to subjects and assessments. A participant talked about getting an internal grant to hire a person to sit down with subject coordinators, work through the graduate attributes, how they are applied in subjects and assessments, and the rewriting of the subject outline. Two other participants talked about the importance of the connection between the program, subject and assessments.
Some of the schools were still in process of developing their approach to mapping. A participant talked about a lack of development due to the database system used, and that the school had been engaged in drawing on course profile data to begin developing more sophisticated maps of where graduate attributes fit into the subjects. This was still not yet at the level of assessments though. At another school, mapping had been a long and still ongoing process of understanding the content being delivered in units. This began with a rough map showing the connection between assessments and graduate attributes, and then drawing on the course profile system to identify the subject learning outcomes to the EA competencies. This process is not yet at the level of mapping by assessments. Much of the process is about reconciling what is recorded in the course profile system with what is actually being delivered. Another school with a more developed process also only mapped to units of study.
Excel and web-based data bases were used for mapping, often drawing on existing course profile systems as existing data sets to export into custom made excel spreadsheets.
Data collection was much less common than mapping among the schools included. Three participants reported that there was no data collection processes in place, with a systematic approach to the connection between learning outcomes and assessments still being developed. The three other schools had different approaches to data collection. One participant reported that full marks were used in their data collection; the school was not yet at the point of assuring particular learning outcomes. The other two schools had well developed data collection systems, drawing on part-marks for specific criteria related to the graduate attributes. One school had the learning outcomes embedded in the assessment criteria that were marked by the academics running the subjects, while the other school drew on a mix of staff and external markers.
Closing the Loop
The schools that were engaged in collecting data had some processes in place to review and engage in continuous improvement. The smallest school of engineering included in the study reported a developed process for closing the loop drawing on learning outcome data. Staff are brought together to discuss and review data, minutes of all decisions are recorded and changes are reported at the university level. Each discipline in the school has a faculty advisory committee that engages in this process of review. One of schools primarily engaged in closing the loop with discussion about mapping rather than data, with the data collection serving more to fulfil accreditation requirements. That said some student assessments were collected as examples of standards, with limited processes around feeding this information through to any kind of curriculum change process. The other school that indicated they had data collection in place did not provide much information about change processes other than to say staff were involved at all stages.
The participants talked about a diverse variety of challenges in implementing graduate attributes across engineering programs. Communication was mentioned, specifically communicating across the faculty what is being done and why, and attempting to foster a program view where staff get a sense of program design and how they fit into the whole. Involving students in the discussion could also be challenging, particularly in programs that have a large percentage of off-campus students.
Participants also mentioned challenges in using the technology associated with the process, getting the resources to support the work, and the size and scale of the work. Thinking specifically about mapping, there was a huge amount of work involved in linking the university graduate attributes to the EA competencies.
It was also mentioned that staff may not be well equipped to teach the skills, particularly as schools take on staff with more specific skills in areas of research. Moreover these staff tended to have developed skills in a different way to students, through their work in the industry rather than through coursework. Some staff also felt that a lot of the outcomes were things that could not be assessed and were implicit in the completion of the program.
A participant talked about some concerns about the requirements for TEQSA, that the EA accreditation requirements were quite settled, but that the requirements for TEQSA were in a state of flux. How to get engineering degrees to align with the AQF was also of concern, considering the requirement for the development of generic competencies within the limited time available with students.
The lack of embeddedness was raised as a challenge by a participant that despite recognition that particular attributes were important limited marks were associated with them and students could pass without adequately demonstrating these attributes at a graduate level.
Much of the discussion about challenges concerned efforts to engage staff in the discussion of learning outcomes, mapping, and data collection. Generally the level of staff buy-in was thought to be fair, yet varied across the schools. Some staff were particularly engaged in thinking about teaching and learning and were easy to engage. It was noted that most staff see the value of it, but there is an emphasis on wanting improvements for students, and staff not always seeing the connection between these processes and improvements in the students’ experiences.
The school with the most developed assurance processes included in the sample described buy-in as very strong, supported by strong professional development (internal and external) and the requirement to work around these processes when preparing course specifications and learning outcomes.
One of the other schools with data collection in place talked about some of the critical issues that affected ongoing engagement. Much of the initial stages of the process had not been particularly participatory, the EA guidelines are a requirement and represent the industry’s requirements in new graduates, and the faculty graduate attributes represent an attempt to unify the EA guidelines with the university’s graduate attributes. As much of the learning outcomes were not up for discussion, the participant thought that staff felt sidelined by the part of the course that dealt with what were perceived to be generic skills. Attempts to buy-in really began at the point of mapping and working with staff to think specifically about how attributes were being developed and assessed in their units.
Solutions to Challenges
In addressing some of the challenges discussed above, participants talked about the use of professional development to build competencies and raise awareness of university processes and their purposes. Having course development processes explicitly acknowledge assurance processes was also seen as important in having staff see assurance as part of their business. Proper resourcing of assurance was also seen as important, being able to hire extra staff to build capacity during key times, as well as running workshops with staff. One of the participants thought that because quality assurance had been pushed at a university level, it had been easier to build engagement in the school.
- Philosophy: Being able to show the link between assessment and program level outcomes;
- Motivators: Developing students into professionals, to form a coherent program, accountability to the profession;
- Mapping: Participants mostly mapped to the Australian Nursing and Midwifery Council competencies as these fulfilled many of the university graduate attributes. All schools had developed mapping at the assessment level, with some engaging in a more consultative process;
- Data Collection: None of the schools were in the process of collecting outcome specific data. Clinical placements were used to assure some of the ANMC competency standards.
- Challenges: Getting clinical placements at the appropriate time in the program, the changing demands of nursing graduate skills, how to maintain quality with lower funding, how to manage the different agendas shaping curriculum. While staff engagement was generally reported to be good, change could be problematic for some staff.
- Solutions: Flexibility in delivery, links to industry to keep across demands on staff, proper resourcing of change.
When asked about the philosophy behind their assurance processes, the participants referred to being able to show how students develop the graduate attributes over the program. It was about how assessments and units linked to the program level, and to ensure that there are opportunities to develop GAs systematically distributed across the units. It was also suggested that as well as at the end point, at all points in the program it should be understood what attributes have been achieved.
The identified motivators to systematise the development of graduate attributes were much more varied than the philosophy. One participant suggested that motivation came from the challenges of developing graduates beyond the kind of concrete thinking employed in high school, to becoming independent professionals able to make decisions. The inclusion and development of graduate attributes were vital to ensure students became competent professionals. Another participant said the push for change came from surveys that found students didn’t see the connection between the units to form a coherent program, or the connection between different parts of their learning experience. Other participants talked about accountability to the profession through ensuring the program delivered on what it set out to develop in graduates.
Each of the schools talked about the relationship between the university level graduate attributes and the Australian Nursing and Midwifery Council competencies. These were connected using a mapping process, which the participant indicated was relatively easy as most of the competencies were consistent with the university requirements. The threshold learning outcomes for health were also mentioned as a consideration in developing unit learning outcomes. Primarily the participants indicated their programs were mapped to the competency requirements, which also served for accreditation purposes, but that where the university attributes were not covered by the competencies, these would also be mapped. One participant described this as meeting the accreditation requirements, with the additional inclusion of the attributes that made the graduates distinct from other universities.
All of the participants indicated that GAs were communicated to students, although there was not a clear sense of the level of student awareness of them. Along with course outlines, unit coordinators were expected to make what attributes were covered by the unit clear. One school had a course website designed to help students understand their progression through the program, and another had a booklet about the units and how they connect to the program level outcomes.
All of the schools had developed mapping at the assessment level. One participant described the mapping process as drawing on existing understandings of what was delivered in the units, and then getting the unit coordinators to re-check what outcomes are associated with their units. Another participant described the relationship between the two levels of mapping, that the ANMAC map shows the relationship between competencies and units, while a more detailed curriculum map was used internally to understand the distribution of program and unit level learning outcomes. While the previous two schools consulted unit coordinators in updating maps, one school had an extended consultation exercise around working with staff to define the learning outcomes for each unit and how they fit the competency standards. This was described as a process of starting with the program level outcomes, determining the sequencing of the content of the units, and then doing the more detailed consultation to be clear about how program outcomes are achieved. One school reported undertaking the work of mapping and identifying learning outcomes to be addressed in units within the teaching and learning team.
None of the schools were in the process of collecting outcome specific data to ensure students were achieving the program learning outcomes. The implication was that by passing the program all the competency standards were passed. Partly the lack of data collection was due to the lack of consistent measurement of outcomes across the program. That said other assurance processes were in place though clinical supervision where specific competencies were assessed.
A varied list of challenges was identified by the participants. Access to quality clinical placements at the appropriate point in the program was identified, which was particularly problematic where clinical supervision was used as a kind of capstone. The constantly changing and complex demands on nursing graduates were also identified, with the onus on universities to keep pace with the future demands. A participant discussed the challenge of ensuring students receive a good learning experience in an environment with reduced funding to deliver programs. Also mentioned were providing mixed modes of learning, the level of detail required for mapping, how to include AQF requirements, how to link assessment to learning outcomes in a meaningful way, and how to measure the achievement of learning outcomes. It was also suggested that there are lots of different agendas shaping curriculum and managing these can be difficult.
Staff engagement was discussed as one of the major elements of getting coordination on the distribution of competencies and learning outcomes in programs. There was a sense that there wasn’t significant resistance or negativity, but that taking up people’s time can frustrate staff. Change can be challenging to staff, particularly those that have been in academia for a long time and for staff that lack experience or knowledge in education and curriculum development. Engagement across many campuses can also be challenging.
Some of the solutions employed included a degree of flexibility in the program, in order to enable students to take on clinical placements when they came available. A strong industry linkage was seen as the solution to the challenge of keeping the program relevant to the future needs of the nursing industry. The provision of resources, strong university level leadership, and professional development were also identified as solutions to some of the challenges.