Archive

Posts Tagged ‘evaluation’

New Learning Spaces: Method of Evaluation

December 17, 2012 Leave a comment
Observation in evaluating learning spaces

Observation in evaluating learning spaces

This is a continuation of ideas being developed around the evaluation of learning spaces at City University London.

At City University London there is a need to better understand the effectiveness of new learning spaces that are being created. City University London is currently engaged in the redevelopment of its estate. A major part of this is the re-conceptualization of the Learning Spaces. The Learning Development Centre (LDC) is working closely with City’s Property and Facilities to ensure that pedagogical principles are considered in the redesigning of City University’s learning spaces.

Understanding the effectiveness of new learning spaces is crucial for two reasons: to evaluate the effectiveness of a space newly created and to prepare better for future learning spaces design and construction. As such, this formal evaluation will be a ‘post-occupancy evaluation’ of the space. It is this stage of the evaluation cycle that presents the greatest challenges in aligning the evaluation method with the rational and practical outcomes that drove design intent. However, it is also crucial as the formative model for a full design and evaluation process, and as a source of data for new informal and collaborative spaces (Lee, 2009 in Radcliffe et al, 2009).

At a broad level, it is important for educational developers and education researchers to better understand how lecturers and students relate to the new built environment and what this means for the exchange of knowledge. To this purpose, it is understood that efforts to develop more effective learning spaces need to be informed by the extensive research into environmental behaviour and psychology (Jamieson, 2007).

To this end, I am found that the observation method is a popular tool in evaluating new learning spaces (Radcliffe et al, 2009).  Observations are builds on the principle that for research into the use and effectiveness of the new learning spaces, it is best to observe what actually happens in the natural setting (Descomber, 2003) rather than to ask for thoughts retrospectively.

In line with the epistemology of participating observation, this study would enable the research team to participate in natural learning situations, enabling better understanding of the learning processes involved in the new spaces. The observations will take place in the natural learning spaces as the research team is interested in the effects of the environment on learning as it happens, rather than they happen under artificially created conditions. This allows the research team to record information as it happens and record critical incidents as they occur (Creswell, 2009).

The observation method has a number of characteristics which cannot be found in other education research methods and which are better suited in understanding the new learning spaces. These include:

  1. It directly records what the user does in the space, as distinct from what they say they do are their perception of the room.
  2. Observation is well matched with other research methods being applied in understanding the SLE and learning spaces. As it is more about the behaviour it complements well other research methods that rely mainly on sharing thoughts.
  3. When combined with contextual information, which will be the case here, observation can give significant insight on the effects of the learning space on behaviour.

We are continually working on developing our research methods for evaluating the learning spaces. Please do share your thoughts and experiences of evaluating learning spaces.

Nominal Group Technique

March 1, 2012 3 comments

I attended an ELESIG webinar on Student Engagement Methods: a focus on the Nominal Group Technique (NGT) on 29 February 2012. The session focused on how this technique could be used to elicit feedback from students (or staff) on their experiences of learning technology or for them to give feedback on their learning. This technique encourages active engagement and the webinar explored how the technique works in a face-to-face environment.

Background

NGT is a structured face-to-face technique which was originally developed in the 1970s as a technique for making decisions in large groups. The term Nominal Group arises from the fact that the activity is a group only in name as it relies on individual contributions in a group environment. (O’ Neil and Jackson 1983, cited in Varga-Atkins et al. 2011)

NGT Structure

In an NGT activity, the facilitator introduces the structure and purpose of the activity to participants (staff or students). The question that participants need to answer must be focused and easily understood. (Varga-Atkins et al., 2011) The recommended number of participants for an NGT activity is 8-10 participants, but if you have multiple facilitators you can run a number of groups on the same topic.

Varga-Atkins et al. (2011) have outlined the stages of the activity as follows:

Post-its

Kakachu (2008) Post-its CC BY-NC-SA 2.0

Stage 1: Individual responses to a single question are written on a series of post-its in silence. The facilitator leads a round robin of the group as each participants reads out their response and the facilitator places the post-its on a whiteboard and numbers them.  If NGT participants are providing multiple responses then the round robin continues until all response are read out.

Stage 2: Clarification can be asked for by group participants at this stage. Responses are consolidated into themes by the group.

Stage 3: Each individual ranks their top 5 responses. The facilitator then calculates the ranking results and shares with the group.

Mock NGT

To demonstrate the technique in action the webinar facilitators got all participants to respond to a question using the online whiteboard. This technique didn’t quite work in an online environment as you could see others responses while you were formulating your own ideas. It also proved difficult to complete the ranking stage on a whiteboard.

Benefits

The webinar participants outlined the following benefits for NGT participants:

  • Everyone has an opportunity to input a response
  • The technique provides for a wide breadth of responses
  • The technique can generate a large volume of data in a short amount of time
  • The structure of the technique helps to moderate group dynamics
  • The technique can promote a sense of ownership for participants
  • Students have said that they enjoy this method of providing feedback as it avoids survey fatigue

The webinar participants outlined the following benefits for facilitators:

  • Ensures quiet participants take part
  • Provides for an immediate response

Challenges

The webinar participants outlined the following challenges for NGT participants:

  • Consensus can be interpreted in a variety of ways
  • Grouping the ideas can be subjective
  • Participants might be too shy to ask for clarification
  • The process needs time to be explained. The webinar facilitators suggested sending a  reminder email about process and task to participants ahead of the activity
  • Participants may feel under pressure to perform in public
  • The activity is fast paced and participants have to stay alert

The final report generated from the NGT activity can be circulated and then sent to participants for further clarification.

When is the NGT useful?

Varga-Atkins et al. (2011 p. 5) have summarised when the technique is useful in different research and evaluation contexts:

Research Purpose For evaluation and decision making
Topic When you have one topic to explore
Likely research questions What changes might you make to your programme/curriculum?
Participants Participants with different power relations within the same group.

I’d be interested in finding out from your comments if you think this might be a useful technique to elicit student feedback on their experience of using learning technology and/or to get feedback on their learning.

References:

Varga-Atkins, T., with contributions from Bunyan, N; McIsaac, J; Fewtrell J. (2011) The Nominal Group Technique: a practical guide for facilitators. Written for the ELESIG Small Grants Scheme. Liverpool: University of Liverpool. October. Version 1.0 [online] Available from: http://www.slideshare.net/tundeva/the-nominal-group-technique-a-practical-guide-for-facilitators (Accessed: 29.2.12)

%d bloggers like this: