«Case study: Using Cloudworks for an Open Literature Review Rebecca Galley, Gráinne Conole and Panagiota Alevizou. The Open University, UK Email: ...»
OU Learning Design Initiative
Case study: Using Cloudworks for an Open Liturature Review: January, 2010
Case study: Using Cloudworks for an Open Literature Review
Rebecca Galley, Gráinne Conole and Panagiota Alevizou.
The Open University, UK Email: R.Galley@open.ac.uk
This case study is one of a series exploring the ongoing use and development of the
Cloudworks site. This case study will focus on an expert elicitation Cloudscape
established to support a literature review project led by the University of Exeter and funded by the Higher Education Academy (HEA): “The positioning of educational technologists in enhancing the student experience.” We will look in detail at the way the site was utilised by the project teams and other participants in the review, and evaluate the site’s effectiveness in supporting this piece of research. We will make recommendations for the development of support resources associated with the site and suggest factors that might impact on the success of similar activities.
1 OU Learning Design Initiative Case study: Using Cloudworks for an Open Liturature Review: January, 2010 Contents
1.3 Focus of the case study
1.4 Project perspectives
2 Overview of interventions and activity
2.2 Lay-out and structure
2.3 Promotion and initial engagement
2.4 Facilitation and leadership
2.5 Technical development
3 Evaluation questions
3.1 Critical success factor 1
3.1.1 Sense of belonging to a Community of Practice
3.1.2 Development of professional knowledge
3.2 Critical success factor 2
4.1 Evaluation methodology
4.2.1 Patterns of activity
4.2.2 Analysis of interactions
4.2.3 Telephone Interview
5 Conclusions and recommendations
5.1 Sense of belonging to a Community
5.2 Development of knowledge and understanding
5.3 Guidance and support
5.4 A review of the barriers and enablers
1.1 Introduction This case study is one of a series exploring the ongoing use and development of the Cloudworks site. Cloudworks is a social networking site for sharing and discussing learning and teaching ideas. The core objects in the site are ‘Clouds’ which can be anything to do with learning and teaching (a discussion, a description of a tool or resource, an example of a teaching intervention). Clouds can be grouped into ‘Cloudscapes’. The evaluation criteria used are two of the critical success factors for Cloudworks, formulated as part of the JISC OULDI project (see Page 5).
Conole (2009a) identified nine types of Clouds and Cloudscapes; one of these she entitled ‘Expert elicitation’ – i.e. those Clouds or Cloudscapes where the author is explicitly asking for contribution to an idea, project or question by experts in the
“A mechanism for gathering views, references and resources from experts in the field around a particular research topic/issue” Conole (2009) There are a number of Clouds and Cloudscapes asking for feedback and answers from expert communities. Some of them are informal and spontaneous (such as the ‘Using Twitter with students’ Cloud http://cloudworks.ac.uk/index.php/cloud/view/2398), which emerged from a conversation initiated in Twitter and then transferred to Cloudworks. Others are more formal in nature; explicitly eliciting information from a targeted user group. This case study will focus on an expert elicitation Cloudscape established to support a literature review project led by the University of Exeter and funded by the Higher Education Academy (HEA): “The positioning of educational technologists in enhancing the student experience” (http://cloudworks.ac.uk/index.php/cloudscape/view/1872).
We will look in detail at the way the site was utilised by both the project teams and other participants in the review, and evaluate the site’s effectiveness in supporting this piece of research. The Cloudworks team were involved in supporting and guiding the construction of the Literature Review Cloudscape, which provided the team with an opportunity to trial a range of support interventions, and collect evidence about
what information and guidance may prove useful for researchers wanting to conduct similar research on the site. This case study aims, therefore, to provide an analysis of the use and activity of the site in relation to the Exeter project to inform the future development of the Cloudworks site, and associated support resources and documents. It may also be of interest to other researchers concerned with developing web 2.0 sites or activities with the aim of supporting the development of professional knowledge and practice.
For interest, the final Exeter Literature Review report is available via the EvidenceNet wiki site at: http://evidencenet.pbworks.com/2009-Synthesis-Projects and it has also been linked to from its e-Learning page at http://evidencenet.pbworks.com/eLearning.
The objective of the Exeter study was to:
“identify in the literature the most effective positioning of educational technologists within institutions in order to maximize their positive impact upon the student experience. In particular, it sought to identify any direct relationship between the work of educational technologists and the enhancement of the student experience.” (Browne and Beetham, 2009, p.4).
The Exeter team had originally planned a desk-based literature review, with some online engagement of the educational technologist community to synthesise the literature. However, it was decided that a recently completed review (Shurville et al.,
2009) had similar scope, and the focus of the project was modified to promote the project as a process that encouraged the HE community as a whole, and educational technology staff in particular, to identify the literature they judged of most value to them.
The methodology chosen by the Exeter team was a variation on the Delphi methodology (Linstone and Turoff, 1975) This methodology commonly uses a panel of experts who are unknown to each other. Questionnaires are used to elicit the opinions of the experts and each expert communicates only with the lead researcher, rather than directly with the other experts. In the first stage of the process a set of
open questions are asked and the results of these are carefully analysed to identify key themes and a more structured questionnaire produced, the results of which are again analysed and the questions refined. Thus, the process leads to a convergence of findings or a consensus. In the case of the Exeter Literature review, the methodology was adapted, using Cloudworks, to promote a divergence of views, and participants were able to communicate with each other. A framework of nine open
questions was used to structure the activity:
Q1: What is the relevance of the student experience to the role of the educational technologist?
Q2: Where should educational technologists be 'positioned'?
Q3: Are educational technologists impacting on changing pedagogies?
Q4: What are the career trajectories and challenges for educational technologists?
Q5: How do educational technologists gain institutional seniority and influence?
Q6: What are the different emphases in the roles of educational technologists?
Q7: To what extent does an educational technologist have to navigate between 'innovative' trends and established practices?
Q8: What is the relevance of educational technologists in relation to educational strategic missions?
Q9: Is the role of the educational technologist relevant to the contribution of the University to the wider knowledge economy?
The Exeter Project Lead closely facilitated the process and discussion on Cloudworks.
He also regularly summarised the discussions at key points, identifying emerging themes and refocusing discussion.
Contributions were made in response to the questions between 28th August and 13th November with most activity occurring in the first four weeks. In addition, an independent parallel conversation took place on the ALT-MEMBERS mailing list in response to the advertisement email. This was an unexpected outcome, and cannot
be included in this evaluation report for ethical reasons, but provides a useful opportunity to begin to compare the activity on the Cloudworks site with an alternative and established professional discussion forum and community. The archived discussion is available for ALT members to view at https://www.jiscmail.ac.uk/cgibin/webadmin?A0=alt-members).
1.3 Focus of the case study It is important to note that the validity and reliability of the Literature Review itself is not in question; the focus of this case study will be firmly framed around factors
relating to critical success factors 1 and 4 of the JISC-OULDI project plan. These are:
Critical success factor 1: A body of evidence sufficient to demonstrate that the Cloudworks website has created real enhancement in the professional knowledge and understanding of participants and increased their sense of belonging to a community of practice.
Critical success factor 4: Resources and guidance that are regarded by users as clearly and effectively supporting them in the intended task/ skills/ knowledge acquisition.
We will explore how far the Cloudworks site has added value to the study, especially in terms of supporting sustained and lively activity, and development of professional knowledge (critical success factor 1). We will further identify which interventions or activity improved levels of focused participation, and which did not. The findings of this study will lead to a series of recommendations about the future development of the site as a whole, and user support resources and materials in particular (critical success factor 4).
This case study will provide a detailed evaluation of the ways in which participants in the Exeter Literature Review interacted with the site and each other. For the
purposes of this study these participants will be categorised as follows:
o Exeter project team (which will include Project Lead and Co-author) o Cloudworks Team (OULDI-JISC Project lead, Cloudworks Developer and OULDI-JISC Project Officer)
o Project associates (which will include members of the wider Open University team and the HEA primary contact) o Other participants (who were not connected with either project)
1.4 Project perspectives Recently we have been developing a framework to enable us to systematically position dialogic transactions and patterns of activity, so that we can more reliably evaluate these in relation to a) a developing community, b) the development of professional knowledge and c) sustained participation.
The meaning and use of the term ‘community’ has tended to shift from location to relationship specific over the past 20 years, in part to account for the notion and
increasing prevalence of online or virtual community:
group, or socially visible boundaries. It does imply participation in an activity system about which participants share understandings concerning what they are doing and what that means in their lives and for their communities”.
These relational communities cannot be considered constant or permanent in the way a geographically defined community might be, and so the process of community formation and growth becomes of greater concern. Rheingold's early definition of
virtual community has been influential in shaping new definitions:
"virtual communities are social aggregations that emerge from the Net when enough people carry on those public discussions long enough, with sufficient human feeling, to form webs of personal relationships in cyberspace"
Wenger (1998) and Brown and Duguid (2001) describe these loosely connected webs as 'networks of practice’ and Wittel (2001) 'network sociality' and differentiate them from communities proper. For Wenger, Communities of Practice are cohesive, and
share historical processes developed from strong ties; networks are more fragile and focus on relational interaction. Our position is that ‘community development’ is a process or lifecycle, rather than an absolute state. This means that we do not believe it will be possible to say that a new community ‘has developed’ or has not, or that there is value in pursuing this. Our interest is in the process of evolution from loosely tied webs or networks to the more cohesive productive groups that can be seen to emerge from transient but repeated and iterative collaborative activity that happens
within, across and between groups from more established Communities of Practice:
“a persistent, sustained [socio-technical] network of individuals who share and develop an overlapping knowledge base, set of beliefs, values, history and experiences focused on a common practice and/ or mutual enterprise”
2 Overview of interventions and activity Early in the project, the Cloudworks team became aware of a tension between the dual objectives of the Exeter project (to generate a list of literature and also to gather opinions, attitudes and experience, related to the theme) and Cloudworks’ functional
purpose as a social networking platform to debate and exchange ideas: