INTRODUCTION
The Principles of Legal Research (PLR) website was first developed in the summer of 2009 as a joint project between the librarians at the Brian Dickson Law Library and the Faculty of Common Law (English) at the University of Ottawa. The Faculty of Law at the University of Ottawa is divided into two sections of Common Law and Civil Law. For a number of years, librarians at the Law Library have been actively involved in the development and delivery of mandatory legal research courses that are required of all students in their first year of law school (both Common Law and Civil Law). In recent years, this task has become increasingly challenging, as student enrolment has increased while faculty numbers have remained static.
The Library and the Faculty therefore approached the University of Ottawa's Centre for University Teaching (CUT), a centre which provides consultation services to the university's faculties regarding teaching practices, to request a grant to help fund the development of a unique mix of blended and community learning strategies in legal research. In the summer of 2009 this grant was approved, and the PLR online modules were conceived as one part of this hybrid learning model.Footnote 2
The goals of the modules are as follows:
-
1. Free up class time for legal research strategies.
-
2. Increase student-instructor interaction.
-
3. Increase student self-assessment opportunities.
-
4. Accelerate theory-to-practice transfer of knowledge.
-
5. Support professors delivering first-year seminars.
-
6. Reinforce students' capacity to undertake independent learning in both new and sub-topic areas required in the legal profession.
In February 2011, the Law Library started working with the University of Ottawa's E-Learning Centre to move the modules out of our learning management system, Blackboard, and into a more accessible, open access version. Since the beginning of 2015, the Law Library has been working to upgrade the visual presence, programming languages and other web elements in the PLR website to conform to the Web Content Accessibility Guidelines (WCAG) in order to reduce barriers for people with disabilities.
RESEARCH METHODOLOGY
This study uses a quantitative method, with a ten question survey (nine multiple choice and one open ended question) to evaluate user experience. A group of 34 enrolled law students from both Common Law and Civil Law sections participated in this user experience study. Both anglophone and francophone participants were represented in this data collection.
RESEACH ETHICAL CONSIDERATION
The University of Ottawa's Research Ethics Board has decided that this study does not need a research ethical approval, as defined by the Canada's Tri-Council Policy Statement (TCPS 2), because this project is an evaluation of a service and quality assurance study.
USER EXPERIENCE STUDY: MULTIDISCIPLINARY APPROACH
This user experience study aims to investigate the real experience of law students with the PLR website of the University of Ottawa's Brian Dickson Law Library. Their feedback will be used for future development planning as well as analysing user behaviour trends. In this section, authors review the academic context applied to this user experience study. Some real examples are given to clarify the actual practices. Finally, the authors will analyse the findings from the questionnaire through a multidisciplinary lens.
COMPUTER-HUMAN INTERACTION: CROSS-CULTURAL TECHNOLOGY DESIGN
Computer-Human Interaction research states that humans relate to interactive products in various different ways. Demographic and cultural elements such as nationality, geography, language, gender, age, sex, education and other factors tend to contribute to these differences in interactions. Marcus and Gould (Reference Marcus, Gould and Jacko2012, 342)1 suggested that culture should be thought of from the initial design process, so that the computer-based products can later be fine tailored to target end-users. For example, people whose first language is written and read from left-to-right (like English and French) need to have a scroll tab on the right hand side when navigating websites. This scroll tab position coordinates with their eye movement; but it is totally opposite for those used to an Arabic script. Those used to Arabic would need to have the scroll tab displayed on the left hand side of the website since Arabic languages are written and read from right-to-left.
Educational institutions should take cross-cultural technology design theory into consideration when designing a computer-based product. Canada is a culturally diverse country and universities and colleges are enrolling more and more international students. The application of cross-cultural technology design theory can therefore ensure that every student will have a positive online learning experience.
Because the University of Ottawa is a bilingual university (English and French), the PLR website presents all information in both official languages. Content has to be accurately translated in both official languages and every webpage has to display a link or button to the other language in the top menu bar, the most obvious location. This menu bar will display on every page and the scroll tab is positioned on the right hand side. This is an example how the learning modules are customized specifically for law students who are studying either in English or French or both languages at the University of Ottawa's Faculty of Law.
Later in this paper, readers will note the second question in the survey, ‘The Principles of Legal Research Website contains accurate and useful information?’, asks participants about the accuracy and usefulness of information and the fourth question, ‘When you were looking for specific information, you found the navigational system (e.g. menu bar, drop-down option, table of contents) in the Principles of Legal Research Website to be well designed?’, asks about the navigational system presented on the site. These questions reflect the cross-cultural technology design concept.
USABILITY
User experience (UX) is emphasised in the practice of interaction design. The UX is a study on how an interactive product is used by targeted end-users in a real situation (Sharp, Rogers and Preece Reference Sharp, Rogers and Preece2007, 15)2. Do the users have a good experience or feeling about the product? Is it difficult to use? Did users use the product for the purpose for which it was originally designed? An interactive product can be a kiosk, website, database, applications, hardware, or software. In this user experience study, the usability is only an element of the user experience being investigated.
Usability is a theory and practice that examines how users can use an interactive product effectively and satisfactorily to achieve specified tasks with existing functionality (Te'eni, Carey, and Zhang Reference Te'eni, Carey and Zhang2007, 4)3. An ideal usability scenario is when a person uses a product for their own needs, while also achieving the product's purpose, with either little or no challenges. For example, online income tax filing applications should be designed and programmed to operate with the least amount of steps possible and should automatically calculate one's tax return amount every time new claim amounts are entered. The task should be quick and easy to accomplish. Usability has been applied to the PLR website in many ways.
One example of this is that the information on the site has been organized by topic. Each topic is separated into its own learning module. On the homepage, users will see icons, short introductory sentences, and automatic expanding and collapsing boxes under the title, ‘View Table of Contents’. When one clicks on this text, a list of sub-headings will appear in the expanding space underneath. This technical design was established to ensure a cleaner look to the homepage and to present a hierarchical schema to the subject areas. It is also expected to help users in searching for information by subject. In the questionnaire, the third question, ‘Overall, the Principles of Legal Research Website presents an interface (e.g. graphics, layout, colours) that stimulates your learning?’ and the ninth question, ‘The Principles of Legal Research Website is easy to use. In other words, you can explore on your own without formal workshops or tutorials in class?’ reflect the usability theory.
Automatic expanding and collapsing content box.
INFORMATION ARCHITECTURE: NAVIGATION SYSTEM
The third focus is on the navigation system, a curious part of all information based interactive products. Morville and Rosenfeld (Reference Morville and Rosenfeld2006, 115)4 state that navigation systems on the web can help users avoid getting lost and being frustrated, especially when they have to search information in a large website. The navigation system focuses on the user's capacity to locate certain pages, paragraphs, text or sections of information they are seeking. Navigation systems on the web can be in various styles such as a search box, menu bar, A-Z index, site map, etc. If users cannot find what they are looking for, or if they have to spend a long time finding information they need, they may then move on to a competitor's product that offers better service. The navigation system of the information architecture foundation of the site is reflected in the fourth question in the questionnaire, ‘When you were looking for specific information, you found the navigational system (e.g. menu bar, drop-down option, table of contents) in the Principles of Legal Research Website to be well designed?’, The PLR website contains many different styles of navigational tools including text links, global and local menu bars, drop-down list, and visualization. Figure 2 below shows the two styles of navigational tools. The global menu bar on the top of the page stays at the same location on each page. The local menu bar on the left side, is customized to match each sub-topic of learning modules. This menu only displays after users have selected a module from the homepage.
Global (top) and local (left) menu bars: Navigational tools.
LIBRARY SCIENCES: PROJECT EVALUATION
For libraries, when a new project has been implemented into library service and operations, an evaluation process or performance measurement should be conducted, after an appropriate timeframe, to measure the effectiveness of the new project. Hernon, Dugan, and Matthews (Reference Hernon, Dugan and Matthews2014, 3)5 state that ‘evaluation is a decision-making tool that is intended to assist library staff in allocating necessary resources to those activities and services’. In some cases, project evaluators may come up with an intervention plan based on evidence they have found. Evaluators will present the final report to the library director or stakeholders, which can certainly help in making future decisions.
The University of Ottawa's Brian Dickson Law Library has conducted this user experience study on the PLR website hoping to find the system's strengths and weaknesses, in order to improve the online learning modules for the benefit of our law students’ learning experiences. Al-Hakim and Xu's case study research (Reference Al-Hakim, Xu and Sarmento2005, 316)6 shows that IT professionals and system users have different points of view when it comes to data quality. IT specialists are often concerned more with technical issues and believe emerging technologies will be able to help produce high-quality data by itself, whereas system users believe some sort of direct human action is more likely to control the quality of information. These different attitudes inspired the authors to create the second question in this user experience study survey, ‘The Principles of Legal Research Website contains accurate and useful information?’. Since the e-learning site was originally built jointly by the systems team (IT specialists) and the content team (law librarians), it was important to the authors to seek the system users’ opinions to determine whether the site contained the desired amount of high-quality information.
When looking at the questions contained the project's questionnaire, the fifth question relates specifically to social media: ‘In your opinion, should the Principles of Legal Research Website introduce Web 2.0 functions like RSS, Blogging, Social Networking (Facebook, Twitter, and Mendeley)?’. The sixth question asks, ‘Would you use a mobile (Smartphone) version of the Principles of Legal Research Website if it was available in the future?’. Other questions relate to user behaviour situations, such as the seventh question; ‘The Principles of Legal Research Website often provides links to external resources. Do you now prefer to read from an electronic version rather than print if both options are available?’ and the eighth question; ‘If you have a question regarding content in the Principles of Legal Research Website, you are more comfortable sending an e-mail to law librarians or instructors rather than meeting with them in person?’. The purpose of these questions is to evaluate targeted user needs before the Law Library invests more time and resources into future online development.
FINDINGS
This section discusses the survey questions (also listed in Figure 3) and the findings. The initial question examines law student demographics based on their enrolment status. (Question 1, ‘Student Demographics: You are currently a student in …?’) There were 28 responses from Common Law students, 3 from Droit Civil students and 3 students from the National Program.Footnote 3 Out of 34 participants, 22 indicated that they were full time students, whereas 12 chose not to answer this question. When asked about nationality, 21 law students indicated they were Canadian or currently held permanent resident status. 13 participants preferred not to specify their legal status or nationality.
Survey questions.
Question 2 probes for information about the accuracy and usefulness of material contained within the PLR Website (Question 2, ‘The Principles of Legal Research Website contains accurate and useful information?’). The majority (52 %) of participants agreed strongly that the e-learning site presents accurate and useful information, while 39 % indicated they somewhat agree. Only 3 % and 6 % of respondents specified somewhat disagree and strongly disagree respectively. Overall, the findings were positive.
The next question (Question 3, ‘Overall, the Principles of Legal Research Website presents an interface (e.g. graphics, layout, colours) that stimulates your learning?’) inquired about design aspects such as graphics, layout, and colours that are present on the site. The authors sought to look closely at those elements to see whether these design aspects were stimulating for the students' learning experience. The user experience study showed that 42 % strongly agree and 36 % somewhat agreed with this statement. Only 6 % indicate they somewhat disagree and 10 % strongly disagree. However, 6 % of all participants chose not applicable as their response, which did not provide a statistical value for this user experience study. There is perhaps a need for this question to provide a more detailed meaning of ‘stimulating interface’ in the questionnaire.
Next we investigated the law students’ experience with the website based on their own real life scenario, while they were looking for specific information. (Question 4, ‘When you were looking for specific information, you found the navigational system (e.g. menu bar, drop-down option, table of contents) in the Principles of Legal Research Website to be well designed?’). Did they find the navigational system to be well designed? Student feedback was positive. The majority agree (42 % for strongly agree, 39 % for somewhat agree) with few disagreeing (10 %, for both somewhat disagree and strongly disagree). Only a few people (ie. 3) skipped this question.
Question five asks student whether the site should be introducing some sort of Web 2.0 functionality, which would allow students to generate content directly in the site. (Question 5, ‘In your opinion, should the Principles of Legal Research Website introduce Web 2.0 functions like RSS, Blogging, Social Networking (Facebook, Twitter, and Mendeley)?’). The addition of this aspect would allow for the creation of discussion forums and greater interaction on the site. The answers we received show that the majority of participants (64 %) think that the addition of Web 2.0 design would not add to the value of the e-learning site. Of the respondents, 23 % supported the Web 2.0 idea, while 13 % chose to select the option of ‘don't know’. The fact that the participants were generally of a tech savvy generation, but did not greatly favour further technical innovation is of great interest to the authors. An in-depth explanation for this is offered in the discussion section of this paper.
The next question tested the user's desire to view the website from their smartphone. Currently, the PLR website only has two versions; the desktop/laptop and tablet version. (Question 6, ‘Would you use a mobile (Smartphone) version of the Principles of Legal Research Website if it was available in the future?’) Responses to this question revealed that 60 % of respondents would like a mobile version of the site created, while 33 % were against the suggestion. Only 7 % did not indicate a preference (ie. don't know).
The PLR website often refers to external print and digital materials. Responses to question 7 (‘The Principles of Legal Research Website often provides links to external resources. Do you now prefer to read from an electronic version rather than print if both options are available?’) showed that 73 % of participants preferred to read from an electronic version rather than a print one, even if both options are available through our library. However, 17 % valued the print collection for their additional reading more than electronic versions. Only 10 % of those responding provided no statement.
Question 8 (‘If you have a question regarding content in the Principles of Legal Research Website, you are more comfortable sending an e-mail to law librarians or instructors rather than meeting with them in person?’) creates a scenario where students have a question about content in the PLR website. Are they more comfortable sending an e-mail to a law librarian or instructor or would they rather attempt a face-to-face enquiry? More than half of the respondents disagreed, (48 % somewhat disagreed and 17 % strongly disagreed), suggesting they preferred face-to-face communication.
The next question focused on how easy the site was to use, assuming users can explore the whole site on their own without formal workshops or tutorials in class. (Question 9, ‘The Principles of Legal Research Website is easy to use? In other words, you can explore on your own without formal workshops or tutorials in class.’) Almost half of the respondents, 48 %, agreed strongly demonstrating a common belief that they can learn to use the site independently. While 21 % specified that they somewhat agreed, 17 % somewhat disagreed, and 10 % strongly disagreed. 4 % provided no view. Overall, half of the participants felt they still need some sort of introductory classes.
Finally, the open-ended question (Question 10, ‘Other comments?’) asked students to provide any comments based on their own learning experience. Only one student provided a comment which read, ‘I thought it was very useful first year’.
DISCUSSION
Results from the survey indicate various strengths and weaknesses of the PLR website. Demographically, survey participants were mostly enrolled as full time students and held Canadian nationality or permanent legal resident status. However, 13 students did not specify their nationalities. It is possible they were concerned about their privacy and the negative consequences of sharing personal information. The unidentified nationality information in this survey would not affect the essential user experience study analysis, However, the authors may have missed a chance to investigate users’ experiences based on cultural differences in this paper, assuming those 13 students were international students.
A strength of the e-learning site is its stimulating interface design, which helps students have a positive experience during their interactions with the site. There are graphics to make pages look attractive. Graphics are a key element in any text-based websites. In this case, it helps liven-up the potentially dry legal-style text. The content is not packed tightly into to a page as in books. Information is clearly divided into different subpages and paragraphs.
Another finding was that the navigational system in the site has been designed and implemented effectively. Users can search the whole site and find specific information quickly, even though there are many pages with both French and English content, which can make for rather a large site. A significant challenge is that it is not easy to manage both languages; we have to make sure that they are coordinated and translated correctly. The development team had to go through complex files and folders to locate a particular page. For users, it can be a similar experience, which is why they depend so much on the navigational system to locate information. This is the reasoning behind the main content page of the PLR website, showing icons and a brief description of each chapter right up front, which acts much like the table of contents in a book. In addition, every content page has a bottom link, allowing users to change current language content between French and English easily with a single click. This option is extremely useful when students want to know legal terminologies in both languages.
Some findings of the survey have shown that the majority of law students do not want Web 2.0 functionality (user-generated information found in social media) to be embedded in the e-learning site. This finding might suggest that students have drawn a clear line between academic sites and sites they interact with for other purposes. As we know, much of Web 2.0 is often used for social purposes. Students may want to separate their private lives and academic learning. The idea that students do not see a benefit to implementing Web 2.0 functionality into an advanced e-learning site, is very interesting to the authors, as it suggests we should look carefully at which technologies we add to the learning modules. Just because a new technology exists, does not necessarily mean it will be appreciated by our users in an e-learning context.
Mobile computing is an emerging trend among library users for accessing information like e-books, e-journals and databases. We may notice that many scholarly publishers are advancing their products and services with smartphone devices. Smartphones are more affordable, lighter weight, and smaller than laptop or desktop computers. However, the small screen of smart phones limits how people can navigate. However, these screens are still good enough to read short texts, listen to music, or watch movies. This could be a reason why 60 % of the law students surveyed supported the idea of a mobile version of the PLR website. The interface design would use the same theme colour and texts, but the navigational layout will need to be very different to ensure it is suitable and compatible with the small screen of a smartphone. The interface design is likely to consist of drop-down content boxes sitting on top each other. It will only show content for the selected content box, one at a time. This design will function requiring minimal clicks or screen slides from users and should be easy to navigate with one or two fingers.
The next discussion item is about the user's preference between reading print and online material. There is no surprise that the majority of participants indicated their personal preference for online resources. Currently, there is so much content available online and new digital content is constantly being produced. In law, some legal primary resources from the federal and provincial governments have been distributed to the public only in an online format, as has been seen with the discontinuance of the printed copy of the Canada Gazette, which was effective as of April, 1, 2014 (Canada Gazette, Reference Gazette2014). Today, the definition of a good law student or lawyer includes having advanced legal database and Internet searching skills, which can help a lawyer win a case or provide accurate legal advice. For law students, online research skills provide an advantage in writing their research papers. The results from this survey question link to the next question on communication behaviour between students and law librarians. Law students still feel strongly that when they have a question, they prefer to ask a law librarian in person, rather than by e-mail. Despite the existence of modern technology, users still value face-to-face interaction. This reminds us that the library as a physical space is still an essential service. Libraries cannot simply discontinue in-person service or eliminate physical collections and move to a completely virtual model. Integrating both online learning and in-class teaching is most likely the best formula for the Law Library.
In the final multiple choice question, asking about users' overall experience with these learning modules, we were looking for whether users were learning and navigating the site independently. The response we received was very positive. Most students are confident that they can use the site without having strong technical skills and legal research knowledge. It was our original plan to have modules for the first year students in law, who are just starting enter into a new field and did not yet know legal content very well. In this user experience study we learned that system design and information organisation are very important. Information has to be structured logically from basic to intermediate and then to the advanced level.
CHALLENGES
This user experience study had some challenges with the number of participants. There was not a large number of participants, as compared with the total number of students in various law programmes at the University of Ottawa. Due to the academic cycle, the authors did not have a lot of time to recruit more participants. We had to conduct this survey in a timely manner, right after the legal research courses ended, to guarantee participants still had a fresh memory of the website. Future research could use this study as a pilot, recruiting a larger cohort of participants.
FUTURE DEVELOPMENT PLANNING
The University of Ottawa's Law Library has a plan to develop the mobile version of these learning modules as a priority task, especially now that we have feedback on the need for a mobile version. Currently, users can only access the content from desktop, laptop, and tablet. The current mobile version makes the text appear too small on smartphone devices. In the near future, users will see some interactive features such as video tutorials. At the time of writing, we were also working on improving our learning module to meet the web accessibility standards, especially ensuring users who use our site with text-to-speech software programs will not have any technical troubles with our content.
CONCLUSION
A user experience study can help to prove how well an online product like the PLR website of the University of Ottawa's Brian Dickson Law Library is actually received. The results and feedback from this user experience study survey can be used to build a strategic plan to improve the product. This user experience study has had some unexpected results, such as students’ reluctance to have social media on the same system. Social media is looked at more for social and personal usage. Another unexpected result is that a majority of students still believe in seeking help from law librarians or instructors by having face-to-face communication, it being more fruitful than only contacting them through e-mail.
Overall, students found the PLR website of the University of Ottawa's Brian Dickson Law Library to be very well produced, designed, and the content well organised. There are some strong elements of the website, including the navigation system, look and feel, and quality of content. Finally, this study helped the authors to understand the trend of mobile computing in the Law Library service, based on our finding that many participants were very enthusiastic to see our modules available on their smartphones.
Appendix 1: Questionnaire in English
-
1. Student Demographics: You are currently a student in …?
Common Law Civil Law National Program (JD/LLL) Full time Part time 1st year 2nd year 3rd year 4th year More Canadian or Permanent resident International -
2. The Principles of Legal Research Website contains accurate and useful information?
Strongly disagree Somewhat disagree Somewhat agree Strongly agree Don't know -
3. Overall, the Principles of Legal Research Website presents an interface (e.g. graphics, layout, colours) that stimulates your learning?
Strongly disagree Somewhat disagree Somewhat agree Strongly agree Don't know -
4. When you were looking for specific information, you found the navigational system (e.g. menu bar, drop-down option, table of contents) in the Principles of Legal Research Website to be well designed?
Strongly disagree Somewhat disagree Somewhat agree Strongly agree Don't know -
5. In your opinion, should the Principles of Legal Research Website introduce Web 2.0 functions like RSS, Blogging, Social Networking (Facebook, Twitter, and Mendeley)?
Yes No Don't know -
6. Would you use a mobile (Smartphone) version of the Principles of Legal Research Website if it was available in the future?
Yes No Don't know -
7. The Principles of Legal Research Website often provides links to external resources. Do you now prefer to read from an electronic version rather than print if both options are available?
Yes No Don't know -
8. If you have a question regarding content in the Principles of Legal Research Website, you are more comfortable sending an e-mail to law librarians or instructors rather than meeting with them in person?
Strongly disagree Somewhat disagree Somewhat agree Strongly agree Don't know -
9. The Principles of Legal Research Website is easy to use? In other words, you can explore on your own without formal workshops or tutorials in class.
Strongly disagree Somewhat disagree Somewhat agree Strongly agree Don't know -
10. Other comments? …………………………………………………….
Common Law Appendix 2: Questionnaire in French
-
1. Démographique des étudiants: Vous êtes actuellement étudiant(e) en … ?
Common Law Droit civil Programme national (J.D./LL.L) À temps plein À temps partiel 1ère année 2ème année 3ème année 4ème année Plus Canadien(ne) ou résident permanent International -
2. Le site Web portant sur les principes de la recherche juridique contient des informations exactes et utiles?
Fortement en désaccord Plutôt en désaccord Plutôt en accord Fortement d'accord Ne sais pas -
3. Dans l'ensemble, le site Web portant sur les principes de la recherche juridique présente une interface (graphiques, mise en page, couleurs) qui stimule votre apprentissage?
Fortement en désaccord Plutôt en désaccord Plutôt en accord Fortement d'accord Ne sais pas -
4. Quand vous avez été à la recherche d'informations spécifiques, vous avez trouvé que le système de navigation (par exemple : barre de menu, liste déroulante, la table des matières) dans le site Web portant sur les principes de la recherche juridique est bien conçu?
Fortement en désaccord Plutôt en désaccord Plutôt en accord Fortement d'accord Ne sais pas -
5. A votre avis, le site Web portant sur les principes de la recherche juridique devrait introduire des fonctions 2.0 comme RSS, blogs, réseaux sociaux (Facebook, Twitter, et Mendeley)?
Oui Non Ne sais pas -
6. Souhaiteriez-vous utiliser une version mobile (Téléphone intelligent) du site Web portant sur les principes de la recherche juridique si elle serait disponible dans le futur?
Oui Non Ne sais pas -
7. Le site Web portant sur les principes de la recherche juridique fournit souvent des liens vers des ressources externes. Avez-vous maintenant préféré lire à partir d'une version électronique plutôt que sur papier si les deux options sont disponibles?
Oui Non Ne sais pas -
8. Si vous avez une question concernant le contenu du site Web portant sur les principes de la recherche juridique, vous êtes plus à l'aise à envoyer un courrier électronique aux bibliothécaires de droit ou aux instructeurs plutôt que de les rencontrer en personne?
Fortement en désaccord Plutôt en désaccord Plutôt en accord Fortement d'accord Ne sais pas -
9. Le site Web portant sur les principes de la recherche juridique est facile à utiliser? En d'autres termes, vous pouvez explorer par vous-mêmes sans ateliers formels ou de séances en classe.
Fortement en désaccord Plutôt en désaccord Plutôt en accord Fortement d'accord Ne sais pas -
10. Autres commentaires? …………………………………………………………………
