Week 7

Web Evaluation


The process of web evaluation insluding formative and sumative assesment

Web Formative and Summative Assessment

There is a limited research base on the use of online tools to deliver formative and summative assessments. However, the research base on traditional testing formats is relevant and provides insight into the experiences of learners. To frame the theoretical framework for this study, we present the literature demonstrating that (a) formative assess­ments can serve as effective test preparation events, (b) providing multiple formative assessments can influence learners’ test perceptions, and (c) migrating traditional multiple-choice tests to an online testing protocol provides no universal performance or perception variances.

Impact of Formative Assessment on Learning and Achievement

The decision to use formative assessment in instruction is typically motivated by an attempt to provide the instructor with an accurate esti­mation of student ability at a particular point in the course, or to provide the students with an assessment task similar in nature to the summative test (Buchanan, 1998). This allows the student to identify strengths and weaknesses and to better prepare for the “real” exam. One of the great advantages of online test programs is the ability to deliver practice tests that serve as formative assessment tools for the students. Practice tests have been shown to increase students’ final outcome performance by roughly twelve percent (Bocij & Greasley, 1999; also see Carrier & Pashler, 1992; Dempster, 1997; Glover, 1989; McDaniel, Kowitz, & Dunay, 1989). Delivering practice tests online may provide an additional benefit to the student by allowing her or him to complete the test conveniently without the environmental distractions that are common during in-class practice tests.

Because different conceptualizations for “practice test” or “practice quiz” are common, there are dramatically different educational, cognitive, and theoretical implications when employing the different strategies of practice testing; thus, operationalization is key. In this discussion, unless otherwise noted, practice quizzes and formative assess­ments refer to assessment tools that are completed by students prior to a summative (graded) assessment. These practice tests are similar to summative assessments in format and difficulty level, but do not impact the students’ course grade and are comprised of a different set of items.

The utility of formative assessment is partly reliant upon the manner through which the feedback is provided to the learner. The most desirable feedback approach appears to be immediate post-performance reporting,

which provides feedback directly after the entire quiz or test has been completed (King & Behnke, 1999). This method takes advantage of a primary benefit of computer-assisted assessment by supplying timely feedback (Clariana, Ross, & Morrison, 1991; Jongekrijg & Russell, 1999), while avoiding the problem of inducing anxiety or distraction that can arise when providing performance indicators directly after each item (Wise, Plake, Eastman, Boettcher, & Luken, 1986; Wise, Plake, Pozehl, Barnes, & Lukin, 1989). The anxiety induced in item-by-item feedback has been shown to hamper performance through motivational processes such as learned helplessness or externalized attributions of control over perfor­mance (Boggiano & Ruble, 1986).

Formative Assessment and Students’ Perceptions of Tests

The benefits of repeated formative assessment for students are likely to rest in their perceptions of test preparedness for the summative measure. Bandura (1986) proposed repeated exposure to successful testing experiences for students with high anxiety would promote self-efficacy for later tests. The use of formative assessments (where no evaluative pressure is imposed) as practice for tests is likely to increase the probability that students will have a positive experience in the testing event with respect to anxiety. In these formative assessment experiences, perceived threat, self-awareness, cognitive test anxiety, and emotion­ality should all be lower than in standard summative assessment sessions (Kurosawa & Harackiewicz, 1995; Schwarzer & Jerusalem, 1992). With the suppression of these affective detractors, the student is more likely to be able to benefit from self-regulatory processes in the practice testing session, leading to higher performance, growth, and subsequent success (Bandura, 1986; Schutz & Davis, 2000).

Online Summative Assessment

Summative assessment in an online environment differs in form and function from the formative assessment process. Not only are the summative assessments graded, but the methods through which students access and respond to the tests usually differ. The summative assessment process requires high levels of control and security in the testing process to ensure reliability and validity in scores, attention to technical problems that may arise during the testing session, and assurance that the online nature of the testing process itself has no impact on actual performance. An additional concern that is often raised by instructors considering online summative assessment is that online testing will induce heightened levels of anxiety over the test, leading to performance levels that underestimate true ability.

The advantages for providing course tests online can include flexibility in delivering tests to students and efficiency in scoring, depending upon the method of delivery chosen by the instructor. With the online delivery of tests, students are not necessarily bound by the traditional artificial academic scheduling constraints. Specifically, (a) they can complete exams at different times of the day to fit their convenience; (b) they can poten­tially complete the tests in different locations if the test is not a required “closed-book” exam; and (c) unless there is an explicit reason for a time limit, students can take as long as needed to complete the exam. In a similar line, an additional benefit that can be gained through online summative assessment is that additional class time may be gained in traditional on-campus courses. That is, rather than taking a class period to have the students complete the course exam, the instructor can use the class period for instruction.

In perhaps the most complete examination of online summative assess­ment to date, Bocij & Greasley (1999) reported that students claimed online testing was superior because they were less distracted with the process of handwriting their responses, which helped them maintain focus on the test items and were less panicked. The lower levels of panic were impacted in part by the fact that online tests took less time to complete. Students in Bocij & Greasley’s (1999) work reported the tests were fair, unbiased, and “less threatening than conventional examinations” (p. 14). Finally, the authors reported that performance gains were noted in the online testing conditions, but these effects were not present for the high ability students who appeared to be unaffected by test delivery format.

Present Investigation

As mentioned earlier, this investigation addressed two research questions. The first was a comparison of the effect of delivering course exams online versus in class on paper. This portion of the study involved examining the affective experiences of one instructor’s students. The students were enrolled in the same course, separated by one year. The only evaluative difference existing between the two courses was the method of delivering the course exams. For the first group of students, all tests were delivered in class on paper. For the second group, all tests were delivered online in a computer-based testing laboratory staffed by testing proctors who ensured the security of the testing process and corrected any technical issues that arose. Students’ levels of cognitive test anxiety, emotionality, and perceived threat of tests were compared to determine if there were dif­ferential perceptions of tests for students experiencing the two alternate methods of test delivery. These data were intended to examine the extent to which online testing leads to heightened levels of fear, anxiety, or worry over tests. The hypothesis underlying this question was that the method of presentation would have no meaningful detrimental impact for the students in any of these variables.

The second part of the study examined the relationships among the use of online formative assessments, student performance, and test perceptions. For both groups of students, online practice tests were made available as a test preparation option for only the third exam. It was expected that the students using online formative assessment tests (as practice) would have higher rates of performance on subsequent summa­tive assessment measures. Due to the differential patterns of behavior and performance traditionally noted in students with test anxiety based in part on study strategies (Naveh-Benjamin, McKeachie, & Lin, 1987), no a priori predictions regarding the relationship between online formative assessment and test perceptions were reasonable.

Analyzing existing web evaluation forms

Web analytics is the measurement, collection, analysis and reporting of internet data for purposes of understanding and optimizing web usage

There are two categories of web analytics; off-site and on-site web analytics.

Off-site web analytics refers to web measurement and analysis irrespective of whether you own or maintain a website. It includes the measurement of a website's potential audience (opportunity), share of voice (visibility), and buzz (comments) that is happening on the Internet as a whole.

On-site web analytics measure a visitor's journey once on your website. This includes its drivers and conversions; for example, which landing pages encourage people to make a purchase. On-site web analytics measures the performance of your website in a commercial context. This data is typically compared against key performance indicators for performance, and used to improve a web site or marketing campaign's audience response.

Historically, web analytics has referred to on-site visitor measurement. However in recent years this has blurred, mainly because vendors are producing tools that span both categories.

Online form

Online form tracking and analysis is somewhat nascent to say the least. A quick search on Google yields little on the subject beyond the odd discussion group. However, UK-based solutions provider Speed Trap has recently included the functionality in its Prophet E-Business Intelligence Suite. Prescience or wishful thinking? We think prescience: the form tracking industry seems here to stay. After all, it’s another way that the web can measure marketing initiatives in a way that offline programs cannot.

This article introduces the online form tracking market at its earliest stage, and identifies the needs and measures for form tracking as a subset of web analytics.

Form filling plays a significant role for most websites. Often a form is an important market research tool or a key lead generator for further offline and online marketing initiatives. They help you learn about user needs and better shape products or services for them.

However, they are often perceived as the bane of web user’s existence, getting in the way and obstructing them from the content they want.

Given this dichotomy – and the ongoing necessity of online forms – the need to optimise online form usage and fulfillment has never been greater.

What to measure

The items to track a form can be group into four areas: Form Fulfillment, Form Failure, Form Abandonment and Form View Only. The relevant factors to measure vary, but generally include questions that need to be answered as to why a form is completed or not.

Form Fulfillment - The form is completed and submitted with no problems. One or two attempts may have occurred, but you collect the minimum information required. Relevant factors include:

• How long did it take to complete?
• How much information over the minimum was collected?
• Did the form fill sequence work?
• Where was the information entered first?
• What optional fields were left vacant?

Form Failure - The form is completed or partially finished but no acknowledgement from the site that the form was submitted, likely rendering the data lost. This can occur with people who are less web savvy, or if the ’submit’ button is harder to find. Relevant factors include:

• What information/fields were entered?
• What sequence was used?
• Not entered
• Design/layout
• Browser compatibility
• Use of autofill

Form Abandonment - The form is attempted but abandoned. No attempt to submit. Relevant factors include:

• What information was entered?
• What sequence is used?
• Form length
• Labeling
• Design layout
• Incentives
• Use of autofill
• Distractions
• Security Concerns

Form View Only - A user has merely viewed the form and clicked away. Relevant factors include:

• What sequence is used?
• Form length
• Labeling and/or instructional copy
• Design layout
• Incentives
• Distractions
• Security Concerns

Build or buy

Preliminary research revealed that the preference in the early days (a few years ago) for form tracking was to purpose-build an application. This was mainly because few solutions existed in the marketplace. Today, purchased applications are winning favour and those with an understanding of form tracking’s importance are leaning that way.

Mike Keyes, of Minneapolis-based web marketing consultancy Ciceron, said he built a tracking application several years ago, but plans to replace it with a purchased solution. “We realised there was no way for us to cost-effectively keep up with the innovation that is occurring in the industry right now,” said Mr Keyes.

Jim Cuene ,who works for a large financial services company, said has been looking for a form tracking tool for some time. He had previously used Atlas Net Conversions usability service but felt it was cost prohibitive. “It’s ( Atlas) a great snapshot of how forms are used, but it is too expensive,” said Mr. Cuene.

Consequently the buy option in relation to form tracking is becoming more realisable as the need for overall ‘out of the box’ web analytics solutions grow. Providers such as Web Side Story (HBX On Demand) and Speed Trap are including form tracking as part of a fully featured web analytics package.

Thus as the cost comes down and the functionality increases, the buy option is fast becoming a no-brainer.

Producing and utilizing of web evaluation form

Checklist for Evaluating Web Resources
Is the Web a good research tool? This question is dependent on the researcher's objective. As in traditional print resources one must use a method of critical analysis to determine its value. Here is a checklist for evaluating web resources to help in that determination.

Authority:
-Is the information reliable?
-Check the author's credentials and affiliation. Is the author an expert in the field?
-Does the resource have a reputable organization or expert behind it?
-Are the sources of information stated? Can you verify the information?
-Can the author be contacted for clarification?
-Check for organizational or author biases.

Scope:
-Is the material at this site useful, unique, accurate or is it derivative, repetitious, or doubtful?
-Is the information available in other formats?
-Is the purpose of the resource clearly stated? Does it fulfill its purpose?
-What items are included in the resource? What subject area, time period, formats or types of material are covered?
-Is the information factual or opinion?
-Does the site contain original information or simply links?
-How frequently is the resource updated?
-Does the site have clear and obvious pointers to new content?

Format and Presentation:
-Is the information easy to get to? How many links does it take to get to something useful?
-What is the quality of the graphical images? Do these images enhance the resource or distract from the content?
-Is the target audience or intended users clearly indicated?
-Is the arrangement of links uncluttered?
-Does the site have its own search engine?
-Is the site easily browsable or searchable?

Cost and Accessibility:
-Is the site available on a consistent basis?
-Is response time fast?
-Does the site have a text-based alternative?
-How many links lead to a dead-end?
-Is this a fee-based site? Can non-members still have access to part of the site?
-Must you register a name and password before using the site?

Other Tips:
-Check the header and footer information to determine the author and source.
-In the URL, a tilde ~ usually indicated a personal web directory rather than being part of the organization's official web site.
-In order to verify an author's credentials, you may need to consult -some printed sources such as Who's Who in America or the Biography Index.
-Check and compare the web site to others which are both similar and different.

Exposure to the concept of reliability and validity of web evaluation form

Reliability and Validity for Quality
Information is everywhere on the Internet, existing in large quantities and continuously being created and revised. This information exists in a large variety of kinds (facts, opinions, stories, interpretations, statistics)and is created for many purposes (to inform, to persuade, to sell, to present a viewpoint, and to create or change an attitude or belief). For each of these various kinds and purposes, information exists on many levels of quality or reliability. It ranges from very good to very bad and includes every shade in between.

Getting Started: Screening Information
Pre-evaluation

The first stage of evaluating your sources takes place before you do any searching. Take a minute to ask yourself what exactly you are looking for. Do you want facts, opinions (authoritative or just anyone's), reasoned arguments, statistics, narratives, eyewitness reports, descriptions? Is the purpose of your research to get new ideas, to find either factual or reasoned support for a position, to survey opinion, or something else? Once you decide on this, you will be able to screen sources much more quickly by testing them against your research goal. If, for example, you are writing a research paper, and if you are looking for both facts and well-argued opinions to support or challenge a position, you will know which sources can be quickly passed by and which deserve a second look, simply by asking whether each source appears to offer facts and well-argued opinions, or just unsupported claims.

Select Sources Likely to be Reliable
Becoming proficient at selecting sources will require experience, of course, but even a beginning researcher can take a few minutes to ask, "What source or what kind of source would be the most credible for providing information in this particular case?" Which sources are likely to be fair, objective, lacking hidden motives, showing quality control? It is important to keep these considerations in mind, so that you will not simply take the opinion of the first source or two you can locate. By thinking about these issues while searching, you will be able to identify suspicious or questionable sources more readily. With so many sources to choose from in a typical search, there is no reason to settle for unreliable material.