Application Knowledge Assessment Comprehension V

EVALUATION

Evaluation questions are intended to require students to use the implicit cognitive process of making judgments about what they read or heard. We all tend to evaluate many things we encounter. " I like the looks of that new car." "This television set is unreliable." "He is a bore." Evaluation questions are intended to require students to manifest their judgment-making process by (a) clearly identifying a judgment, and (b) clearly and logically supporting the rationale for that judgement based on information they read or heard. Students should be able, is they say, "That book was great," to support that judgement by clearly explaining their definition of great, and then how specific information they read or heard substantiates that judgment. Their evaluation is not adequate if they are able to make a judgment but not support it, even when asked. For instance, they may say, "I don't know why I think the book is great, it just is, that's all." The two types of evaluation, objective and subjective, are based on different criteria; one that is internal to the selection read or information heard, and the other that is external to it. Evaluation-objective is based on internal criteria and evaluation-subjective is based on external criteria.

The Assessment Tasks illustrated here are for reading comprehension of two selections written at a fifth-grade level of difficulty. One selection has a narrative discourse structure (literature) and the other an expository discourse structure (science).

EVALUATION-OBJECTIVE

Evaluation-objective questions involve a criterion that is internal to what students read or heard. It focuses on how well a certain judgment can be rationally and explicit supported by information presented in a selection or lecture. Evaluation-objective is a dispassionate judgment that the information supports or does not support a specific judgment. For instance, these questions ask for an objective evaluation of a selection students read. Based on the facts in the story, was it reasonable for Mr Linnehan to go to California? Why do you say that? The questions elicit both a judgment statement (Was it reasonable?) and a substantiation of that judgment. (Why do you say that?). Students should be able to use information presented in a selection to clearly identify a judgement about whether Mr. Linnehan's action was reasonable, and then to clearly and rationally support whatever judgement they made in terms of information in the selection.

EVALUATION-SUBJECTIVE

Subjective evaluation uses the personal (subjective) feelings of the evaluator. It uses as a criterion some personal bias, belief or preferences of the person. These personal feelings are external to the selection read or information heard. For instance, this question asks for a subjective evaluation: Do you think it fair to the family for Mr. Linnehan to go to California? Why do you say that? The question asks for both a judgement (Do you think it was fair...?) and a substantiation of that judgement (Why do you say that?). If two people have quite different feelings about what constitutes fair on the part of the head of the family, they will give different answers to the question. Each judgment may be equally correct, but their explanation of their judgments will be quite different. The adequacy of their answers will depend on how logically and explicitly they support their judgments.

Application Knowledge Assessment Comprehension IV

This is the fourth post devoted to assessing application knowledge. The reason for the many illustrations of Assessment Tasks for application knowledge is that these tasks are like the Application-Practice Tasks used in the second stage of instructing for application knowledge. Those tasks thus serve a double duty, assessment and instruction.

This post will focus on application-knowledge Assessment Tasks for two categories of comprehension, both involving extrapolation. Assessment Tasks for application knowledge of comprehension generalizations are intended to have students manifest the implicit process of (a) cueing the retrieval of information about a relevant comprehension generalization, and then (b) transferring that information when recognizing novel objects and actions that are examples of the comprehension generalization. The categories of comprehension are determined by the two categories of extrapolation questions asked, consequence and analogy.

EXTRAPOLATION

Extrapolation questions are intended to require students to use the implicit cognitive process of extending ideas beyond the information they read or heard in order to infer consequences and analogies. Extrapolation refers to making predictions about future events by reasoning from past events. For instance, if you were interested in how many miles per gallon you could get at speeds of sixty, seventy or eighty miles per hour, you could make some extrapolations based on information given in the chart, shown below, about mileage gotten at speeds of thirty, forty and fifty miles per hour. Make extrapolations yourself by doing this exercise.

Notice how your predictions about mileage are probably based on an assumption that the relationship existing between speed and mileage found at speeds from thirty to fifty will hold up at speeds of sixty to eighty. If you did not assume that the same relationship will hold up, then you would have predicted differently. Whatever your prediction, you should be able to identify the assumption you did follow about a relationship, and to explain how your answers are logically based on that assumption.

The illustrations of extrapolation Assessment Tasks shown here are for reading comprehension because the questions are for information students read. The same kinds of questions would be for listening comprehension if students heard the information in a lecture or television program. The consequences and analogy questions asked are contained in selection-question units. There are two selections at a fifth-grade level of difficulty; one has a narrative discourse structure (literature) and the other an expository discourse structure (science).

EXTRAPOLATION-CONSEQUENCE

Questions for extrapolation-consequence ask students to predict the consequence of changing one item from a previously interpreted cause-effect relationship.

EXTRAPOLATION-ANALOGY

Analogy refers to a likeness between two things based on a similarity of attributes, circumstances or events. Questions suitable for extrapolation-analogy ask students to use a previously identified cause-effect relationship to infer what an analogous one might be.

Application Knowledge Assessment Comprehension III

This post and the next two posts focus on application-knowledge objectives that are somewhat different than the application knowledge of subject-specific generalizations presented in prior posts. These objectives are for comprehension, either reading or listening. Application-knowledge objectives are for reading comprehension when students read the information presented, and listening comprehension when the information is presented orally, such as in a lecture or television program. Comprehension objectives can be used in almost any subject, ranging from literature to physics.

Assessment Tasks for application knowledge of comprehension objectives are intended to have students manifest the cognitive process of (a) cueing the retrieval of information about a relevant comprehension generalization from long-term memory, and then (b) transferring that information when recognizing novel objects and actions that are examples of the comprehension generalization. Categories of comprehension are commonly determined by the categories of questions asked. For instance, students' reading comprehension ability is measured by asking them questions about a selection they read [1]. An illustrative categorization of objectives is shown below.

1. Categories of questions, such as Interpretation-Conclusion

2. Discourse structure of selection, such as narrative or expository

3. Readability level of selection, such as fifth-grade level or college level

The six categories of questions students are asked are shown in the Comprehension chart, above. The other two variables refer to critical characteristics of the selection students read before being asked questions. All of the illustrations of Assessment Tasks shown here for reading comprehension are based on two written selections. One is a narrative-discourse literature story at a fifth-grade readability level entitled, "The Old Fashioned Ice Cream Freezer." The other is an expository-discourse science article at a fifth-grade readability level entitled, "The Penguin People." The two selections are not shown here. Students read a selection they have not seen before, and then are asked questions about it. They are not allowed to look back at the selection while answering questions. As with all Assessment Tasks for application knowledge, students are asked to justify their answers.

INTERPRETATION QUESTIONS

Interpretation questions are intended to require students to use specific items of information from the selection in order to identify both major and underlying ideas in a selection.

INTERPRETATION-SUMMARIZATION

Questions for interpretation-summarization ask students to identify the major idea presented in the entire selection or in a designated section of it. The questions are intended to require students to use the implicit cognitive process of retrieving from memory relevant items of encoded information and then synthesizing them into one concise statement covering all those items. Synthesizing requires transfer of information just read. Their summaries should usually be no longer than one sentence. They should be asked to justify their summarizations.

INTERPRETATION-CONCLUSION

Questions for interpretation-conclusion ask students to identify ideas in an unstated cause-effect relationship that can be logically inferred from items directly stated in the selection. These questions usually present a cause-effect relationship that is not directly stated in the selection, although both the cause items and effect items are directly stated in the selection. The questions are intended to require students to use the implicit cognitive process of retrieving from long-term memory information about the relevant context in which the cause and effect items occurred in the selection, and then using that information to logically relate the two items as cause and effect. Logically relating the items requires transfer of information just read. Students should be asked to justify their conclusions.

__________________

1. Tuinman, J.I. (1972). Understanding comprehension: Aspects of its measurement. In L.A. Harris, & C.B. Smith (Eds.), Indivildualizing reading instruction: A reader. New York: Rinehart and Winston.

2. National Assessment of Educational Progress. (1972, May). Report 02-R-00. Reading summary.Denver, CO: Education Commission of the States.

3. (For example). Ryan, F.J. (1973). Differentiated effects of levels of questioning on student achievement. Journal of Experimental Education, 41, 63-67.

4. Ruddell, R.B., & Williams, A.C. (1972). A research investigation of a literacy teaching model: Project delta (EPDA Project No. 00526). Washington, DC: U.S. Office of Education.

5. Mueller, D.F. (1972). Teacher questioning practices in reading. Reading World, 12, 136-145.

6. (For example). Wolf, W., King, M.L., & Huck, C.S. (1968). Teaching critical reading to elementary school children. Reading Research Quarterly, 3, 435-438.

Application Knowledge Assessment II

Assessment Tasks for application knowledge are intended to manifest students' implicit cognitive process of (a) cuing retrieval of information about a relevant generalization from long-term memory, and then (b) transferring that information when recognizing novel objects and actions that are examples of the generalization. Objectives for application knowledge usually identify the generalization. In the typical Assessment Task for application knowledge students are given a number of possible novel objects or actions and asked to identify those that are examples of a generalization. Novel objects and actions are ones students have not seen before. The objects and actions must be novel so that transfer is required. If they are not novel then transfer is not required, and the Assessment Task is actually for recall knowledge. As we found in an earlier post application knowledge is frequently confused with recall knowledge. Where possible, students should also be asked to explain why they chose the objects or actions they did, and perhaps also why they did not choose the others. Both explanations provide the teacher a diagnostic understanding of the relative adequacy of each student's application knowledge of a generalization. Teachers' later instruction is based on their understanding of the particular adequacy of each student.

WRITING

Here is an illustration of an Assessment Task for a writing objective. The objects are sentences. The sentences are novel for students because they have not seen them before. The objective identifies the generalization, redundancy in sentences. The task's directions calls it sentences that are too wordy. The Assessment Task shown here is used early in instruction for the objective so it contains short sentences. As students' application knowledge of the generalization increases the teacher will make the sentences progressively longer, eventually providing paragraphs, and more.

An alternative Assessment Task might present each sentence separately, and ask if it is too wordy, and why. Diagnosis of students' performances of Assessment Tasks should be considered a regular part of assessment. Diagnosis of students' application knowledge is guided by the teacher's own mental model of students' implicit process of application knowledge, which involves (a) retrieving from long-term memory information about a relevant generalization, and then (b) transferring that generalization when recognizing novel object or actions that are examples of it. When students' written explanations are not clearly indicative of adequacy or inadequacy in terms of the mental model, the teacher will want to meet individually with them so students can explain their choices orally, and the teacher can ask questions about the generalization and their transferring of it to the sentences selected.

SCIENCE

Here is an Assessment Task for a science objective. The objective identifies the physics generalization, thermodynamics. The objects and actions in this Assessment Task are novel because the teacher has not shown them to students before. Notice that the name of the generalization is not given in the Assessment Task. Students must retrieve information about the generalization from long-term memory on their own. This is somewhat like the study of novices and experts in physics who sorted cards that illustrated different physics generalizations. That study was described in an earlier post.

Teachers' frame of reference in diagnosing students' performances of the Assessment Task is the teachers' own mental model of the implicit cognitive process of application knowledge. The teachers' diagnosis of students' performances uses that mental model in judging the adequacy of students' written explanations and what they say to the teacher in their later individual oral explanations. The teacher will want to understand what physics generalization, if any, each student chose to use, and why he or she chose to use it. The teacher will also want to understand how students transferred that retrieved physics generalization to the actions, including why they did not choose the other two actions. Teachers' instruction for the objective is based on their diagnosis of students' performance of the task.

MATHEMATICS

Here is an illustration of an Assessment Task for a mathematics objective. The objects are intended to be novel for students because the teacher has not shown fractions in rectangles before.

Teachers' diagnosis of students following their completion of the Assessment Task will often have to be done individually because in the average mathematics curriculum fractions are introduced in the primary grades, before students are able to express themselves in writing nearly as well as they are able to orally. Teachers' frame of reference in diagnosing students' performances is based on the teachers' own mental model of students' implicit cognitive process of application knowledge.

Application Knowledge Assessment I

Assessment Tasks, like instructional tasks, are response-demand activities that function as explicit manifestations of students' implicit and unobservable cognitive processes. Assessment Tasks for application-knowledge objectives are designed to cue students' implicit cognitive process of recognizing whether novel objects or actions are examples of the generalizations that are appropriate for the objectives.

Teachers will usually need to engage in diagnosis of students' performance of Assessment Tasks in order to determine students' understanding, which will enable them to adapt their instruction to students' understanding. Diagnosing students' performance of Assessment Tasks is less complex with a lower-order cognitive process like recall knowledge than it is with a higher-order cognitive process like application knowledge. Recall knowledge involves the retrieval of information encoded in long-term memory in much the same way it was originally presented as sensory input. Diagnosis involves simply determining whether the retrieved information is correct For instance, here are two Assessment Tasks for recall-knowledge objectives. The first is knowing about leaves, and it was shown in a more complete form in an earlier posting.

Here is an Assessment Task for the recall-knowledge objective of knowing about thermodynamics.

These two Assessment Tasks are not adequate for application knowledge because they do not ask students to recognize whether novel objects or actions are examples of a generalization. They do not require the transfer of previously encoded information to novel objects or actions, ones not encountered previously. Here is an Assessment Task for the application-knowledge objective of knowing what a summary is. Students are assigned to read an article in their science texts that is entitled, "Treasure on the Ocean Floor." It is three pages long. They have not read it before. When students finish reading the teacher projects this exercise on the screen.

Notice that the novel objects in the Assessment Task consist of examples of three possible summaries for the science article. The task is intended, first, to cue students' retrieval of information about the generalization summary from their long-term memories, and then, second, to have them transfer that information when choosing the statement that best exemplifies an adequate summary of the article. The directions identify the generalization to be retrieved. The sentences are the novel objects to which that generalization summarization should be transferred. Notice that students have a one-in-three chance of simply guessing the right answer. Or they might not use the intended generalization at all, such as picking out the third statement, which is the correct one, because it is the longest. The only way teachers are able to determine if students possess an adequate application knowledge of the generalization summary is to use an active diagnosis consisting of asking them follow-up questions, such as "Why do you think the third one is the best summary? What is wrong with each of the other two summaries?" The key to teachers' accurate diagnosis is asking questions that are based on the teachers' mental model of the implicit cognitive processes involved in students' application knowledge. Teachers use the mental model of application knowledge like a map in locating where each student is relative to the map.

Teachers' diagnostic ability is a significant factor in the effectiveness of their instruction [1]. Some even regard teachers' diagnostic ability as one of four major components of teaching expertise [2]. There is evidence that teachers' diagnostic ability contributes significantly to students' academic achievement [3]. Teachers obviously must possess an adequate knowledge of the subject matter they teach. However, teachers with high levels of subject-matter knowledge often overestimate students' actual understanding [4]. Teachers' overestimation may be caused by a tendency to use their own subject-matter knowledge as the model against which to judge students' performance, thus they think that if they understand then their students must also [5]. Teachers' diagnostic ability will be the greatest when it is based both on their subject-matter knowledge and on their mental model of the implicit cognitive processes students are being asked to use, such as application knowledge.

Incidentally, with reference to the Assessment Task for summarization described above, asking students to read the story and write their own summary, even though it is a less wordy task, stull requires the more complex implicit cognitive process of procedural knowledge. Instruction for the procedural knowledge of summarizing is most effective once students have already learned an application knowledge of summaries. They will have a great deal of difficulty learning to write their own summaries before they have even learned to recognize a good summary when they see it. For instance, think back to the novice-expert physics study described in an earlier posting, where the novices were unable to recognize the physics generalizations exemplified in novel physics problems shown on cards. Because the novices were unable to recognize the examples of the physics generalization shown on the cards, they would not be able to solve those physics problems unless they were told which physics generalization to use with each physics problem. Until they learn application knowledge of those physics generalizations, instruction for a procedural knowledge of them will be ineffective.

____________________

1. (For example). Shepard, L.A. (2001). The role of classroom assessment in teaching and learning. In V. Richardson (Ed.), Handbook of research on teaching (pp. 1066-1101) Washington DC: American Educational Research Association.

2. (For example). Weinert, F.E., Helmke, A., & Schrader, R.W. (1992). Research on the model teacher and the teaching model. In F.K. Oser, A. Dick, & J.L. Patry (Eds.), Effective and responsible teaching--The new synthesis (pp. 249-260). San Francisco: Jossey-Bass.

3. Seidel, T., & Shavelson, R.J. (2007). Teaching effectiveness research in the last decade in disentangling meta-analysis results. Review of Educational Research, 77, 454-449.

4. Nathan, M.J., & Petrosino, A.J. (2003). Expert blind spot among preservice teachers. American Educational Research Journal, 40, 905-928.

5. Nickerson, R.S. (1999). How we know--and sometimes misjudge--what others know: Imputing one's own knowledge to others. Psychological Bulletin, 125, 737-759.

Application Knowledge Objectives

Teachers can categorize an objective as Application Knowledge if they consider that students demonstrate mastery of the objective when they are able to recognize whether novel objects or actions are examples of the generalization that is appropriate for the objective. In order for teachers to consider that a certain objective refers to Application Knowledge, they must be able to identify a generalization for the objective. The generalization is usually identified in the objective. Here are examples of objectives derived from a variety of subjects and curriculum materials that can be categorized as Application Knowledge because they, more or less, identify the generalization that would be learned as Application Knowledge.

POSSIBLE APPLICATION KNOWLEDGE OBJECTIVES

Social Studies

1. Understand that groups of people form governments in order to make decisions that affect all

members of the society. People in a democracy govern themselves directly or through the

persons they elect.

2. Apply their knowledge that the Constitution of the United States is the basis for the way our

government works.

3. Be able to demonstrate that individuals differ in physical traits because of heredity, but that

their likenesses are greater than their differences.

Science

1. Identify examples which show that metabolically active cells are usually small because

materials can move into, through and out of small cells quickly.

2. Understand that the stars and the sun seem to rise in the east, move across the sky from the

east to west, and set in the west because the earth is spinning in an easterly direction.

3. Know that levers make it easier to lift heavy weights because they shift some of the weight to

a fulcrum. Moving the fulcrum to different places on a bar makes the same object harder or

easier to lift.

Mathematics

1. Recognize examples of the fraction one-half.

2. Understand the principle of place value for ones, tens and hundreds.

3. Demonstrate that in a plane, if two lines are perpendicular to the same line, they are parallel.

Language

1. Realize that good writers avoid redundancy by expressing their ideas in ass few words as

possible.

2. Demonstrate that words beginning with the written letter b begin with the same sound in the

words boy, bat and bone.

3. Pick out in selections they have written examples of the major story elements of narrative

text: Setting, Theme, Plot and Resolution.

Most of these objectives would refer to Recall Knowledge if their wordings were treated as target information to be memorized more or less as stated in the objective. For example, the third science objective would refer to Recall Knowledge if this was the Assessment Task for it: Why do levers make it easier to lift heavy weights? The task refers to Recall Knowledge because it asks students to retrieve verbal information from long-term memory pretty much as it was originally encoded. The third science objective would refer to Application Knowledge only if the Assessment Task consists of objects or actions that are examples and non-examples of the generalization fulcrum that students have not seen before, and students are asked to identify which are examples, and then to explain why each is either an example or non-example. Students often fail to learn desired Application Knowledge because their teachers confuse it with Recall Knowledge. Consequently, students are able to recite definitions of generalizations but are unable to recognize novel objects and actions that are examples of those generalizations.

The next five posts deal with Assessment Tasks for Application Knowledge. These tasks are very important because they provide the teacher a frame of reference for instruction, and a basis for diagnosing students' moment-by-moment learning of Application Knowledge. In addition, Assessment Tasks for Application Knowledge look like the Application-Practice Tasks that are used in the second step of instructing for Application Knowledge. This is analogous to the way Assessment Tasks for Recall Knowledge look like the Recall-Practice Tasks that are used in the second step of instructing for Recall Knowledge.