Application Knowledge Assessment Comprehension V

EVALUATION

Evaluation questions are intended to require students to use the implicit cognitive process of making judgments about what they read or heard. We all tend to evaluate many things we encounter. " I like the looks of that new car." "This television set is unreliable." "He is a bore." Evaluation questions are intended to require students to manifest their judgment-making process by (a) clearly identifying a judgment, and (b) clearly and logically supporting the rationale for that judgement based on information they read or heard. Students should be able, is they say, "That book was great," to support that judgement by clearly explaining their definition of great, and then how specific information they read or heard substantiates that judgment. Their evaluation is not adequate if they are able to make a judgment but not support it, even when asked. For instance, they may say, "I don't know why I think the book is great, it just is, that's all." The two types of evaluation, objective and subjective, are based on different criteria; one that is internal to the selection read or information heard, and the other that is external to it. Evaluation-objective is based on internal criteria and evaluation-subjective is based on external criteria.

The Assessment Tasks illustrated here are for reading comprehension of two selections written at a fifth-grade level of difficulty. One selection has a narrative discourse structure (literature) and the other an expository discourse structure (science).

EVALUATION-OBJECTIVE

Evaluation-objective questions involve a criterion that is internal to what students read or heard. It focuses on how well a certain judgment can be rationally and explicit supported by information presented in a selection or lecture. Evaluation-objective is a dispassionate judgment that the information supports or does not support a specific judgment. For instance, these questions ask for an objective evaluation of a selection students read. Based on the facts in the story, was it reasonable for Mr Linnehan to go to California? Why do you say that? The questions elicit both a judgment statement (Was it reasonable?) and a substantiation of that judgment. (Why do you say that?). Students should be able to use information presented in a selection to clearly identify a judgement about whether Mr. Linnehan's action was reasonable, and then to clearly and rationally support whatever judgement they made in terms of information in the selection.

EVALUATION-SUBJECTIVE

Subjective evaluation uses the personal (subjective) feelings of the evaluator. It uses as a criterion some personal bias, belief or preferences of the person. These personal feelings are external to the selection read or information heard. For instance, this question asks for a subjective evaluation: Do you think it fair to the family for Mr. Linnehan to go to California? Why do you say that? The question asks for both a judgement (Do you think it was fair...?) and a substantiation of that judgement (Why do you say that?). If two people have quite different feelings about what constitutes fair on the part of the head of the family, they will give different answers to the question. Each judgment may be equally correct, but their explanation of their judgments will be quite different. The adequacy of their answers will depend on how logically and explicitly they support their judgments.

Application Knowledge Assessment Comprehension IV

This is the fourth post devoted to assessing application knowledge. The reason for the many illustrations of Assessment Tasks for application knowledge is that these tasks are like the Application-Practice Tasks used in the second stage of instructing for application knowledge. Those tasks thus serve a double duty, assessment and instruction.

This post will focus on application-knowledge Assessment Tasks for two categories of comprehension, both involving extrapolation. Assessment Tasks for application knowledge of comprehension generalizations are intended to have students manifest the implicit process of (a) cueing the retrieval of information about a relevant comprehension generalization, and then (b) transferring that information when recognizing novel objects and actions that are examples of the comprehension generalization. The categories of comprehension are determined by the two categories of extrapolation questions asked, consequence and analogy.

EXTRAPOLATION

Extrapolation questions are intended to require students to use the implicit cognitive process of extending ideas beyond the information they read or heard in order to infer consequences and analogies. Extrapolation refers to making predictions about future events by reasoning from past events. For instance, if you were interested in how many miles per gallon you could get at speeds of sixty, seventy or eighty miles per hour, you could make some extrapolations based on information given in the chart, shown below, about mileage gotten at speeds of thirty, forty and fifty miles per hour. Make extrapolations yourself by doing this exercise.

Notice how your predictions about mileage are probably based on an assumption that the relationship existing between speed and mileage found at speeds from thirty to fifty will hold up at speeds of sixty to eighty. If you did not assume that the same relationship will hold up, then you would have predicted differently. Whatever your prediction, you should be able to identify the assumption you did follow about a relationship, and to explain how your answers are logically based on that assumption.

The illustrations of extrapolation Assessment Tasks shown here are for reading comprehension because the questions are for information students read. The same kinds of questions would be for listening comprehension if students heard the information in a lecture or television program. The consequences and analogy questions asked are contained in selection-question units. There are two selections at a fifth-grade level of difficulty; one has a narrative discourse structure (literature) and the other an expository discourse structure (science).

EXTRAPOLATION-CONSEQUENCE

Questions for extrapolation-consequence ask students to predict the consequence of changing one item from a previously interpreted cause-effect relationship.

EXTRAPOLATION-ANALOGY

Analogy refers to a likeness between two things based on a similarity of attributes, circumstances or events. Questions suitable for extrapolation-analogy ask students to use a previously identified cause-effect relationship to infer what an analogous one might be.

Application Knowledge Assessment Comprehension III

This post and the next two posts focus on application-knowledge objectives that are somewhat different than the application knowledge of subject-specific generalizations presented in prior posts. These objectives are for comprehension, either reading or listening. Application-knowledge objectives are for reading comprehension when students read the information presented, and listening comprehension when the information is presented orally, such as in a lecture or television program. Comprehension objectives can be used in almost any subject, ranging from literature to physics.

Assessment Tasks for application knowledge of comprehension objectives are intended to have students manifest the cognitive process of (a) cueing the retrieval of information about a relevant comprehension generalization from long-term memory, and then (b) transferring that information when recognizing novel objects and actions that are examples of the comprehension generalization. Categories of comprehension are commonly determined by the categories of questions asked. For instance, students' reading comprehension ability is measured by asking them questions about a selection they read [1]. An illustrative categorization of objectives is shown below.

1. Categories of questions, such as Interpretation-Conclusion

2. Discourse structure of selection, such as narrative or expository

3. Readability level of selection, such as fifth-grade level or college level

The six categories of questions students are asked are shown in the Comprehension chart, above. The other two variables refer to critical characteristics of the selection students read before being asked questions. All of the illustrations of Assessment Tasks shown here for reading comprehension are based on two written selections. One is a narrative-discourse literature story at a fifth-grade readability level entitled, "The Old Fashioned Ice Cream Freezer." The other is an expository-discourse science article at a fifth-grade readability level entitled, "The Penguin People." The two selections are not shown here. Students read a selection they have not seen before, and then are asked questions about it. They are not allowed to look back at the selection while answering questions. As with all Assessment Tasks for application knowledge, students are asked to justify their answers.

INTERPRETATION QUESTIONS

Interpretation questions are intended to require students to use specific items of information from the selection in order to identify both major and underlying ideas in a selection.

INTERPRETATION-SUMMARIZATION

Questions for interpretation-summarization ask students to identify the major idea presented in the entire selection or in a designated section of it. The questions are intended to require students to use the implicit cognitive process of retrieving from memory relevant items of encoded information and then synthesizing them into one concise statement covering all those items. Synthesizing requires transfer of information just read. Their summaries should usually be no longer than one sentence. They should be asked to justify their summarizations.

INTERPRETATION-CONCLUSION

Questions for interpretation-conclusion ask students to identify ideas in an unstated cause-effect relationship that can be logically inferred from items directly stated in the selection. These questions usually present a cause-effect relationship that is not directly stated in the selection, although both the cause items and effect items are directly stated in the selection. The questions are intended to require students to use the implicit cognitive process of retrieving from long-term memory information about the relevant context in which the cause and effect items occurred in the selection, and then using that information to logically relate the two items as cause and effect. Logically relating the items requires transfer of information just read. Students should be asked to justify their conclusions.

__________________

1. Tuinman, J.I. (1972). Understanding comprehension: Aspects of its measurement. In L.A. Harris, & C.B. Smith (Eds.), Indivildualizing reading instruction: A reader. New York: Rinehart and Winston.

2. National Assessment of Educational Progress. (1972, May). Report 02-R-00. Reading summary.Denver, CO: Education Commission of the States.

3. (For example). Ryan, F.J. (1973). Differentiated effects of levels of questioning on student achievement. Journal of Experimental Education, 41, 63-67.

4. Ruddell, R.B., & Williams, A.C. (1972). A research investigation of a literacy teaching model: Project delta (EPDA Project No. 00526). Washington, DC: U.S. Office of Education.

5. Mueller, D.F. (1972). Teacher questioning practices in reading. Reading World, 12, 136-145.

6. (For example). Wolf, W., King, M.L., & Huck, C.S. (1968). Teaching critical reading to elementary school children. Reading Research Quarterly, 3, 435-438.

Application Knowledge Assessment II

Assessment Tasks for application knowledge are intended to manifest students' implicit cognitive process of (a) cuing retrieval of information about a relevant generalization from long-term memory, and then (b) transferring that information when recognizing novel objects and actions that are examples of the generalization. Objectives for application knowledge usually identify the generalization. In the typical Assessment Task for application knowledge students are given a number of possible novel objects or actions and asked to identify those that are examples of a generalization. Novel objects and actions are ones students have not seen before. The objects and actions must be novel so that transfer is required. If they are not novel then transfer is not required, and the Assessment Task is actually for recall knowledge. As we found in an earlier post application knowledge is frequently confused with recall knowledge. Where possible, students should also be asked to explain why they chose the objects or actions they did, and perhaps also why they did not choose the others. Both explanations provide the teacher a diagnostic understanding of the relative adequacy of each student's application knowledge of a generalization. Teachers' later instruction is based on their understanding of the particular adequacy of each student.

WRITING

Here is an illustration of an Assessment Task for a writing objective. The objects are sentences. The sentences are novel for students because they have not seen them before. The objective identifies the generalization, redundancy in sentences. The task's directions calls it sentences that are too wordy. The Assessment Task shown here is used early in instruction for the objective so it contains short sentences. As students' application knowledge of the generalization increases the teacher will make the sentences progressively longer, eventually providing paragraphs, and more.

An alternative Assessment Task might present each sentence separately, and ask if it is too wordy, and why. Diagnosis of students' performances of Assessment Tasks should be considered a regular part of assessment. Diagnosis of students' application knowledge is guided by the teacher's own mental model of students' implicit process of application knowledge, which involves (a) retrieving from long-term memory information about a relevant generalization, and then (b) transferring that generalization when recognizing novel object or actions that are examples of it. When students' written explanations are not clearly indicative of adequacy or inadequacy in terms of the mental model, the teacher will want to meet individually with them so students can explain their choices orally, and the teacher can ask questions about the generalization and their transferring of it to the sentences selected.

SCIENCE

Here is an Assessment Task for a science objective. The objective identifies the physics generalization, thermodynamics. The objects and actions in this Assessment Task are novel because the teacher has not shown them to students before. Notice that the name of the generalization is not given in the Assessment Task. Students must retrieve information about the generalization from long-term memory on their own. This is somewhat like the study of novices and experts in physics who sorted cards that illustrated different physics generalizations. That study was described in an earlier post.

Teachers' frame of reference in diagnosing students' performances of the Assessment Task is the teachers' own mental model of the implicit cognitive process of application knowledge. The teachers' diagnosis of students' performances uses that mental model in judging the adequacy of students' written explanations and what they say to the teacher in their later individual oral explanations. The teacher will want to understand what physics generalization, if any, each student chose to use, and why he or she chose to use it. The teacher will also want to understand how students transferred that retrieved physics generalization to the actions, including why they did not choose the other two actions. Teachers' instruction for the objective is based on their diagnosis of students' performance of the task.

MATHEMATICS

Here is an illustration of an Assessment Task for a mathematics objective. The objects are intended to be novel for students because the teacher has not shown fractions in rectangles before.

Teachers' diagnosis of students following their completion of the Assessment Task will often have to be done individually because in the average mathematics curriculum fractions are introduced in the primary grades, before students are able to express themselves in writing nearly as well as they are able to orally. Teachers' frame of reference in diagnosing students' performances is based on the teachers' own mental model of students' implicit cognitive process of application knowledge.

Application Knowledge Assessment I

Assessment Tasks, like instructional tasks, are response-demand activities that function as explicit manifestations of students' implicit and unobservable cognitive processes. Assessment Tasks for application-knowledge objectives are designed to cue students' implicit cognitive process of recognizing whether novel objects or actions are examples of the generalizations that are appropriate for the objectives.

Teachers will usually need to engage in diagnosis of students' performance of Assessment Tasks in order to determine students' understanding, which will enable them to adapt their instruction to students' understanding. Diagnosing students' performance of Assessment Tasks is less complex with a lower-order cognitive process like recall knowledge than it is with a higher-order cognitive process like application knowledge. Recall knowledge involves the retrieval of information encoded in long-term memory in much the same way it was originally presented as sensory input. Diagnosis involves simply determining whether the retrieved information is correct For instance, here are two Assessment Tasks for recall-knowledge objectives. The first is knowing about leaves, and it was shown in a more complete form in an earlier posting.

Here is an Assessment Task for the recall-knowledge objective of knowing about thermodynamics.

These two Assessment Tasks are not adequate for application knowledge because they do not ask students to recognize whether novel objects or actions are examples of a generalization. They do not require the transfer of previously encoded information to novel objects or actions, ones not encountered previously. Here is an Assessment Task for the application-knowledge objective of knowing what a summary is. Students are assigned to read an article in their science texts that is entitled, "Treasure on the Ocean Floor." It is three pages long. They have not read it before. When students finish reading the teacher projects this exercise on the screen.

Notice that the novel objects in the Assessment Task consist of examples of three possible summaries for the science article. The task is intended, first, to cue students' retrieval of information about the generalization summary from their long-term memories, and then, second, to have them transfer that information when choosing the statement that best exemplifies an adequate summary of the article. The directions identify the generalization to be retrieved. The sentences are the novel objects to which that generalization summarization should be transferred. Notice that students have a one-in-three chance of simply guessing the right answer. Or they might not use the intended generalization at all, such as picking out the third statement, which is the correct one, because it is the longest. The only way teachers are able to determine if students possess an adequate application knowledge of the generalization summary is to use an active diagnosis consisting of asking them follow-up questions, such as "Why do you think the third one is the best summary? What is wrong with each of the other two summaries?" The key to teachers' accurate diagnosis is asking questions that are based on the teachers' mental model of the implicit cognitive processes involved in students' application knowledge. Teachers use the mental model of application knowledge like a map in locating where each student is relative to the map.

Teachers' diagnostic ability is a significant factor in the effectiveness of their instruction [1]. Some even regard teachers' diagnostic ability as one of four major components of teaching expertise [2]. There is evidence that teachers' diagnostic ability contributes significantly to students' academic achievement [3]. Teachers obviously must possess an adequate knowledge of the subject matter they teach. However, teachers with high levels of subject-matter knowledge often overestimate students' actual understanding [4]. Teachers' overestimation may be caused by a tendency to use their own subject-matter knowledge as the model against which to judge students' performance, thus they think that if they understand then their students must also [5]. Teachers' diagnostic ability will be the greatest when it is based both on their subject-matter knowledge and on their mental model of the implicit cognitive processes students are being asked to use, such as application knowledge.

Incidentally, with reference to the Assessment Task for summarization described above, asking students to read the story and write their own summary, even though it is a less wordy task, stull requires the more complex implicit cognitive process of procedural knowledge. Instruction for the procedural knowledge of summarizing is most effective once students have already learned an application knowledge of summaries. They will have a great deal of difficulty learning to write their own summaries before they have even learned to recognize a good summary when they see it. For instance, think back to the novice-expert physics study described in an earlier posting, where the novices were unable to recognize the physics generalizations exemplified in novel physics problems shown on cards. Because the novices were unable to recognize the examples of the physics generalization shown on the cards, they would not be able to solve those physics problems unless they were told which physics generalization to use with each physics problem. Until they learn application knowledge of those physics generalizations, instruction for a procedural knowledge of them will be ineffective.

____________________

1. (For example). Shepard, L.A. (2001). The role of classroom assessment in teaching and learning. In V. Richardson (Ed.), Handbook of research on teaching (pp. 1066-1101) Washington DC: American Educational Research Association.

2. (For example). Weinert, F.E., Helmke, A., & Schrader, R.W. (1992). Research on the model teacher and the teaching model. In F.K. Oser, A. Dick, & J.L. Patry (Eds.), Effective and responsible teaching--The new synthesis (pp. 249-260). San Francisco: Jossey-Bass.

3. Seidel, T., & Shavelson, R.J. (2007). Teaching effectiveness research in the last decade in disentangling meta-analysis results. Review of Educational Research, 77, 454-449.

4. Nathan, M.J., & Petrosino, A.J. (2003). Expert blind spot among preservice teachers. American Educational Research Journal, 40, 905-928.

5. Nickerson, R.S. (1999). How we know--and sometimes misjudge--what others know: Imputing one's own knowledge to others. Psychological Bulletin, 125, 737-759.

Application Knowledge Objectives

Teachers can categorize an objective as Application Knowledge if they consider that students demonstrate mastery of the objective when they are able to recognize whether novel objects or actions are examples of the generalization that is appropriate for the objective. In order for teachers to consider that a certain objective refers to Application Knowledge, they must be able to identify a generalization for the objective. The generalization is usually identified in the objective. Here are examples of objectives derived from a variety of subjects and curriculum materials that can be categorized as Application Knowledge because they, more or less, identify the generalization that would be learned as Application Knowledge.

POSSIBLE APPLICATION KNOWLEDGE OBJECTIVES

Social Studies

1. Understand that groups of people form governments in order to make decisions that affect all

members of the society. People in a democracy govern themselves directly or through the

persons they elect.

2. Apply their knowledge that the Constitution of the United States is the basis for the way our

government works.

3. Be able to demonstrate that individuals differ in physical traits because of heredity, but that

their likenesses are greater than their differences.

Science

1. Identify examples which show that metabolically active cells are usually small because

materials can move into, through and out of small cells quickly.

2. Understand that the stars and the sun seem to rise in the east, move across the sky from the

east to west, and set in the west because the earth is spinning in an easterly direction.

3. Know that levers make it easier to lift heavy weights because they shift some of the weight to

a fulcrum. Moving the fulcrum to different places on a bar makes the same object harder or

easier to lift.

Mathematics

1. Recognize examples of the fraction one-half.

2. Understand the principle of place value for ones, tens and hundreds.

3. Demonstrate that in a plane, if two lines are perpendicular to the same line, they are parallel.

Language

1. Realize that good writers avoid redundancy by expressing their ideas in ass few words as

possible.

2. Demonstrate that words beginning with the written letter b begin with the same sound in the

words boy, bat and bone.

3. Pick out in selections they have written examples of the major story elements of narrative

text: Setting, Theme, Plot and Resolution.

Most of these objectives would refer to Recall Knowledge if their wordings were treated as target information to be memorized more or less as stated in the objective. For example, the third science objective would refer to Recall Knowledge if this was the Assessment Task for it: Why do levers make it easier to lift heavy weights? The task refers to Recall Knowledge because it asks students to retrieve verbal information from long-term memory pretty much as it was originally encoded. The third science objective would refer to Application Knowledge only if the Assessment Task consists of objects or actions that are examples and non-examples of the generalization fulcrum that students have not seen before, and students are asked to identify which are examples, and then to explain why each is either an example or non-example. Students often fail to learn desired Application Knowledge because their teachers confuse it with Recall Knowledge. Consequently, students are able to recite definitions of generalizations but are unable to recognize novel objects and actions that are examples of those generalizations.

The next five posts deal with Assessment Tasks for Application Knowledge. These tasks are very important because they provide the teacher a frame of reference for instruction, and a basis for diagnosing students' moment-by-moment learning of Application Knowledge. In addition, Assessment Tasks for Application Knowledge look like the Application-Practice Tasks that are used in the second step of instructing for Application Knowledge. This is analogous to the way Assessment Tasks for Recall Knowledge look like the Recall-Practice Tasks that are used in the second step of instructing for Recall Knowledge.

Application Knowledge II

Application knowledge is a crucial component in students' ability to solve problems [1]. Students often possess a rudimentary procedural knowledge of how to carry out certain procedures, but do not posses the application knowledge about when to carry out those procedures when solving problems. For example, possessing application knowledge of the physics generalizations for diffusion, photosynthesis and velocity is necessary in solving many chemical, biological and physical problems [2]. Application knowledge of the mathematics generalization for the addition principle in probability is necessary in solving many mathematics problems [3]. It is analogous in playing chess to knowing how to properly move chess pieces on a board (which novices have), but not possessing an application knowledge of numerous key playing positions and the most appropriate next move or chess pieces for each position (which experts have) [4].

A classic study of novices and experts in college physics demonstrates the relationship between application knowledge of physics generalizations and ability in problem solving [5]. Undergraduate physics majors (novices) and college faculty in physics (experts) were individually given 24 cards showing physics problems they had not seen before and were asked to sort the problems into six or more separate categories in any way they liked, and to do so in less than a minute. The problems were depicted as drawings on the cards, such as a drawing of two weights joined by a rope that ran over a pulley. When they finished sorting they were asked to explain the rationale for their classifications. The members of both groups were alike in that they created about 8 categories, and finished in about 40 seconds. But they were unlike in the kinds of categories they created, and in the explanations they gave. Experts sorted the cards into categories corresponding to the physics generalizations exemplified in the problems. Consequently, some problems with pulleys were placed in a pile with some problems having inclined planes if they were examples of the same physics generalization. Experts utilized their application knowledge of the physics generalizations in sorting the problems into categories. Novices, on the other hand, sorted the cards into categories corresponding to their physical characteristics. They placed all cards with pulleys into one pile and all cards with inclined plans in another, and so forth. The novices did not possess the application knowledge needed in recognizing the drawings as examples of specific generalizations. They had to fall back on their application knowledge of shapes. The novices would have been unable to solve the problems shown on the cards unless they were told which equations to use. They are like the chess novices who know the basic procedures for moving chess pieces properly but are unable to recognize examples of key chess positions. Whatever instruction the novice physics students received for the physics generalizations was not effective in developing an application knowledge of them. The novices likely possess only recall knowledge of the physics generalizations: meaning that they are able to recall definitions of the generalizations, but are unable to recognize novel examples of them, such as with the drawings of physics problems.

Students often face severe difficulties in learning application knowledge because many generalizations are abstract and complex [6]. Teachers frequently fail to appreciate that. Duckworth noted that: "Teachers are often, and understandably, impatient for their students to develop clear and adequate ideas, but putting ideas in relation to each other is not a simple job. It is confusing, and that confusion does take time. All of us need time for our confusion if we are to build the breath and depth that gives significance to our knowledge." [7]

Students often intuitively acquire application knowledge of faulty generalizations, and that hinders their learning new application knowledge [8]. Examining studies of students' learning of faulty generalizations provide us a clearer understanding of the implicit cognitive processes involved in application knowledge.

In a study at the high school level a researcher identified students in a physics class who were able to calculate correctly the speed and position of moving objects when given the appropriate equation. Those students were given an assessment task for application knowledge. They were asked to draw the path of a ball that was kicked off a cliff. They had not previously talked about such a situation. Most of them showed the ball going straight out from the cliff and then falling straight down. They drew this path in spite of the fact that it clearly was not a correct example of the physics generalizations about motion. According to the physics generalizations about motion the ball should have fallen off the cliff in a graceful parabola [9]. Those students clearly did not possess an application knowledge of the physics generalizations about motion; although they might have had a recall knowledge of it, meaning that they could retrieve a definition from their long-term memories.

Many young adults go through high school and university physics courses without ever giving up their application knowledge of faulty pre-Newtonian generalizations about motion [10]. For example, the teacher in this study has been instructing college students for an application knowledge of the generalization about Newton's law of inertia. It states, in part, if an object is in motion it continues in a straight line unless it is acted upon by physical forces. Following instruction, students were given an assessment task for application knowledge. The assessment task (a curved pellet shooter) is shown below. They have not seen this curved pellet-shooter example before. They were asked to draw the path of a pellet when it emerged from the curved tube though which it had been shot. Their two most common performances are shown. The correct performance is A [11].

Another study reported that after two months of instruction for an application knowledge of the physics generalization acceleration in an introductory physics course, only 40 percent of the students could correctly perform application knowledge assessment tasks for the generalization [12].

Research indicates that students of all ages, preschool through Ph.D., and in many subject areas, hold onto their application knowledge of faulty generalizations tenaciously even when the evidence that initially produced the generalizations is discredited. They hold on because they believe the faulty generalizations were initially derived from "real life" experiences [13]. For example, many students believe the faulty generalization that heavier objects displace more water than lighter objects. Based on the faulty generalization, they predict that the heavier of two metal cubes of equal size would displace more water in a container than would the lighter of the two metal cubes. They place the cubes in the container of water one at a time and measure the height of the water before and after each cube is placed in the container. They are surprised when both cubes raise the water to an equal level, but they are undaunted. A student explains, "Your tricked us. You brought magic water." Students were willing to violate their understanding about the nature of water in order to defend their faulty generalization about weight determining the displacement of water [14].

Another group of students had acquired an application knowledge of a faulty generalization about heavier objects falling faster than lighter objects. On the basis of that faulty generalization they predicted that if a heavier metal ball and a lighter wooden ball of the same size were dropped at the same time from the same height, the metal ball would hit the ground first. When they tested out their prediction by dropping the two balls at the same time, and watching the balls hit at the same time, they refused to accept the contradictory information. Some claimed they observed the metal ball hitting first. Others claimed the balls must be the same weight even though the scales showed different weights [15].

A teacher tried to change students' application knowledge of the faulty generalization that electricity wears out as it travels from a battery around a circuit containing a light bulb. Students predicted that if the teacher placed two ammeters to measure current on each side of the light bulb, they would show less current on one side. When the ammeters were in place and the current turned on, the ammeters showed the current on each side of the bulb was the same. But instead of changing their faulty generalization, students tended to reject the data on methodological grounds. They claimed the ammeters were not accurate, the light bulb was bad, and the battery did not work [16].

Students' difficulties solving science problems can frequently be traced to their an application knowledge of faulty generalizations. One study reported that 80 percent of the university-level engineering students it focused on were proficient in solving word problems only when they were given the algebraic equations that were appropriate for solving the problems. They were like the novice chess players who knew how to move chess pieces properly, but lacked application knowledge of key playing positions. The difficulties engineering students were having were traced, in part, to their lack of a very basic application knowledge of what "X" means in algebraic equations, such as "6X + 5 = 17 [17].

Many of the difficulties students have in carrying out multi-digit addition and subtraction with regrouping have been traced to their inadequate application knowledge of generalizations about place value [18]. In a large national study it was found that less than 50 percent of third graders possess application knowledge of place value in the hundreds digit, and only 64 percent in the tens digit. One-third of third graders gave incorrect answers on two-digit subtraction problems involving regrouping, and half did so for three-digit problems involving regrouping [19]. Students' severe deficiencies in their application knowledge of key mathematics generalizations, and the resulting inadequate performance in arithmetic computation, have been documented by extensive research [20]. The majority of students in elementary school acquire the ability to "run off" arithmetic procedures, but few acquire an application knowledge of the generalizations that are the basis of those procedures. They consequently have great difficulty finding errors in their work and in solving word problems [21].

A four-year longitudinal study of fifth and eight-graders' acquisition of recall knowledge and application knowledge of such social studies generalizations as freedom and representativeness found that some acquired recall knowledge of the generalizations, but no application knowledge of them [22]. This was evidenced by their ability to recall definitions of the terms, and their inability to recognize novel examples of the terms. Reviews of large-scale studies of students' attitudes and knowledge about government found that students typically exhibit some strong feelings about their government, but possess by little actual recall knowledge of it, and no application knowledge [23]. One study reported that 80 percent of fourth-graders chose the presidency as the most important role for a adult, but less than 25 percent of them could describe any of the president's duties [24].

Studies indicate that basing instruction on current textbooks results in students' inadequate acquisition of application knowledge because the texts have major instructional weaknesses. Examinations of social studies textbooks found that they do not provide the kind of information about generalization examples, both familiar and novel, that are needed in developing students' application knowledge of social studies generalizations [25]. In fact, the authors of the textbooks seem to assume that students reading the texts have already acquired application knowledge of the generalizations that the texts are presumably intended to teach.

A growing realization among teachers of the general ineffectiveness of most traditional textbook-based instruction is causing many of them to provide their students more "hands-on" laboratory activities. But these laboratory activities are unlikely, by themselves, to increase students' application knowledge of generalizations. They are often just confusing. What is needed is more emphasis on "minds-on" activities [26]. The activities should provide students both appropriate application-knowledge instructional tasks to perform actively, and appropriate application-knowledge guidance while they perform those tasks.

Asian children in elementary school consistently out perform children from the U.S. Studies over the last two decades of U.S. and Asian classrooms indicate a reason U.S. children perform poorly is that Asian children receive more instruction for application knowledge while they are learning mathematical procedures. For example, Japanese and Taiwanese first and fifth-graders score much higher on test items measuring application knowledge of place value and of problem solving in multi-digit addition and subtraction with regrouping than do U.S. first and fifth-graders [27]. Korean children aged 6, 7 and 8 carry out multi-digit addition and subtraction with regrouping much more accurately and have a greater understanding of what they are doing than do their U.S. counterparts [28]. Korean, Japanese and Chinese children possess a higher degree of application knowledge of generalizations for place value than do children in the United States [29]. School in Mainland China, Japan, the former Soviet Union and Taiwan all begin instruction for application knowledge of generalizations for place value and procedures in multi-digit addition and subtraction with regrouping earlier than do schools in the U.S., and they complete this instruction earlier [30]. Studies in Japanese, Chinese and U.S. classrooms found that Asian teachers provide instruction for application knowledge while U.S. teachers seldom do. The result is that Asian children understand why a certain mathematical procedure works, how to use it, and how to spot and correct errors [31].

____________________

1. (For example). Brewer, W.F., & Chinn, C.A. (1991). Entrenched beliefs, inconsistent information, and knowledge change. In L. Birnbaum (Ed.). The International Conferences of the Learning Sciences: Proceedings of the 1991 Conference (pp. 67-73). Charlottesville, VA: Association for the Advancement of Computing in Education.

2. Schoenfeld, A.H. (1985). Mathematical problem solving. New York: Academic Press.

3. Anderson, J.R., & Lebiere, C. (1998). Atomic components of thought. Mahwah, NJ: Lawrence Erlbaum Associates.

4. Ericsson, K.A., Charness, N., Feltovich, P., & Hoffman, R.R. (Eds.). (2006). The Cambridge handbook of expertise and expert performance. Cambridge, UK: Cambridge University Press.

5. Chi, M.T.H., Feltovich, P.J., & Glaser, R. (1981). Categorization and representation of physics knowledge by experts and novices. Cognitive Science, 5, 121-152.

6. Chi, M.T.H. (2000) Cognitive understanding levels. In A.E. Kazkin (Ed.). Encyclopedia of psychology (pp. 146-151).

7. Duckworth, E. (1987). "The having of wonderful ideas" and other essays on teaching and learning. New York: Teachers College, Columbia University. p. 82.

8. Chi, M.J.H. (2005). Common sense conceptions of emergent processes: Why some misconceptions are robust. Journal of the Learning Sciences, 14, 161-199.

9. Strieley, T. (1988), August/September). Physics for the rest of us. Educational Researcher, 17(6), 7-12.

10. (For example). Champagne, A.S., Klopfer, L.E., & Anderson, J.H. (1980). Factors influencing the learning of mechanics. American Journal of Physics, 48, 1074-1079.

11. Adapted from Green B.F., McCloskey, M., Caramazza, A. (1985). The relationship of knowledge to problem solving, with examples from kinematics. In S.F. Chipman, J.W. Segal, & R. Glasser. (Eds.). Thinking and learning skills (Vol. 2, pp. 127-140).

12. Reif, F. (1987). Instructional design, cognition, and technology: Application to the teaching of scientific concepts. Journal of Research in Science Teaching, 24, 309-324.

13. (For example). Carpenter, T.P., Moser, J., & Romberg, T. (Eds.). (1982). Addition and subtraction: A developmental perspective. Hillsdale, NJ: Erlbaum.

14. Linn, M.C. (1983). Content, context, and process in adolescent reasoning. Journal of Early Adolescence, 3, 63-82.

15. (For example). Eylon, B. & Linn, M.C. (1988). Learning and instructor: An examination of four research perspectives in science education. Review of Educational Research, 58, 251-301.

16. Johusa, S., & Dupin, J.J. (1987). Taking into account student conceptions in instructional strategy: An example in physics. Cognition and Instruction, 4, 117-135.

17. Clement, J., Lockhead, J., & Monk, G. (1979). Translation difficulties in learning mathematics. American Mathematics Monthly, 88 (4), 287-290.

18. Fuson, K.C. (1990). Conceptual structures for multi-digit addition, subtraction, and place value. Cognition and Instruction, 7, 343-403.

19. Kouba, V.L., Brown, C.A., Carpenter, T.P., Lindquist, M.M., Silver, E.A., & Swafford, J.O. (1988). Results of the fourth NAEP assessment of mathematics: Number, operations and word problems. Arithmetic Teacher, 35, (8), 14-19.

20. (For example). Resnick, L.B., & Omanson, S.F. (1990). Learning to understand arithmetic. In R. Glaser (Ed.), Advances in instructional psychology (Vol. 3, pp. 41-95). Hillsdale, NJ: Erlbaum.

21. Brown, A.L., & Campione, J.C. (1990). Interactive learning environment and the teaching of science and mathematics. In M. Gardner, J.G. Greeno, F. Reif, A.H. Schoenfeld, & E. Stage (Eds.), Toward a scientific practice of science education (pp. 111-140). Hillsdale, NJ: Erlbaum.

22. Sinatra, G.M., Beck, I.L., & McKeown, M.G. (1992). A longitudinal study of the characteristics of young students' knowledge of their country's government. American Educational Research Journal, 29, 633-661.

23. (For example). McKeown, M.G., & Beck, I.L. (1990). The assessment and characteristics of young learners' knowledge of a topic in history. American Educational Research Journal, 27, 588-627.

24. Greenstein, F.J. (1965). Children and politics. New Haven: Yale University Press.

25. Beck, I.L., McKeown, M.G., & Gromoll, E.W. (1989). Learning from social studies texts. Cognition and Instruction, 6, 99-158.

26. (For example). Smith, E.L., & Anderson, C.W. (1984). The planning and teaching of intermediate science study: Final report. (Research Series No. 147). East Lansing: Michigan State University, Institute of Research on Teaching.

27. Stigler, J.W., Lee, S.Y., & Stevenson, H.W. (1990). The mathematical knowledge of Japanese, Chinese, and American elementary school children. Reston, VA: National Council of Teachers of Mathematics.

28. Song, M., & Ginsburg, H.P. (1987). The development of informal and formal mathematics thinking in Korean and U.S. children. Child Development, 57, 1286-1296.

29. Miura, I.T., Kim, C.C., Chang, C-M, & Okamato, U. (1988). Effects of language characteristics on children's cognitive representation of number: Cross-national comparisons. Child Development, 59, 1445-1450.

30. Fuson, K.C., Stigler, J.W., & Bartsch, K. (1988). Grade placement of addition and subtraction topics in China, Japan, the Soviet Union, Taiwan, and the United States. Journal for Research in Mathematics Education, 19, 449-458.

31. Perry, M. (2000). Explanation of mathematical concepts in Japanese, Chinese, and U.S. first and fifth-grade classrooms. Cognition and Instruction, 18, 181-207.