Discourse cohesion in text and tutorial dialogue
16
pages
English
Documents
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
16
pages
English
Documents
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
Publié par
Langue
English
InformationDesig nJournal15(3), 199–213 ©2007JohnnjaminsBe PublishingCompanyArthurC.Graesser,MoongeeJeon,YanYan&ZhiqiangCaiDiscourse cohesion in text and tutorial dialogueKeywords: cohesion, discourse types, readability, software tools, automatic analysis There has been a dramatic increase in computer analyses of large text corpora during the last decade. This can Discourse cohesion is presumably an important facilitator partly be explained by revolutionary advances in compu-of comprehension when individuals read texts and hold tational linguistics (Jurafsky & Martin, 2000; Walker conversations. This study investigated components of et al., 2003), discourse processes (Pickering & Garrod, cohesion and language in different types of discourse 2004; Graesser et al., 2003), the representation of world about Newtonian physics: A textbook, textoids written by knowledge (Lenat, 1995; Landauer et al., 2007), and experimental psychologists, naturalistic tutorial dialogue corpus analyses (Biber et al., 1998). Because thousands of between expert human tutors and college students, and texts can be quickly accessed and analyzed on thousands AutoTutor tutorial dialogue between a computer tutor of measures in a short amount of time, data mining is and students (AutoTutor is an animated pedagogical emerging as a standard methodology in a broad spec-agent that helps students learn about physics by holding trum of fields.conversations in natural ...
Voir
Publié par
Langue
English
Arthur C. Graesser, Moongee Jeon, Yan Yan & Zhiqiang Cai Discourse cohesion in text and tutorial dialogue
Keywords: cohesion, discourse types, readability, software tools, automatic analysis Discourse cohesion is presumably an important facilitator of comprehension when individuals read texts and hold conversations. This study investigated components of cohesion and language in different types of discourse about Newtonian physics: A textbook, textoids written by experimental psychologists, naturalistic tutorial dialogue between expert human tutors and college students, and AutoTutor tutorial dialogue between a computer tutor and students (AutoTutor is an animated pedagogical agent that helps students learn about physics by holding conversations in natural language). We analyzed the four types of discourse with Coh-Metrix, a software tool that measures discourse on different components of cohesion, language, and readability. The cohesion indices included co-reference, syntactic and semantic similarity, causal cohesion, incidence of cohesion signals (e.g., connectives, logical operators), and many other measures. Cohesion data were quite similar for the two forms of discourse in expository monologue (textbooks and textoids) and for the two types of tutorial dialogue (i.e., students interacting with human tutors and AutoTutor), but very different between the discourse of expository monologue and tutorial dialogue. Coh-Metrix was also able to detect subtle differences in the language and discourse of AutoTutor versus human tutoring.
Information Design Journal () , 99–23 © John Bnejamins Publishing Company
ere ha een a dramati inreae in omputer analye of large text orpora during the lat deade. i an partly e explained y reolutionary adane in ompu -tational linguiti (Jurafky & Martin, 2000; Walker et al., 2003), dioure proee (Pikering & Garrod, 2004; Graeer et al., 2003), the repreentation of world knowledge (Lenat, 1995; Landauer et al., 2007), and orpu analye (Bier et al., 1998). Beaue thouand of text an e quikly aeed and analyzed on thouand of meaure in a hort amount of time, data mining i emerging a a tandard methodology in a road pe -trum of field. Reearher at the Unierity of Memphi hae reently deeloped a ytem alled Coh-Metrix ( http:// ohmetrix.memphi.edu, Graeer et al., 2004), a ompu -tational tool that produe meaure of the linguiti and dioure harateriti of text (oth printed text and tranript of oral dioure). e alue on the Coh-Metrix meaure an e ued to inetigate the oheion of the expliit text and the oherene of the mental repreentation of the text. Our definition of cohe-sion onit of linguiti harateriti of the expliit text that play ome role in onneting idea in the text. Coherence inlude harateriti of the text (i.e., apet of oheion) that are likely to ontriute to the oherene of mental repreentation.
Arthur C. Graesser et a • l. Discourse cohesion in text and tutorial dialogue
Reearher at the Unierity of Memphi hae alo deeloped an intelligent tutoring ytem alled Auto -Tutor (Graeer et al., 2005; Graeer, Lu et al., 2004). AutoTutor i a learning enironment that tutor tudent y holding a oneration in natural language AutoTu -. tor tutor tudent in Newtonian qualitatie phyi, omputer literay, ritial thinking, and other topi that exhiit explanation and eral reaoning. e dialogue of AutoTutor i uffiiently tale that it hold it own in onering with tudent for hour. It i alo deigned to mimi the dioure pattern of human tutor (Graeer, Peron, & Magliano, 1995). e purpoe of the preent tudy wa to ue the Coh-Metrix tool to analyze the omponent of ohe -ion and language in different type of dioure aout Newtonian phyi. We analyzed a ample of hapter in a textbook , textoids written y experi-mental pyhologit, naturalistic tutorial dialogues etween expert human tutor and ollege tudent, and AutoTutor tutorial dialogues etween a omputer tutor and tudent. One trong irtue of thi tudy wa our attempt to ahiee information equialene with repet to the ontent oered in the four orpora. We did thi y filtering ontent that oered the ame et of ore ontrut in phyi, namely Newtonian law of fore and motion. Gien thi ontrol oer informa -tion equialene, we inetigated how the four type of dioure differ with repet to language and oheion. We might expet the two type of expoitory mono -logue (textook and textoid) to e different from the two type of interatie dialogue (human tutor and AutoTutor). We might alo expet differene etween the two monologue type or etween the two dialogue typer. For example, textook and textoid preumaly differ from eah other eaue textook are written y profeional writer, wherea textoid are generated y experimental pyhologit to atify methodologi -al ontraint. What i le lear i the nature of thee
idj (), , -
differene. If the Coh-Metrix tool i alid and ueful, it hould detet utle and explainale differene in the four type of dioure. e preent tudy inetigated whether thi i indeed the ae. Coh-Metrix ere are approximately 60 indie in the Coh-Metrix erion (. 2.0) that i aailale to the puli. Aſter the uer of Coh-Metrix enter a text into the We ite, it print out meaure of the text on indie that pan different leel of dioure and language. Coh-Metrix wa deigned to moe eyond tandard readaility formula, uh a Fleh-Kinaid Grade Leel (Klare, 1974–1975). Suh formula rely exluiely on word length and entene length. For example, in the Fleh-Kinaid Grade Leel index (Figure 1) words refer to mean numer of word per entene and syllables refer to mean numer of yllale per word. (1) Grade Leel = .39 * Word + 11.8 * Syllale - 15.59 Sentene length and word length do in fat routly predit reading time (Haerlandt & Graeer, 1985), ut ertainly there i more to reading diffiulty than word and entene length. ere mut alo e important deeper meaure of language and oheion. Coh-Metrix aim to proide thee deeper meaure. e Coh-Metrix indices (i.e., meaure, metri) oer multiple leel of language and dioure. Some indie refer to harateriti of indiidual word, a ha een ahieed in many other omputer failitie, uh a WordNet (Fellaum, 1998) and Linguiti Inquiry Word Count (Penneaker & Frani, 1999). Howeer, the majority of the Coh-Metrix indie inlude deeper or more proeing-intenie algorithm that analyze yntax, referential oheion, emanti oheion, and dimenion of the ituation model. Coh-Metrix i the only omputer faility aailale to the puli for free
Arthur C. Graesser et a • l. Discourse cohesion in text and tutorial dialogue
that analyze language and dioure on a road et of omponent at multiple leel. A naphot of the landape of indie i proided in thi etion. Reearher at the Unierity of Memphi hae oer 600 indie in their internal omputer ytem, of whih 60 are aailale on the puli We ite ( http:// ohmetrix.memphi.edu /). i artile foue exlu -iely on the et of pulily aailale meaure. Reearh -er at the Unierity of Memphi hae alo ealuated the auray of the Coh-Metrix indie in oer 60 pulihed tudie, whih an e aeed at the puli We ite. Howeer, it i eyond the ope of thi tudy to reiew the reearh in thee aement. Word measures . Coh-Metrix meaure word on a large numer of harateriti, mot of whih will not e defined in thi artile (ee the help ytem on the We ite http://ohmetrix.memphi.ed u). ere are meaure of word frequeny in the Englih language, whih i aed on the CELEX lexion (Baayen et al., 1993) and other imilar lexion. Coh-Metrix alo ditinguihe etween ontent word (e.g., noun, main er, adjetie) and funtion word (e.g., prepoition, artile), aed on tandard part-of-peeh ategorie that are aepted in the omputational linguiti ommunity. Seeral word indie are diretly releant to oheion, oherene, and omprehenion diffiulty. In partiular, there are word lae that hae the peial funtion of onneting laue and other ontituent in the text (Halliday & Haan, 1976; Louwere, 2002; Sander & Noordman, 2000). e ategorie of onnetie in Coh-Metrix inlude additie ( also, moreover ), temporal ( and then, aſter, during ), aual (because, so ), and logial operator ( therefore, if, and, or ). e additie, temporal, and aual onnetie are udiided into thoe that are poitie ( also, because ) or negatie ( but, however ). e word indie inlude negation ( not, n’t ) that pan different leel of ontituent truture and ariou onditional expreion ( if, given ). Negation, onditional
idj (), , -
expreion, and negatie onnetie are predited to e affiliated with omplex oneptualization and rhetorial truture, uh a ounterfatual, hypothetial world, multiple perpetie, qualifiation, hedge, and argu -mentation. A higher inidene of thee word hould therefore predit text diffiulty. e incidence of eah word la i omputed a the numer of ourrene per 1000 word. An inidene ore i neeary for omparing text of different ize. A text with higher oheion would hae a higher ini -dene of word lae that onnet ontituent. Syntax . Coh-Metrix analyze entene yntax with the aitane of a yntati parer deeloped y Char -niak (2000). e parer aign part-of-peeh ategorie to word and yntati tree truture to entene. Our ealuation of eeral parer howed etter perfor -mane of Charniak’ parer than other major parer when omparing the aigned truture to judgment of human expert (Hemphill et al., 2006). Coh-Metrix ha eeral indie of yntati omplexity, two of whih (the mean numer of modifier per noun-phrae, and the numer of word efore the main er of the main laue) are reported in thi artile. e mean numer of modifier per noun-phrae i an index of the omplexity of referening expreion. For example, very large accel-erating objects i a omplex noun-phrae with 3 modifier of the head noun objects . e numer of word efore the main er of the main laue i an index of yntati omplexity eaue it plae a urden on the working memory of the omprehender (Graeer, Cai, Louwere, & Daniel, 2006). Referential and semantic cohesion . Referential ohe-ion our when a noun, pronoun, or noun phrae refer to another ontituent in the text. For example, in the entene As the earth orbits the sun, it exerts a force, the word it refer to the word earth y irtue of a ynta-ti rule of pronoun aignment. A referring expreion (E) i the noun, pronoun, or noun-phrae that refer to
Arthur C. Graesser et a • l. Discourse cohesion in text and tutorial dialogue
another ontituent (C). C i deignated a the referent of E. In the example entene, the word it i the referring expreion E, wherea the referent C i the word earth . One form of o-referene that ha een exteniely tud -ied i argument oerlap (Kinth & an Dijk, 1978). i our when a noun, pronoun, or noun-phrae in one entene i a o-referent of a noun, pronoun, or noun-phrae in another entene. e word “argument” i ued in a peial ene in thi ontext, namely it i a ontrat with prediate in propoitional repreentation. e argument oerlap index of Coh-Metrix urrently onid -er exat mathe of argument etween two entene. e alue of thi metri, whih arie from 0 to 1, i the proportion of adjaent entene pair that hare a ommon argument in the form of an exat math. Another form of o-referene i tem oerlap, where a noun in one entene ha a imilar morphologial root (i.e., lemma) a a ontent word in another entene. For example, onider the two entene As the earth orbits the sun, it exerts a force. e orbit is not perfectly round. Orbits and orbit hae ommon tem, o there i tem oerlap, een though one i a main er and the other a noun. e alue of thi metri i the proportion of adja -ent entene pair that hae a tem oerlap. Yet another form of o-referene i anaphori pronominal o-referene. A pronoun ( he, hers, it ) in one entene refer to a referent in another entene. A pronoun an preent a oherene prolem when the omprehender doe not know the referent of the pronoun. Pronoun oſten require a onerational or oial ontext to reole their referent, a oppoed to their referring to other text ontituent. Coh-Metrix ompute the referent of pronoun on the ai of yntati rule, emanti fit, and dioure pragmati y ome exiting algorithm in omputational linguiti (ee Jurafky & Martin, 2000; Lappin & Lea, 1994). e alue of thi metri i the proportion of adjaent entene pair in whih the eond entene ha a
idj (), , -
pronoun that an e uefully linked to a ontitu -ent in the preiou entene y exeuting the pronoun aignment mehanim. In addition to referential oheion indie, Coh-Metrix ha indie that ae the extent to whih the ontent of entene, turn, or paragraph i imilar emantially or oneptually. Coheion and oherene are predited to inreae a a funtion of imilarity. Latent Semanti Analyi (LSA) i the primary method of omputing imilarity eaue it onider impliit knowledge. LSA i a mathematial, tatitial tehnique for repreenting world knowledge, aed on a large orpu of text. e entral intuition i that the mean -ing of a word i aptured y the ompany of other word that urround it in naturaliti doument; two word hae imilar meaning to the extent that they hare imilar urrounding word. LSA ue a tatitial teh -nique alled ingular alue deompoition to ondene a ery large orpu of text to 100-500 tatitial dimen -ion (Landauer et al., 2007). e oneptual imilar -ity etween any two text exerpt (e.g., word, laue, entene, text) i omputed a the geometri oine etween the alue and weighted dimenion of the two text exerpt. e alue of the oine arie from 0 to 1. LSA-aed oheion wa meaured in two way releant to the preent tudy: (1) LSA imilarity etween adjaent entene and (2) LSA imilarity etween adjaent para -graph. Lexial dierity proide a imple, ut le ompu -tationally expenie, approah to omputing emanti oheion of a text. e lexial dierity metri in Coh-Metrix i the type-token ratio ore. i i the numer of unique word in a text (i.e., type) diided y the oerall numer of word (i.e., token) in the text. A low alue mean there i a large amount of redundany in the ontent word of a text. Coheion and oherene hould inreae inerely with type-token ratio. Situation model dimensions . M any apet of a text
Arthur C. Graesser et a • l. Discourse cohesion in text and tutorial dialogue
an ontriute to the situation model (or mental model), whih i the referential ontent or miroworld of what a text i aout (Graeer et al., 1994; Kinth, 1998). Text omprehenion reearher hae inetigated at leat fie dimenion of the ituational model (Zwaan & Radanky, 1998): auation, intentionality, time, pae, and protagonit. A reak in oheion or oherene our when there i a diontinuity on one or more of thee ituation model dimenion. Wheneer uh diontinuitie our, it i important to hae onnetie, tranitional phrae, ader, or other ignaling deie that oney to the reader that there i a diontinuity; we refer to thee different form of ignaling a particles . Coheion i failitated y partile that larify and tith together the ation, goal, eent, and tate oneyed in the text. Coh-Metrix 2.0 analyze the ituation model dimen -ion on auation, intentionality, pae, and time, ut not protagonit. ere are many meaure of the ituation model, far too many to addre in thi artile. e preent tudy onentrated on three indie that meaure oheion on the dimenion of auality, intentionality, and tempo -rality. For aual and intentional oheion, Coh-Metrix ompute the ratio of oheion partile to the inidene of releant referential ontent (i.e., main er that ignal tate hange, eent, ation, and proee, a oppoed to tate). e ratio metri i eentially a onditionalized inidene of oheion partile: Gien the ourrene of releant ontent (uh a laue with eent or ation, ut not tate), what i the denity of partile that tith together the laue? For example, the referential ontent for intentional information inlude intentional ation performed y agent (a in torie, ript, and ommon proedure); in ontrat, the intentional oheion partile would inlude infinitie and intentional onnetie ( in order to, so that, by means of ). Similarly, the referential ontent for auation information inlude ariou lae of eent that are identified y hange-of-tate er and
idj (), , -
other releant lae of er in WordNet (Fellaum, 1998); the aual partile are the aual onnetie and other word lae that denote aual onnetion etween ontituent. In the ae of temporal oheion, Coh-Metrix ompute the uniformity of the equene of main er with repet to tene and apet. e Coh-Metrix help faility i aailale at the We ite for more detail.
AutoTutor Student oneration with AutoTutor were one of the four type of dioure analyzed y Coh-Metrix. Auto -Tutor i a pedagogial agent that help tudent learn y holding a oneration in natural language (Graeer et al., 2005; Graeer, Lu et al., 2004). e learning gain of AutoTutor hae een aeed in the area of omputer literay (Graeer, Lu et al., 2004) and Newtonian phyi (VanLehn et al., 2007). AutoTutor inreae learning y approximately one letter grade when ompared to read -ing textook for an equialent amount of time. AutoTutor’ dialogue are organized around diffi -ult quetion and prolem that require reaoning and explanation in the anwer. e example elow i one of the hallenging quetion on Newtonian phyi. Phyi quetion: If a lightweight ar and a maie truk hae a head-on olliion, upon whih ehile i the impat fore greater? Whih ehile undergoe the greater hange in it motion, and why? Suh quetion require the learner to ontrut approxi -mately 3–7 entene in an ideal anwer and to exhiit reaoning in natural language. e dialogue for one hallenging quetion typially require 50-200 oner -ational turn etween AutoTutor and the tudent. e deep learning expeted to our during thi proe i ditriuted oer many turn. e truture of the dialogue in oth AutoTutor and human tutoring (Chi et al., 2001, Graeer et al., 1995;
Arthur C. Graesser et a • l. Discourse cohesion in text and tutorial dialogue
Van Lehn et al., 2007) an e egregated into three leel or apet: (1) expetation and mioneption-tailored dialogue, (2) a fie-tep dialogue frame, and (3) ompoi -tion of a onerational turn. ee three leel an e automated and produe repetale tutorial dialogue. Expectation and misconception tailored dialogue. i i the primary pedagogial method of affolding good tudent anwer. Both AutoTutor and human tutor typially hae a lit of expetation (antiipated good anwer) and a lit of antiipated misconceptions aoiat-ed with eah main quetion. For example, expetation E and mioneption M are releant to the example phyi prolem. E: e magnitude of the fore exerted y A and B on eah other are equal. M: A lighter or maller ojet exert no fore on a heaier or larger ojet. AutoTutor guide the tudent in artiulating the expe -tation through a numer of dialogue moe: generi pumps (what ele?) to get the tudent to do the talk -ing, hints , and prompts for the tudent to fill in mi -ing word. Hint and prompt are arefully eleted y AutoTutor to produe ontent in the anwer that fill in miing ontent word, phrae, and propoition. For example, a hint to get the tudent to artiulate expeta -tion E might e “What aout the fore exerted y the ehile on eah other?”; thi hint would ideally eliit the anwer “e magnitude of the fore are equal.” A prompt to get the tudent to ay “equal” would e “What are the magnitude of the fore of the two ehile on eah other?” AutoTutor adaptiely elet thoe hint and prompt that fill miing ontituent and therey ahiee pattern ompletion. For thoe tudent who annot fill in the ontent of an expetation aſter multiple onerational turn, AutoTutor tep in a a lat reort and imply expree the expetation a an assertion. AutoTutor end up generating a high proportion of
idj (), , -
pump and hint for artiulate tudent with high knowl -edge, ut a high proportion of prompt and aertion for low eral, low knowledge tudent. e lit of expeta -tion i eentually oered aſter the multi-turn dialogue and the main quetion i ored a anwered. AutoTutor adapt to the learner in other way than affolding them to artiulate expetation. AutoTutor orret mioneption that periodially arie in the tudent’ talk. When the tudent artiulate a mionep -tion, AutoTutor aknowledge the error and orret it. AutoTutor gie feedak to the tudent on their ontri -ution in mot onerational turn. AutoTutor gie hort feedak on the quality of tudent ontriution: poitie (ery good, rao), negatie (not quite, almot), or neutral (uh huh, okay). AutoTutor attempt to anwer the tudent’ quetion when they are aked. e anwer to the quetion are retrieed from gloarie or from paragraph in textook ia intelligent information retrieal. Five-step dialogue frame . i dialogue frame i pre -alent in human tutoring (Graeer et al., 1995; VanLehn et al., 2007) and i alo implemented in AutoTutor. e fie tep of the dialogue frame are: (1) Tutor ak main quetion, (2) tudent gie initial anwer, (3) tutor gie hort feedak on the quality of the tudent’ anwer in #2, (4) tutor and tudent ollaoratiely interat ia expetation and mioneption tailored dialogue, and (5) tutor erifie that the tudent undertand (e.g., Do you understand? ) Managing one conversational turn . Eah turn of Auto-Tutor in the onerational dialogue ha three information lot (ontituent). e firt lot of mot turn i hort feedak on the quality of the tudent’ lat turn (i.e., poitie, negatie, or neutral). e eond lot adane the oerage of the ideal anwer with either pump, hint, prompt for peifi word, aertion, orretion of mioneption, or anwer to tudent quetion. e third lot i a ue to the tudent for the floor to hiſt from
Arthur C. Graesser et a • l. Discourse cohesion in text and tutorial dialogue
AutoTutor a the peaker to the tudent. For example, AutoTutor end eah turn with a quetion or a geture to ue the learner to do the talking. Dioure marker (e.g., and also, okay, well ) onnet the utterane of thee three lot of information within a turn. e three leel of AutoTutor go a long way in imulating a human tutor. AutoTutor an keep the dialogue on trak eaue it i alway omparing what the tudent ay to antiipated input (i.e., the expeta -tion and mioneption in the urriulum ript). Pattern mathing operation and pattern ompletion mehanim drie the omparion. ee mathing and ompletion operation are aed on latent emanti analyi (Landauer et al., 2007) and ymoli inter -pretation algorithm (Ru et al., 2006) that are eyond the ope of thi artile to addre. AutoTutor annot interpret tudent ontriution that hae no mathe to ontent in the urriulum ript. For example, AutoTutor annot explore topi hange and tangent a tudent introdue them. Howeer, aailale tudie of naturali -ti tutoring (Chi et al., 2001; Graeer et al., 1995) reeal that (a) human tutor rarely tolerate true mixed-initia -tie dialogue with tudent topi-hange that teer the oneration off oure and () mot tudent rarely hange topi, ak quetion, and pontaneouly gra the onerational floor. Intead, it i the tutor that drie the dialogue and lead the dane. Using Coh-Metrix to analyze four types of text on Newtonian physics Coh-Metrix wa ued to analyze the language and dioure of four type of dioure on Newtonian phyi: Textbook chapters , textoids , AutoTutor tutorial dialogue, and naturalistic tutorial dialogue . e text-ook orpu wa the firt 8 hapter from Hewitt’ 1998 textook on Conceptual Physics . e textoid orpu wa 12 phyi paage prepared y an den Broek and hi
idj (), , -
olleague for reearh on the ognitie proee that our during iene omprehenion (Kendeou & an den Broek, in pre). e AutoTutor orpu wa dialogue tranript from a pulihed experiment onduted on 10 phyi prolem with 22 tudent (Experiment 1 of VanLehn et al., 2007); there were 213 oneration total eaue een of them were inomplete. e human tutoring orpu inluded dialogue tranript on the ame 10 phyi prolem ut with a different group of 16 tudent, alo in Experiment 1 of VanLehn et al. (2007). e human tutor held oneration with the tudent through omputer mediated oneration. at i, the tudent and tutor were in different room and interated on omputer. e fie human tutor had Ph.D. in phy -i and were highly trained in pedagogy. e tudent in oth tutoring orpora were ollege tudent enrolled in a phyi oure. e ontent of the four orpora were ery imilar in the ene that they oered Newtonian law of fore and motion. e goal wa to ahiee information equialene in domain knowledge among the four type of dioure o that differene in language and dioure ould e attriuted to the type of text (i.e., genre or regiter). One way of iewing our eletion of the four text type i to ro two dimenion. One dimenion ditinguihe expoitory monologues that are deigned to e read (textook and textoid) from onerational dialogues (tutoring with human and AutoTutor). e language of the former i expeted to e more ompat, literate, and truturally dene than the language of dialogue that ha an affinity to the oral tradition (Bier, 1988; Tannen, 1982). Orthogonal to thi monologue-dialogue dimenion i a eond dimenion that ontrat natural and artifiial dioure. Textook and human tutoring are natural, eologially alid dioure ample, wherea textoid and AutoTutor interation hae ome modium of artifiiality. e reearher attempted to make the textoid and AutoTutor interation well-
Arthur C. Graesser et a • l. Discourse cohesion in text and tutorial dialogue
trutured and oherent, of oure, ut in truth they are ontrained y a reearh agenda or omputational algo -rithm. It i an empirial quetion how loe the artifiial dioure ample are to the naturaliti ample. We ued Coh-Metrix 2.0 to analyze the four orpora. Eah onerational turn wa treated a a paragraph in the analye of the two tutoring orpora. erefore, a turn in a dialogue i analogou to a paragraph in an expoitory monologue. Means and standard deviations of the Coh-Metrix indices Tale 1 preent the mean and tandard deiation of the Coh-Metrix indie, egregated y the four orpora. Tale 2 preent a follow-up analyi that egregate tutor turn from tudent turn within the two tutoring orpo -ra. In order to ae whether the mean ignifiantly differ from eah other, one an ompute 95% onfidene interal around eah mean. e general formula i Mean ± [1.96 * SD / SQRT(N)]. For example, the mean numer of negation per 1000 word in AutoTutor dialogue i 11.5 and the tandard deiation (SD) i 6.2. e 95% onfidene interal would e 11.5 ± [1.96 * 6.2 / SQRT(213)]. at i, ore etween 10.6 and 12.4 are not ignifiantly different from the mean of 11.5. It follow that the mean ore for negation in textook (7.9) and textoid (6.8) are learly outide of the range for AutoTu -tor, wherea the mean ore for human tutoring (10.6) i within the range for AutoTutor. e 95% onfidene interal for the textook i 7.9 ± 1.2, or 6.7 to 9.1; the negation in the textoid are within thi range, ut not the AutoTutor dialogue. e 95% onfidene interal for textoid i 2.3 to 11.3, o the textook are within thi range, wherea the AutoTutor dialogue are outide of the range. T-tet an alo e mathematially deried from the mean and tandard deiation in thee tale. e et
idj (), , -
of t-tet would upport the following omparion on the inidene of negation: AutoTutor = human tutoring > textook = textoid. Our diuion of the data elow do not expliitly report inferential tatiti eaue the large numer of tatitial tet would e umerome. Howeer, our laim are upported y tatitial tet at alpha = .05 without adjutment for alpha inflation from multiple tet.
Simple measures of texts e top luter of indie in Tale 1 and 2 preent imple meaure of text. e expoitory monologue (textook and textoid) had a higher Fleh-Kinaid grade leel than the tutoring dialogue (AutoTutor and human tutoring). e higher grade leel an e attriuted to longer entene in the monologue eaue there were mall differene in yllale per word (reall that grade leel i aed on entene length and word length). e monologue alo had more entene in the paragraph than the tutoring eion had entene in the onera -tional turn. e flow of information in tutoring learly ha maller pakage of information (entene, para -graph) ditriuted oer more turn (i.e., paragraph) ompared with the expoitory monologue that are deigned to e read. Stated differently, tutoring i more fragmented and distributed than the dioure deigned for print. A more detailed analyi of the tutoring an e deried from the data in Tale 2. e AutoTutor dialogue wa more fragmented and ditriuted than the human tutoring. AutoTutor had omparatiely more turn, fewer entene per turn, and fewer word per entene. For oth type of tutoring, the ontriution of the tudent were muh horter than the tutor. Mot of the tudent turn were only one entene with 8 or 9 word. It wa the tutor who did mot of the talking in oth AutoTutor and human tutoring. Eduational reearh -
Arthur C. Graesser et a • l. Discourse cohesion in text and tutorial dialogue idj (), , - Table 1. Means and standard deviations for the measures of Coh-Metrix by physics corpora Textbook Textoids AutoTutor Human tutoring tutoring SIMPLE MEASURES OFTEXTS Number of texts 8 12 213 160 Total number of words in the t ext 5967 (2333) 177 (15) 913 (369) 406 (215) Total number of sentences in the t ext 329 (136) 14.4 (2.5) 104.6 (49.7) 36.9 (21.2) Total number of paragraphs/turns in the t ext 76.6 (27.7) 3.9 (1.08) 46.5 (22) 15.5 (11.8) Average words per sentenc e 18.2 (.86) 12.5 (1.97) 9.3 (1.73) 11.5 (2.83) Average sentences per paragraph/tur n 4.3 (.58) 4 (1.54) 2.3 (.17) 2.7 (.84) Average syllables per wor d 1.51 (.03) 1.45 (.13) 1.45 (.06) 1.43 (.08) Flesch-Kincaid Grade level (0-12 ) 9.3 (.46) 6.4 (1.55) 5.2 (.8) 5.8 (1.53) WORD LEVEL Logarithm of frequency of content words 2.15 (.05) 2.3 (.14) 2.27 (.11) 2.31 (.16) Incidence score of all connectiv es 69.3 (5.9) 71.1 (20.4) 59.9 (13.4) 71.5 (52.8) Incidence of positive causal connectiv es 9.4 (2.2) 12.4 (11.8) 11.9 (5.5) 18.4 (16.7) Incidence of negative causal connectiv es 1.35 (.52) 2.72 (4.35) .53 (.8) .36 (1.04) Incidence of positive additive connectiv es 22 (2.8) 29.5 (11.1) 22.5 (8.7) 22.7 (21.2) Incidence of negative additive connectiv es 10.7 (1.8) 6 (5.4) 3.9 (2.6) 6.1 (5.6) Incidence of positive temporal connectiv es 10.4 (2.9) 10.6 (8.9) 14.6 (6) 12.3 (11.8) Incidence of negative temporal connective s .29 (.29) .48 (1.66) .1 (.38) .56 (1.77) Incidence of all logical operators (and +if+or+cond+n eg) 38 (2.7) 32.6 (17.6) 34.6 (9.9) 36.9 (27.6) Incidence of conditionals in the te xt 5.19 (.83) 1.33 (2.41) 4.82 (3.03) 5.38 (4.26) Incidence of negations in the t ext 7.9 (1.7) 6.8 (7.9) 11.5 (6.2) 10.6 (8) SYNTAX Words before main verb of main clause in sentences 5.21 (.35) 3.56 (1.11) 2.52 (.64) 2.76 (1.18) Average number of modifiers per noun phr ase .93 (.07) .75 (.17) .87 (.07) .86 (.13) REFERENTIAL AND SEMANTIC COHESIO N Argument overlap of adjacent Sentences .66 (.03) .53 (.26) .24 (.07) .35 (.12) Stem overlap of adjacent sentenc es 64 (.03) .49 (.25) .23 (.07) .3 (.11) LSA cosine of adjacent sentence to sente nce .36 (.02) .28 (.12) .19 (.08) .21 (.11) LSA cosine of paragraph/turn to paragraph/tu rn .48 (.07) .45 (.18) .32 (.09) .26 (.16) Anaphor pronominal coreference of adjacent sentenc es .26 (.07) .29 (.21) .09 (.04) .21 (.1) Type-token ratio of all content wo rds .36 (.03) .73 (.09) .37 (.09) .57 (.1) SITUATION MODEL DIMENSION S Causal cohesion: Causal particles divided by causal verbs .33 (.08) .26 (.24) .26 (.12) .4 (.22) Intentional cohesion: Intentional particles / intentional actions 1.49 (.2) 1.04 (1.62) .58 (.29) .94 (.8) Temporal cohesion:Tense and aspect repetition scores . (.) . (.) . (.) . (.)
Arthur C. Graesser et a • l. Discourse cohesion in text and tutorial dialogue
Table 2. Means and standard deviations for students and tutor turns AutoTutor (Tutor Turns) SIMPLE MEASURES OFTEXTS Number of texts 213 Total number of words in the t ext 725 (329) Total number of sentences in the t ext 77.7 (38.8) Total number of paragraphs/turns in the t ext 23.4 (11) Average words per sentenc e 9.8 (1.77) Average sentences per paragraph/tur n 3.3 (.31) Average syllables per wor d 1.44 (.06) Flesch-Kincaid Grade level (0-1 2) 5.2 (.84) WORD LEVEL Logarithm of frequency of content words 2.29 (.11) Incidence score of all connectiv es 59.4 (14.2) Incidence of positive causal connectiv es 10.1 (5.6) Incidence of negative causal connectiv es .61 (.99) Incidence of positive additive connectiv es 22.9 (9.4) Incidence of negative additive connectiv es 3.9 (2.4) Incidence of positive temporal connectiv es 15.6 (6.7) Incidence of negative temporal connective s .0 (.0) Incidence of all logical operators (and +if+or+cond+n eg) 32.8 (11.2) ncidence of conditionals in the te xt 5.16 (3.01) Incidence of negations in the te xt 9.8 (6.9) SYNTAX Words before main verb of main clause in sentences 2.72 (.72) Average number of modifiers per noun phr ase .87 (.07) REFERENTIAL AND SEMANTIC COHESIO N Argument overlap of adjacent Sentences .24 (.07) Stem overlap of adjacent sentenc es .21 (.06) LSA cosine of adjacent sentence to sente nce .15 (.08) LSA cosine of paragraph/turn to paragraph/tu rn .44 (.11) Anaphor pronominal co-reference of adjacent sentenc es .09 (.05) Type-token ratio of all content wo rds .42 (.12) SITUATION MODEL DIMENSIONS Causal cohesion: Causal particles divided by causal verbs .21 (.11) Intentional cohesion: Intentional particles / intentional actions .53 (.27) Temporal cohesion:Tense and aspect repetition score s . (.)
idj (), , -
AutoTutor Human Human (Student Tutoring Tutoring Turns) (Tutor (Student Turns) Turns) 213 160 160 190 (91) 309 (153) 102 (92) 26.9 (11.7) 26.7 (14.1) 10.2 (7.9) 23.1 (11) 7.9 (5.9) 7.5 (5.9) 7.6 (3.04) 11.9 (3.04) 9.4 (5.13) 1.2 (.2) 3.9 (1.38) 1.4 (.56) 1.51 (.11) 1.44 (.07) 1.41 (.17) 5.2 (1.48) 6.0 (1.51) 4.9 (2.84) 2.2 (.17) 2.3 (.13) 2.36 (.3) 59 (21.9) 64.1 (19.7) 64.3 (57.2) 17.6 (11.4) 15.4 (8) 19.3 (25.3) .19 (.95) .29 (1.03) .45 (1.86) 19.8 (14.4) 20.5 (10.6) 18.1 (21.7) 4 (6.5) 5.4 (5.1) 7.2 (10.9) 10.6 (10) 11.2 (6.8) 11 (18) .43 (1.87) .29 (1) 1.05 (4.75) 40.4 (16.6) 31.1 (12.2) 44.6 (39.8) 3.27 (5.91) 5.36 (4.55) 3.76 (7.28) 18.4 (12.5) 7.5 (5.8) 24.5 (35.7) 1.92 (1.23) 2.91 (1.43) 2.06 (1.91) .85 (.18) .86 (.14) .68 (.32) .24 (.15) .37 (.14) .28 (.25) .25 (.16) .31 (.14) .23 (.25) .23 (.09) .19 (.09) .23 (.15) .20 (.08) .34 (.16) .25 (.17) .10 (.08) .23 (.12) .17 (.19) .49 (.12) .62 (.1) .76 (.16) .47 (.7) .36 (.23) .39 (.41) .84 (1.25) .82 (.74) .78 (1.12) . (.) . (.) . (.)
Arthur C. Graesser et a • l. Discourse cohesion in text and tutorial dialogue
er enourage atie learning on the part of the tudent, with attempt to get the tudent to do the talk and ation. Howeer, thi i a hallenge een in one-on-one tutoring. Word-level indices e word in the four type of dioure did not appre -ialy ary in word frequeny ut differene did emerge in onnetie, onditional, and negation. e oerall inidene of onnetie wa lower in AutoTutor than the other three genre, whih were approximately the ame. e ditriution of uategorie of onnetie differed among the dioure type. Howeer, it i diffi -ult to diern any imple piture from the data. Differene that emerged etween AutoTutor and human tutoring appear in Tale 2. e tudent learning from AutoTutor had fewer negatie onnetie, nega -tion, logial operator, and onditional expreion, ut approximately the ame numer of poitie aual, addi -tie, and temporal onnetie. Although thi ugget that the human tutor extrated more omplex analytial ontent from the tudent than did AutoTutor, the ditri -ution of word ategorie wa ery imilar for the tutor turn in AutoTutor and the human tutor. i upport the laim that the automated tutor did a reaonale jo imulating the human tutor, at leat from the perpe -tie of the ditriution of word ategorie. Syntax e yntati ompoition of entene ytematially differed among the four type of dioure. e expoi -tory monologue had more omplex yntax than the tutoring dialogue when we examined the mean numer of word efore the main er of the main laue, whih reflet a greater load on working memory. e textook learly had the highet ore on thi yntati index. Within the tutoring dioure, the tudent ontriution
idj (), , -
were not different on thi index for AutoTutor eru human tutor; the tutor ontriution were alo not ignifiantly different for AutoTutor eru human tutor. e other meaure of yntati omplexity, namely the mean numer of modifier per noun-phrae, wa not remarkaly different among the dioure type. Referential and semantic cohesion Referential and emanti oheion wa onitently higher for the expoitory monologue than the tutor -ing dialogue when we examined argument oerlap, tem oerlap, and LSA ore. erefore, in addition to tutoring eing more fragmented and ditriuted, the dioure alo ha lower oheion on thee referential and emanti indie. When the two type of expoi -tory monologue were ompared, oheion wa higher for the textook than the textoid. Unfortunately, it i diffiult to interpret the anaphor pronominal referene index eaue the meaure onfound the inidene of pronoun and the likelihood that the pronoun refer -ent an e reoled; additional analye will need to e onduted to differentiate thee two omponent. Neerthele, the ore were higher for expoitory monologue than the tutoring dialogue. e type-token index of lexial dierity howed extremely high ore for the textoid, followed y human tutoring, and the lowet for the textook and AutoTutor. erefore, there i more redundany in the ontent word of the textook and AutoTutor. e textoid are ery paked with information. Apparently experimental pyhologit fill their timulu text with a large amount of new information, muh more than the profeional writer of textook. In-depth analye of the tutoring eion unoered a few informatie reult. e tudent ontriution in AutoTutor had lower argument oerlap, LSA-turn imilarity, anaphor pronominal referene, and lexi -
