Murphy: Validity

“Validity is a complicated and changing theoretical construct that has evolved
significantly over the past fifty years. While traditional positivistic conceptions of
validity tended to characterize it as one or more characteristics of a test that could
be definitively determined, contemporary theorists have reconstructed the con-
cept. In a now-classic chapter on validity, Samuel Messick (1989a) extends the
definition of validity to focus on the interpretation of scores and to encompass
social consequences:

Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and
actions based on test scores and other modes of assessment. (p. 13) “

Praxis (as a research methodology): Sullivan, Francis

“this study does not fall into the ‘conventional’ categories of ‘method-driven’ or ‘problem-driven’ research (Sullivan & Porter, 1993, pp. 229-30). It did not begin with choosing a standard method to observe practice, nor did it begin with a research question drawn from that practice or from a theory. Rather, the study is better described as ‘praxis,’ in which the methodology functions  ‘in a middle ground between theory and practice, as a heuristic set of filters…for both theory and practice’ (Sullivan & Porter, 1993, p. 229).”

• Citation Link (Sullivan)

Think-aloud protocols: Prior

“concurrent, or think-aloud, protocols…There are questions about how thinking aloud affects the writing process. There also have been questions about the value of the cognitive models typically associated with this line of inquiry. think-aloud protocols have usually been attempted only in laboratory conditions while there has been an intense interest in studies of writing in naturalistic conditions…attention to composing in naturalistic conditions also suggested that many of the key processes were social as well as cognitive…concurrent protocols for the first time began to crack open the notion of “writing,” to reveal the complex, fine-grained, and diverse nature of the acts that are combined under that label,” (180).

• Citation Link (Prior)

Intertextual tracing: Prior

Intertextual tracing is tracing the detectable influences on a text by gathering all of the initiating texts, influencing texts, and source texts and analyzing the effect that each had on the version of the text being analyzed (usually the final version). Examples of things that might be traced are:

• Initiating texts
  • assignment sheets
• Source texts
  • drafts
  • annotated drafts
  • feedback in the form of memos
  • emails
  • handwritten notes
  • verbal comments
• Influencing texts
  • Sources such as books, articles or television shows mentioned as influences by the author

All of these texts form the contextual record of the text.

Methods of intertextual tracing include:

• “Intertextual analysis of…exchanges of talk and text” (179)
• “tracing language across multiple drafts” (175)

—

Paraphrased from pg. 173-179 , boldface terms were boldfaced at some point in the original.

• Citation Link (Prior)

Tracing Writing Processes: Prior

One of the key steps for researchers in tracing writing processes is collecting and keeping track of the textual inscriptions themselves…the more relevant texts you are able to collect, the fuller the view you can develop of the process and its contexts…As a practical matter, it is important to ask participants what the texts are and to add explanatory labels for yourself that include when the text was given to you, what it is, who wrote it, perhaps who wrote on it (it is not unusual for writing in different ink or pencil on a text to mark different writers—different respondents and authors—or different episodes of composing),” (172).

Initiating text: Prior

Texts often respond to other texts that may be treated as initiators. An initiating text does not simply control what follows. It has to go through processes of interpretation and negotiation.

Prior: Authorship

“Understanding how people represent the process and authorship and understanding how a text Is actually produced in practice are related but distinct issues,” (171).

Authorship: Prior

“Goffman (1981) analyzed the everyday notion of the speaker/writer, suggesting that three roles are typically collapsed within that term: the animator, who actually utters/inscribes the words; the author, who selects the sentiments and words; and the principal, whose positions are being represented in the words..This simple division suggests that tracing the writing process also means tracing a structure of participation, of examining who is involved in making the text and in what ways,” (170).

Text: Prior

“When people talk about “text,” there are several different senses that we should be aware of to avoid confusion. Text sometimes means a unique material Inscription…Text is sometimes taken more expansively, to refer as well to the various mental and oral representations of the material texts, regardless of whether they are ever written out… Sometimes, all of these material inscriptions (and perhaps the ideational representations) are idealized in retrospect as “the text,” uniting all moments in the production under a unified label…How we understand text—as a unique material object, as a representation regardless
of medium (including thought and speech), as the ideal that unifies varied acts and objects in a process—is not the issue; the issue is being aware of the different senses, not shifting from one to the other unconsciously,” (169).

• Citation Link (Prior)

Methodological Procedure: Huckin

1. Pose a Research Question. Content analysis proceeds best if the researcher has a good research question to start with, one that: (a) addresses a topic or issue of likely interest to fellow members of the researcher’s discourse community, and (b) constitutes a novel claim about this topic or issue…A research question becomes a hypothesis if it is sufficiently explicit to be tested and, if appropriate, refuted by empirical data. A deductive study requires a hypothesis, an Inductive study does not.
2. Define the Appropriate Construct(s). A good research question will focus on one or more general concepts such as “writing quality” or “bias.” Such a concept is the construct of interest for that study (MacNealy, 1999). If the research question is worded in such a way that the construct of interest is not obvious, the researcher must make a special effort to clarify it…
3. Select an Appropriate Text or Body of Texts as the Study Corpus. An appropriate study corpus is one that will provide a good test of the research question, which means (a) that all the texts included in the corpus correspond in some clear fashion to the research question, and (b) that these corpus texts are representative of some identifiable, larger body of texts…A project aimed at identifying only a small set of surface variables, for example, requires less time and effort per text than a project concerned with deeper, more complex variables; thus, a project of the former type should use a larger corpus than a project of the latter type.
4. Determine Appropriate Units of Analysis (Text Features), UsingMultiple Raters if Possible. The units of analysis, or text features, to be quantified for content analysis must be those that emerge logically from the research question. They must have a direct bearing on the question, in the sense that the incidence of a feature will constitute either direct support or direct nonsupport for it. The categorization of these units of analysis should be readily identifiable and non-overlapping, so as to be codable with some degree of reliability…Working collaboratively with one or more other investigators can help resolve uncertainties and produce a sample list of reference terms…Once the units of analysis have been determined, two or more investigators should independently sort at least 10% of the data into categories, and then the respective sortings should be compared for interrater reliability.
5. Gather Data. If the units of analysis have been narrowly defined in Step 4, data gathering should be a relatively straightforward matter of identifying and counting.
6. Interpret the Findings. The final step is to analyze the data against the research question, that is, to interpret the findings…In an inductive study, the research question is only tentative and may be in need of revision if it does not conform to the data. Indeed, the goal of such a study should be to construct and refine a research question that might stand up to a more deductive, follow-up study.