Loading...

Observing «and» Analyzing Communication Behavior

by John A. Courtright (Author)
©2014 Textbook XIV, 485 Pages

Summary

This is a book about communication behavior: how we conceptualize it, observe it, measure it, and analyze it.
The 1970s and 1980s were times when communication behavior was a primary interest of many communication scholars. The aim of this book is to reignite some interest in and passion about how human communication behavior should be studied. It presents the best advice, techniques, cautions, and controversies from the 1970s and 1980s and then updates them. Several chapters also introduce statistical methods and procedures to allow readers to analyze behavioral data.
This book is a useful resource for communication scholars and graduate students to guide their study of communication behavior.

Table Of Contents

  • Cover
  • Title
  • Copyright
  • About the author(s)/editor(s)
  • About the book
  • This eBook can be cited
  • Table of Contents
  • 1 Preliminaries and Promises
  • Defining Communication Behavior
  • Residual Artifacts of Communication Behavior
  • Outcomes of Communication Behavior
  • Some Controversial Assertions
  • Overview of the Book
  • Moving On
  • Notes
  • 2 Unitizing Communication Behavior
  • A Small Taxonomy of Behavior
  • No Units to be Found
  • Definitions
  • Types of Units
  • Acts as Units
  • Grammatical or Syntactic Units
  • Thought Units
  • Themes
  • Time
  • Codes as Unitizers
  • Turns Versus Thoughts—A Comment
  • Conclusion
  • Notes
  • 3 Coding Communication Behavior
  • Selecting or Creating a Coding System
  • The Role of Theory
  • Select or Create?
  • Only If You Have To
  • Other Considerations
  • Some Practical Concerns
  • Video Recording
  • Transcribe and Check
  • Conclusion
  • Notes
  • 4 Observing and Recording Communication Behavior
  • Conceptualizing the Target Behavior(s)
  • Variable of Interest
  • Sample or Sign?
  • How Many Behaviors?
  • Pulling It Together
  • Dimensions and Measures of Communication Behavior
  • Frequency
  • Latency
  • Intensity
  • Pattern
  • Duration
  • Derived Ratios as Descriptive Indices
  • Time-Sampling
  • Continuous Observation
  • Intrasession Interval Sampling
  • Conclusion
  • Notes
  • 5 Common Issues in Behavioral Observation
  • Observational Setting
  • The Coding Manual
  • Selection of Observers
  • A Word at the Start
  • Observer Training
  • Orientation Session
  • Stage 1. The Coding Manual
  • Stage 2. Coding Simple Examples
  • Stage 3. Code Realistic Examples
  • Stage 4. Practice in the Setting
  • Observer Monitoring
  • Sources of Bias in Observational Research
  • Observer Drift
  • Expectancy Effects
  • Reactivity
  • Observer Cheating
  • Reconciling Observer Discrepancies
  • Arbitrary Selection of Codes
  • Decision by Consensus
  • Employing Additional Observers
  • Conclusion
  • Notes
  • 6 Reliability of Unitizing and Categorizing
  • An Overview of Classical Test Theory
  • True Scores and Error Scores
  • Parallel Tests
  • Split-Half Reliability
  • Reliability as Internal Consistency
  • Reliability of Observational Data
  • Conceptual Definition of Observational Reliability
  • Operational Definition of Observational Reliability
  • Unitizing Reliability
  • Components of Unitizing Reliability
  • Coefficients of Unitizing Reliability
  • Categorizing Reliability
  • How Reliable is Reliable Enough?
  • Scott’s π
  • Cohen’s κ
  • Extensions of κ
  • Category-by-Category Reliability
  • An Alternative Approach
  • A Troublesome Practice
  • Notes
  • 7 Reliability for Quantitative Observational Data
  • Preliminary Decisions
  • Are the Observers a Random or a Fixed Facet?
  • Are the Data Matched or Unmatched?
  • Will the Final Data be a Single or a Composite Score?
  • Reliability as Consistency or Agreement?
  • Intraclass Correlation Coefficients
  • Reliability as Consistency
  • Reliability as Agreement
  • Generalizability Theory
  • Universe of Admissible Observations
  • G Versus D Studies
  • Crossed Versus Nested Facets
  • G Study Variance Components for a p × r Design
  • D Studies for a p × R Design
  • Error Variances
  • An Illustrative Example
  • Single Facet Nested Designs, r : p
  • Multifaceted G and D Studies
  • Conclusion
  • Notes
  • 8 Validity
  • History of the Study of Validation
  • Before the Beginning
  • All Validity Is Created Equal
  • Validity of Inferences, Not Tests
  • The Unitary Model of Validity
  • Theory Versus Practice
  • Taking Stock
  • Validity as Argument
  • Form of the Validity Argument
  • The Interpretative Argument
  • Toulmin’s Model of Inference
  • Validity Argument for Relational Control Coding System
  • A Better Approach
  • What’s Missing?
  • Conclusion
  • Notes
  • 9 Classic Analytic Procedures
  • Markov Chain Analysis
  • An Example of a Markov Chain
  • Markov Chain Assumptions
  • The Order Assumption
  • The Stationarity Assumption
  • The Homogeneity Assumption
  • A Final Word on Markov Chain Analysis
  • Lag Sequential Analysis
  • Conducting the Analysis
  • Next Steps
  • Notes
  • Appendix 9.1 Test for First- Versus Second–Order Markov Process
  • The Pearson X2
  • The LRX2
  • Appendix 9.2 Test of the Stationarity Assumption
  • Pearson X2
  • The LRX2
  • Appendix 9.3 Test of Homogeneity Assumption
  • Pearson X2
  • The LRX2
  • 10 Log-Linear Models of Nominal Observational Data
  • Hypotheses and Sampling
  • Analyzing Two-Dimensional Tables
  • Some Important Notation
  • The Log-Linear Model
  • The Likelihood Ratio Test Statistic
  • The Hierarchy Principle
  • Residual Analysis
  • Analyzing Multidimensional Tables
  • Complete Independence
  • Two Variables Independent of a Third
  • Conditional Independence
  • Homogeneous Association
  • Saturated Model
  • An Example
  • Log-Linear Models and Markov Chains
  • The Order Assumption
  • The Homogeneity Assumption
  • The Stationarity Assumption
  • Final Thoughts About Markov Chains
  • Log-Linear Models and Lag Sequential Analysis
  • So…What’s Different?
  • Wrapping Up
  • Notes
  • 11 Linear Models of Categorical Data
  • The GSK Procedure
  • Defining the Response Functions
  • Defining The Design Matrix
  • Estimating and Testing the GSK Model
  • Estimation
  • Testing
  • Research Examples
  • Example 1: Ellis (1979)
  • Example 2: Courtright et al. (1990)
  • Example 3: Fairhurst et al. (1995)
  • Conclusion
  • Notes
  • 12 Analyzing Counts
  • The Nature of Counts
  • Distributions of Counted Data
  • Binomial Distribution
  • Negative Binomial Distribution
  • Multinomial Distribution
  • Poisson Distribution
  • Logistic Regression—Dichotomous Response Categories
  • Odds and Odds Ratios
  • The Logit
  • Logistic Versus OLS Regression
  • Logistic Regression—Multinomial Response Categories
  • Ordinal Responses
  • Nominal Responses
  • Poisson Regression
  • Modeling the Offset
  • The Dispersion Parameter
  • Groupthink Example
  • Some Remedies for Overdispersion
  • Nonverbal Example—Underdispersion
  • Conclusion
  • Notes
  • 13 Hierarchical Generalized Linear Models
  • A Brief Overview of Hierarchical Linear Modeling
  • Hierarchical Generalized Linear Models (HGLMs)
  • Remember the Link Function
  • Centering
  • Example Analyses
  • Binomial HGLM
  • Poisson HGLM
  • Conclusion
  • Notes
  • 14 Reversing the Sides of the = Sign
  • Deconstructing the Transition Matrix
  • The Disaggregation Approach
  • Continuous Indices Approach
  • A Brief Note About Transformations
  • Conclusion
  • Notes
  • References
  • Author Index
  • Subject Index

| 1 →

Chapter 1

Preliminaries and Promises

“One of the great ironies of the field of communication is that we so seldom study our namesake.” (Folger, Hewes, & Poole, 1984)

Plus ça change, plus c’est la meme chose.” (Karr, 1849)

A large and important segment of what we call human communication is comprised of overt, observable behavior. There is little doubt and no argument in this book that such communication behavior is frequently preceded by unobservable, high-level psychological processes. Similarly, communication behavior often produces unobservable psychological reactions in both the communicator and those who might be observing the communication behavior.

To be clear, however, those psychological processes will not be addressed in this book. They are interesting and important, but they have been examined in detail in many other scholarly venues. This is a book about communication behavior: how we conceptualize it, observe it, measure it, and analyze it. As a consequence, this book is a sort of a “way-back machine,” going way back to the 1980s and before. During those years, communication journals routinely published research in which overt communication behavior was the primary variable of interest. There were numerous books and articles on how to appropriately employ some procedure, methodology, or technique. There were controversies and disagreements, criticisms and rejoinders, not to mention a good deal of solid scholarship.

Where have you gone Joe DiMaggio? If one sets aside for the duration of this argument the excellent research in mass communication using content analytic techniques, the halcyon days of communication behavior have seemingly disappeared. That is not a good thing. If the only purpose of this book was to reminisce about the good old days, the entire enterprise could be wrapped up in a couple of additional paragraphs. That is not the case. This book’s purpose is very much larger than that. The failure of the discipline of Communication to continue and to expand the study of communication behavior has the very real potential to relegate ← 1 | 2 → it to a perpetual second class status in the academic world.1 Someone has to sound the warning.

Condit (2009, p. 9) succinctly expresses this concern when she writes, “Communication studies tried to skip the stage of rigorous observation….The consequence is that we don’t really have much in the way of descriptions of what communication looks like, either at the social or individual levels.” One could go even further. Virtually all of the theories of psychological processes that have emerged in the last several decades have embraced—to a greater or lesser extent—a common assumption. They assume that to some extent and for some reason these unobservable psychological operations influence the overt communication behavior that is actually emitted.2 How does anyone know that assumption is true?

This is more than a loose conceptual thread on the back hem of an otherwise elegantly woven theoretical garment. To the contrary, this is theoretical ground zero. This is where the claim is made that what goes on inside a person’s head has an effect on how he or she behaves. But if researchers are not able to validly conceptualize, systematically observe, and reliably measure that behavior, they will never be able to validate such claims in a convincing manner.

Accordingly, one of the purposes of this book is to nudge communication research back in time a bit; to reignite some interest in and passion about how human communication behavior should be studied. A second purpose is to present once again the best advice, techniques, cautions, and controversies from the 1970s and ’80s. The idea is to dust them off, update them a bit, and collect them in one scholarly container, thus simplifying things for the next generation of behavioral researchers in communication.

Finally, the chapters that follow will intersperse those golden oldies with the best, most exciting work of numerous other disciplines that have not put the study of behavior on the back burner. Areas of study such as clinical psychology, child development, educational measurement, and even personality psychology have continued to study actual behavior, and there is much communication scholars can learn from their advances. Also included from time to time will be some interesting work from ethology and other animal studies.

The overall goal, however, is to produce a monograph that communication scholars and graduate students can actually use to ← 2 | 3 → guide their study of communication behavior. As a result, the intended audience is decidedly not the most savvy and sophisticated methodologists in the discipline of Communication. Those several dozen people are fine on their own. In contrast, the intention is to make each major topic addressed in this book be technically accurate, yet as understandable as possible. Hopefully, researchers will actually use what is written here.

Don’t be confused: no one will find the subsequent chapters to be light bedtime reading, but neither will they be reminiscent of Psychometrika. There will be plenty of references for readers who want fuller, more rigorous, or more sophisticated treatments. This book is intended as a guide, and no guide is successful if it overwhelms with detail those who are new to the journey.

Defining Communication Behavior

The fact that this book is devoted exclusively to the study of communication behavior does not and should not imply that all behavior is communicative. To make such a claim would be absurd. Nevertheless, there must be some way to distinguish what qualifies as communication behavior (and therefore is worthy of investigation) and what does not. Without some conceptual boundaries, scholars end up studying everything—or nothing.

Over the last several decades, several articles and essays have attempted to define what attributes and characteristics must exist for some phenomenon to be called “communication” (see Andersen, 1991; Bavelas, 1990; Dance, 1970; Miller, 1966; Motley, 1990, 1991). There is no doubt that the authors of those thought pieces spent much time and considerable intellectual energy developing their finely nuanced positions. This book will devote no space to either. As a result, communication behavior will be defined as whatever behavior a communication researcher decides to study. No judgments will be made about whether that behavior is “really” communication or not. The goal here is to study communication behavior, not define it. Accordingly, communication behavior is whatever the reader or the author or any other communication researcher says it is.

Would-be behavioral scholars should not become too sanguine, however. Should that same researcher decide to submit his or her work for publication, such open-minded tolerance is unlikely. Editors and manuscript reviewers will almost certainly have opinions ← 3 | 4 → about what is and is not legitimately “communication,” and they will not hesitate to express them. If one’s manuscript falls on the wrong side of those opinions, it will suffer an inglorious fate. That said, none of the examples provided herein will likely offend anyone’s sense of disciplinary propriety. I know communication when I see it, and so do you. And as for those journal Editors …well, they need to get out more. On a more serious note, two types of phenomena as communication behaviors will be included that require additional explanation. Their exclusion would make this book more than slightly incomplete.

Residual Artifacts of Communication Behavior

Many times scholars cannot or do not observe a person engaging in communication behavior, but instead have available for examination the left-over or residual artifact of that behavior; e.g., written documents, highlighted reading materials, computer usage logs, caller ID logs, etc. And do not forget artifacts such as television programming, political commercials, or newspaper coverage, or the legitimacy of research in mass communication will vanish in a definitional poof.

The fact that an investigator did not actually observe the creation of these artifacts cannot be construed to mean that no communication behavior took place. Such artifacts have been studied for decades, frequently under the rubric of content analysis, but with other methodologies as well. They are fundamentally different from, say, interpersonal interaction, and they have their own set of problems and promises to discuss. Discuss them we will.

Outcomes of Communication Behavior

Outcomes are different than artifacts. They are not left behind signs that a behavior occurred, but rather the effect or result of communication behavior having occurred. Both artifacts and outcomes exist because communication behavior has occurred, but they are demonstrably different phenomena and deserve two separate treatments. Outcomes include things such as the cessation of alcohol abuse, contracting or avoiding HIV (because of a prevention campaign), loss of weight, getting divorced, or donating money to a charity. ← 4 | 5 →

As the term “outcome” implies, this category of behavior results as a consequence of something the communicator said or did, and is almost always emitted by someone other than the original communicator. Although this is a major shift in conceptualization about what we might want to study, the procedures and techniques used to study these outcomes are not substantially different from those we would use to study more traditional communication behavior.

Some Controversial Assertions

A good deal of this book will ultimately be a straightforward presentation of various empirical procedures and techniques. Technically accurate, one hopes, but perhaps not scintillating prose. Even some of the controversies that will be introduced (e.g., time sampling) are really other disciplines’ controversies. They have fought and largely settled them in the past. Accordingly, their inclusion will merely be a review the history of those battles.

In recognition of this inevitable fact, the discussion of communication behavior will begin with two assertions that will likely generate controversy of their own. They are stated as if they have the status of “fact,” although they have not been articulated fully in the literature on human communication. Although these assertions will not be explicitly restated each and every time they are relevant in later chapters, the reader should be aware that they are pertinent throughout this book.

First, in the jargon of parametric statistics, virtually all communication behavior is a random effect. Stated differently, whenever we observe communication behavior, we are sampling a small smidgen from an infinitely large universe of possibilities. Communication behavior (perhaps all behavior, but that is not my concern) is decidedly not a fixed effect in which we have purposely included all possible levels of the variable in our analysis. Why then do communication researchers continue to analyze communication behavior as if it were? If this assertion is accurate, we must contemplate completely changing how we approach communication behavior as data, particularly if we want to put communication behavior on the left-hand instead of the right-hand side of the equal sign.3

Second, virtually all communication behavior is nested within some larger social or cultural entity. People communicate while residing within dyads, groups, families, classrooms, homes, television ← 5 | 6 → networks, newspapers, and the list goes on and on. Moreover, these larger (the technical term is “higher level”) units almost certainly have some effect on how the individual (the “lower level” unit) communicates. Moreover, different higher level units will have different effects. If communication scholars are going to produce theoretically meaningful explanations of how individuals behave, information from the higher level unit must be included in our analyses.

Not surprisingly, this will require spending some time discussing multilevel modeling and its applications to communication behavior. Those already familiar with MLM are also aware that there is a large and sophisticated body of literature surrounding this statistical procedure (see Raudenbush & Bryk, 2002). In contrast to that technical sophistication, the treatment of MLM in this book will be more introductory, with numerous references to more technical treatments.

Overview of the Book

The first seven chapters of this book are devoted to observing, measuring, and assessing the quality of observational communication behavior. Two fundamentally different approaches to this observational enterprise are compared and contrasted throughout these chapters. The first of these will be termed the unitize and categorize approach. This will likely be the approach most familiar to empirical scholars in communication. Briefly, a continuous stream of communication behavior is segmented into conceptually meaningful units, and each unit is categorized by applying a welldeveloped coding system. This approach dominated observational research in communication during the 1970s and ’80s (see, for example, Ellis, 1979).

The second approach to be presented is fundamentally different and will be referred to as the observe and record approach. In this research paradigm, no units or categories are used. Instead, a theoretically meaningful behavior (or behaviors) is specified as the target behavior of interest. Subjects are then observed for a specified length of time and the frequency and perhaps the duration of the target behavior(s) is recorded. Although this approach has not been employed often in the study of human communication, it represents the primary approach to behavioral observation in many other disciplines, e.g., behavioral assessment, clinical psychology, ← 6 | 7 → and educational measurement. The observe and record approach is fully developed in these other disciplines and has the potential to be extremely useful in the empirical study of human communication.

Chapter 2 is devoted to issues surrounding “unitizing.” How do we reliably segment a continuous stream of behavior into discrete units that can be labeled or categorized? How many different types of units are there, and what are their strengths and weaknesses? When is unitizing behavior going to be a potentially troublesome concern, and when are units almost inherent in the behavior a researcher chooses to study (e.g., editorial cartoons)?

Chapter 3 will deal with the categorizing or coding of individual units of communication behavior. Decisions about coding schemes and how they will be administered are crucial to behavioral research in many disciplines, but especially so in Communication. Moreover, they are important to communication scholars who would investigate interpersonal interaction (see Folger, Hewes, & Poole, 1984), as well as those who would use content analysis to study mediated communication (see Krippendorff, 2004). Both conceptual issues and some pragmatic “how-to” issues will be presented in this chapter.

Chapter 4 will provide a lengthy presentation of the observe and record approach. How should the target behavior of interest be conceptualized? Will a single behavior suffice or should multiple behaviors be recorded and aggregated? What types of fundamental measures of these behaviors should be recorded? What other indices can be derived from these fundamental measures? These and other similar questions will be addressed.

Chapter 5 will discuss topics that are common to both of these approaches. These include the selection of a setting for observation, the selection and training of observers, the preparation of a coding manual, sources of potential bias in the observational process, and perhaps the most overlooked topic, methods for resolving observer disagreements about what was observed. For reasons that may already be apparent from the topics just mentioned, this chapter assumes a much more pragmatic tone than many of the other chapters in this book.

Chapter 6 and Chapter 7 both involve the topic of reliability of observations. Specifically, Chapter 6 speaks to the reliability of observational data that are nominal or ordinal in their level of measurement. ← 7 | 8 → In addition, the reliability of unitizing procedures is also discussed in this chapter. In contrast, Chapter 7 addresses the topic of the reliability of data that are interval or ratio in their level of measurement. These would include data obtained from rating scales, as well as frequency and durational data from the observe and record approach. A discussion of the types and uses of intraclass correlations begins this chapter. The second half presents an overview of generalizability theory.

Chapter 8 takes a completely different tact on what many would consider an uninteresting aspect of research, namely, the validity of observational data. Research and theory on the validity of empirical data have taken some twists and turns over the last several decades, and many communication scholars have not stayed abreast of these developments. Those scholars, therefore, may well be surprised to learn that contemporary thoughts about validity define that concept squarely as a communicative process. It’s time to pay attention again.

Chapter 9 is the first of several chapters that are devoted to presenting various methods for analyzing behavioral observation data. This chapter discusses two classic analytic procedures: Markov chain analysis and lag sequential analysis. This is a lengthy and detailed chapter, including three detailed appendices. The reason for this length is that no major statistical package (e.g., SAS, SPSS) incorporates these analyses among its procedures. Although some elementary programming can trick the packages into producing the basic transition matrices, the tests of the assumptions of order, homogeneity, and stationarity are nowhere to be found. Consequently, Chapter 9 demonstrates how these tests can be conducted with nothing more than a spreadsheet and a basic understanding of the formulas for the tests.

Log linear modeling is the focus of Chapter 10. The fundamentals of log linear models are introduced, and examples are provided. In addition, this chapter demonstrates how log linear modeling can perform the same three tests of Markov chain assumptions that were discussed in Chapter 9. Finally, lag sequential analyses are again performed and tested, but in this instance using log linear models. The emphasis of this chapter is on the utility of these procedures for modeling all types of categorical data.

Chapter 11 provides a relatively in-depth discussion of how the method of weighted least-squares can be used to analyze categorical ← 8 | 9 → data. This approach was introduced by Grizzle, Starmer, and Koch (1969) and is commonly referred to as the GSK approach. The GSK approach is used to analyze combinations or configurations of probabilities using ANOVA-like nominally coded predictor variables. These combinations certainly include lag 1 or higher transition matrices, thus allowing a researcher to ask whether such matrices produced by two or more subgroups are significantly different. Examples from three different observational data sets are discussed at length.

Chapters 9 through 12 are focused primarily on behavioral communication data gathered using the unitize and categorize approach. In contrast, Chapters 12 and 13 are devoted to the analysis of data obtained by the observe and record approach. Chapter 12 focuses on the analysis of “counts”; i.e., the number of times a target behavior occurs during a specified period of time. Special attention to such data is required because counted data are most often not normally distributed and, thus, violate several assumptions associated with ordinary least-squares analyses. Instead, counted data usually are distributed as binomial, negative binomial, multinomial, or Poisson.

An entire family of analytic procedures known as generalized linear models is required to analyze such data. Chapter 12 introduces the characteristics of these distributions, as well as the generalized linear model. Four different examples are presented to illustrate the analysis of communication data that are distributed as binomial, multinomial, and Poisson. A discussion of both overdispersion and underdispersion is also presented in this chapter.

Chapter 13 combines the distributions discussed in Chapter 12 with the use of multilevel modeling to introduce Hierarchical Generalized Linear Models (HGLMs). A brief introduction to multilevel modeling is followed by a discussion and set of recommendations about the centering of the predictor variables. Two lengthy examples complete the chapter. The first demonstrates the analysis of a binomially distributed random intercept model. The second examines data distributed as Poisson with a random slopes model. Guidelines, suggestions, and most importantly cautions are found throughout the chapter.

Finally, Chapter 14 turns things around a bit by suggesting that observations of communication behavior be used as explanatory variables rather than the dependent variable. This involves ← 9 | 10 → disaggregating the transition matrix in individual probabilities or, alternatively, attempting to transform nominal codes into a relevant ordinal dimension. The conceptual underpinnings of these approaches are discussed, and the computer code for actually performing these manipulations is presented.

Moving On

Details

Pages
XIV, 485
Year
2014
ISBN (PDF)
9781453912454
ISBN (ePUB)
9781454197522
ISBN (MOBI)
9781454197515
ISBN (Hardcover)
9781433124167
DOI
10.3726/978-1-4539-1245-4
Language
English
Publication date
2015 (September)
Keywords
human communication advice controversies
Published
New York, Bern, Berlin, Bruxelles, Frankfurt am Main, Oxford, Wien, 2014. 485 pp.

Biographical notes

John A. Courtright (Author)

John A. Courtright (PhD, The University of Iowa) is Professor in the Department of Communication at the University of Delaware. He has been the editor of Communication Quarterly (1998–2001) and is currently the editor of Human Communication Research.

Previous

Title: Observing «and» Analyzing Communication Behavior
book preview page numper 1
book preview page numper 2
book preview page numper 3
book preview page numper 4
book preview page numper 5
book preview page numper 6
book preview page numper 7
book preview page numper 8
book preview page numper 9
book preview page numper 10
book preview page numper 11
book preview page numper 12
book preview page numper 13
book preview page numper 14
book preview page numper 15
book preview page numper 16
book preview page numper 17
book preview page numper 18
book preview page numper 19
book preview page numper 20
book preview page numper 21
book preview page numper 22
book preview page numper 23
book preview page numper 24
book preview page numper 25
book preview page numper 26
book preview page numper 27
book preview page numper 28
book preview page numper 29
book preview page numper 30
book preview page numper 31
book preview page numper 32
book preview page numper 33
book preview page numper 34
book preview page numper 35
book preview page numper 36
book preview page numper 37
book preview page numper 38
book preview page numper 39
book preview page numper 40
501 pages