• Methods of Data Collection
• Empirical Methods
• User Centered Approach
  • interviews
  • obvservations
  • audio and video taping
  • ethnography
  • action research
  • questionnaires
  • experiments
  • interpretive approaches
  • criteria to compare methods
  • Hawethorne effect
  • ethics

methods of data collection

Obtrusive

• participant observation
• interviews
• questionnaires (beliefs/attitudes)
• diaries (times and events)
• observation
• "think aloud" protocols
• audio/video recording
• physiological traces

Unobtrusive

• gaze or eye movement traces
• system logs
• (hidden) observation
• audio/video recording
• archives

Many of these methods have both qualitative and quantitative variants.

 

empirical methods

• Field studies - observe naturally occurring systems
• Sample surveys
• Interpretive methods - case studies, ethnography, action research, etc.
• Experiments and quasi-experiments - observe under controlled conditions
• Modeling
• Theory

During early stages of design, we are mostly concerned with the first three types of empirical methods because these are most appropriate for the intial stages of inquiry. Later, during evaluation, we often employ the others too.

Runkel & McGrath have proposed a four-quadrant taxonomy of empirical methods with two axes, obtrusvie vs. unobtrusive and abstract vs. concrete. They further add three different criteria we might have for our methods, generalizability, precision, and realism. Their taxonomy can be represented graphically as follows:

user centered approach

All of our designs must be based on users. We gain knowledge of the user and the user's needs by observation.

Interviews

• structured vs. unstructured
• components
  • purpose
  • enumeration of activities ("what")
  • work methods ("how")
  • tracing connections to other activities
  • performance issues
• less structure allows an interviewer to identify exceptions and domain knowledge by "steering" the interview

Observations

• no opportunity to "steer" the observations
• audio or video recording usually done
• transciption (to produce a written "protocol") take lots of time
• concurrent verbal accounts, or "think aloud" protocols are useful
• "co-discovery learning" solicits concurrent verbal accounts by having subjects work together
• passive observation vs. action research

Audio and Video Taping

• requires the right equipment
• lighting, sound, and cinematography are important
• multiple cameras, multiple mics, multiple recorders
• often a typewritten transcription or protocol is made
• transcription, logging and annotation takes a lot of time
• MACSHAPPA is a Macintosh-based tools for analyzing time lines
• NUD*IST is a PC-based tool for analyzing transcripts

Ethnography

• classic use is in sociology and anthropology
• combines interviews and observations but in a fairly passive way

Action research

• the observer/interviewer takes on the role of a subject by participating in the activity under study
• this contrasts with participatory design and cooperative design where the subject takes on the role of observer/interviewer in addition to the primary role of doing the activity

Questionnaires

• highly structured
• static
• use of scales important to get right
  • Lickert scale is 1-5 or 1-7
  • always want an odd number
  • not too few, not too many
  • clearly identify extremes and mid-point
  • use appropriate terms for subject population
  • a scale
    BAD       bad       ok       good       GOOD
    might be used for children
  • use a consistent rating style
• design for efficiency of subjects (time and attention)
• always do a pilot to "debug" the questionnaire
• do a complete analysis of pilot data to uncover coding mistakes

Assurance of anonymity is often a significant issue for subjects when asked to complete questionnaires.

Experiments

• Laboratory vs. experimental method
• Causality
  • random assignment to treatment and control
  • manipulation of independent variables (treatment)
  • control of extraneous variables
• Threats to validity (see Campbell & Cook for examples)
  • Construct validity - are we measuring what we think we are measuring?
  • Internal validity - are the constructs logically related?
  • Statistical validity - could the results be a fluke?
  • External validity - do the results generalize?
  • Ecological (face) validity - are the results realistic?
• Alternative interpretations that may explain the findings
  • novelty
  • not enough practice/training
  • experimenter expectations
  • evaluation apprehension
  • demand characteristics (the "good subject" syndrome)

Type I error is when you observe a difference (a particular difference in the sample means) simply by sampling from the same population -- you find something (a difference) that is not really there.

Type II error is when you fail to observe a difference (at least a particular difference in the means) when sampling from two distinct populations that have population means that differ by an assumed amount -- in this case you don't find something that is there.

Interpretive Approaches

A positivist or reductionist approach, the classic "scientific method", is not always the best way to gain knowledge. More interpretive approaches attempt to explain observations without making some of the assumptions on which the scientific method is based because these assumptions sometimes do not hold.

• There is no objective reality
• Everything we see is a social construction
• Things cannot be "reduced" to their components
• Things cannot be replicated
• We need to understand
  • culture
  • hidden assumptions (ours and others')
  • things in context (what does technology mean?)
  • let the facts guide you (grounded theory)

Criteria to Compare Methods

• Control (internal validity), generalizability (external validity) and realism (ecological validity)
• Natural vs. artificial setting
• Positivist vs. interpretive approaches
• General principles vs. understanding a specific event
• Objective vs. advocacy role of the evaluator
• Time, cost, expertise, or resources available
• Stage of development at which the evaluation is performed

Hawthorne effect

Many variants of this ("good subject" syndrome, etc.) in which the methods used to obtain data affect the data that is gathered.

Ethics

Evaluation must confront a number of ethical issues, which are all-too-often forgotten. Any time that human subjects are involved, not just for experiments, but for all types of observation, a UBC ethics form must be completed.

The forms required by UBC Office of Research Services for experiments involving human subjects are available as MS Word templates on this Web site for you to download.