A short self-report measure of problems with executive function suitable for administration via the Internet

Please download to get full document.

View again

of 6
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Document Description
A short self-report measure of problems with executive function suitable for administration via the Internet
Document Share
Document Transcript
   P  S   Executive function  is an umbrella term that describes a collection of processes making up the central executive component of the working memory model (Baddeley, 2003; Baddeley & Hitch, 1974). These processes include  planning, task coordination, impulse control, and atten-tion. Deficits in executive function are of clinical and sci-entific interest, and they are detectable in normal as well as clinical populations (Chan, 2001). Problems with ex-ecutive function have been shown to be associated with a variety of phenomena among nonclinical populations, in-cluding antisocial behavior (Morgan & Lilienfeld, 2000), aging (Maylor & Reimers, 2007), and the side effects of recreational drug use (McHale & Hunt, 2008). In such contexts, executive dysfunction may be either a cause or an effect of the behavior or condition in question.Most research that involves measurement of executive function uses laboratory measures or other objective cog-nitive tests (e.g., task-switching, verbal-fluency, Stroop, or trail-making tests). Such objective tests are the gold standard in terms of measuring executive problems; how-ever, there are also instances in which other forms of as-sessment are useful. Self-report questionnaire measures of people’s subjective experiences of executive problems are more convenient to administer than most neuro- psychological or cognitive tests. They may also be infor-mative about how “serious” an executive dysfunction is to the person concerned and about how much impact it has on their day-to-day life. Rabbitt, Maylor, McInnes, Bent, and  Moore (1995) described cognitive self-assessment ques-tionnaires as “indispensable instruments for the study of   709 ©  2010 The Psychonomic Society, Inc. A short self-report measure of problems with executive function suitable for administration via the Internet T OM  B UCHANAN University of Westminster, London, England  T HOMAS  M. H EFFERNAN  Northumbria University, Newcastle Upon Tyne, England  A NDREW  C. P ARROTT University of Swansea, Swansea, Wales J ONATHAN  L ING  Keele University, Keele, England  J ACQUI  R  ODGERS  Newcastle University, Newcastle Upon Tyne, England  AND A NDREW  B. S CHOLEY Swinburne University of Technology, Melbourne, Victoria, Australia This article describes a short self-report measure of problems with executive function designed for use in Internet-mediated research. In Study 1, participants completed the online measure (Webexec) using a browser  but under laboratory conditions. They also completed a paper self-report measure of executive problems (the Dysexecutive Questionnaire; DEX) and three objective tasks involving executive function: reverse digit span, semantic fluency (unconstrained), and semantic fluency (constrained). Webexec scores correlated positively with the DEX and negatively with the three executive tasks. Further evidence of construct validity came from Study 2, in which Webexec scores correlated positively with both use of cannabis and prospective memory  problems reported in an online drug questionnaire. Webexec thus appears suitable for online research with normal populations.  Behavior Research Methods2010, 42 (3), 709-714 doi:10.3758/BRM.42.3.709 T. Buchanan, buchant@wmin.ac.uk   P  S   710 B UCHANAN   ET   AL . use in Internet-mediated research. This is thus distinct from other proprietary measures of executive problems that are currently available, even though it seeks to address the same constructs. Although the initial purpose of the questionnaire was to assess perceived side effects of recreational drug use (Rodgers et al., 2006), we expect it to be useful in many fields of online behavioral research. The chosen validation strategy was to compare responses to the questionnaire with an established self-report measure of executive dysfunction, the DEX, and with a series of established laboratory mea-sures of executive function that were conducted under con-trolled conditions. We hypothesized that scores on the new questionnaire would correlate with the DEX global score and with each of the laboratory measures. STUDY 1Method MaterialsWebexec . The second author generated the items of the Web- based executive function questionnaire ( Webexec ). The six items, shown in Table 1, were designed to tap into experiences of prob-lems with different aspects of executive function. No assumptions were made about fractionation of executive function, and a global score was calculated by summing scores on each of the items. This global score was intended to reflect participants’ overall experience of executive problems rather than any specific aspect thereof. The questionnaire was designed to be as brief as possible, so as to reduce the load on research participants.For purposes of data acquisition, a set of simple Perl CGI scripts were created and were hosted on the University of Westminster Web server. The Webexec items were presented on a single page, with the following instructions:Please answer all the questions below as best you can. Your answers will be confidential, and no information about your identity will be requested or recorded. If you change your mind about taking part in this research, please feel free to stop at any time. Please rate the extent to which you have problems in the following areas by picking the appropriate option from the drop-down menu below each item.For each item, participants responded on a 4-point scale (1    no  problems experienced  ; 2    a few problems experienced  ; 3    more than a few problems experienced  ; 4    a great many problems ex- perienced  ). The total scale score was computed by summing the responses to the six items. On completing the scale, which takes less than 5 min, participants clicked a button labeled “Finished.” They then saw a second Web page thanking them for their participation. A randomly generated six-digit participation code was also shown on this page. This served two functions: First, they could use it as an identifier if they subsequently wished to contact us (e.g., for ret-rospective withdrawal of consent); second, participants were asked to write the number down on their paper questionnaire so that their responses to the Web questionnaire could be linked with the other data they provided. everyday cognition” (p. S149). Although self-report tests also have significant disadvantages (especially the fact that people may not be able to evaluate and report their own impairments accurately), there is evidence that they do have some validity and utility. A number of self-report tests of executive function exist, including the Dysexecu-tive Questionnaire (DEX; Wilson, Alderman, Burgess, Emslie, & Evans, 1996) and the Frontal Systems Behavior Scale (FrSBe; Grace & Malloy, 2001). Measuring Executive Function Online In the context of Internet-mediated research, there are many situations in which one might want to obtain an index of executive problems (see, e.g., Maylor & Reimers, 2007, for online research on the cognitive effects of aging). It is entirely possible to implement objective cognitive tests online—see, for example, the PsychExperiments project described by McGraw, Tew, and Williams (2000) or May-lor and Reimers’s task-switching test. However, develop-ing such measures requires levels of technical expertise or resources that are beyond many researchers interested in online data collection. Furthermore, the way in which such measures are implemented may make assumptions about software available on the devices used by participants to access the Web (e.g., requiring a particular browser plugin to run). On the other hand, self-report questionnaires are considerably easier to implement and can be constructed in ways that make few if any assumptions about the de-vices on which they will be completed.A self-report measure of executive function may thus  be useful for online research. Such an instrument requires validation against objective measures of executive func-tion, and it needs to be validated in an Internet format (see Buchanan et al., 2005, for a discussion of how the psycho-metric properties of a paper questionnaire may sometimes change when it is adapted for online use).There are also advantages to making online ques-tionnaires short in order to reduce load on participants. Ga lesic and Bosnjak (2009) found that reducing ques- tionnaire length in an online survey produced higher par-ticipation rates, and they found some indications that data quality will be poorer for questions placed later in a longer questionnaire. There is a trade-off between questionnaire length and reliability, so the optimum design is probably a questionnaire that is as short as possible while still meet-ing reliability requirements. Aims The aim of the present project was to develop and validate a short, public domain, self-report questionnaire suitable for Table 1 Webexec Items 1. Do you find it difficult to keep your attention on a particular task?2. Do you find yourself having problems concentrating on a task?3. Do you have difficulty carrying out more than one task at a time?4. Do you tend to “lose” your train of thoughts?5. Do you have difficulty seeing through something that you have started?6. Do you find yourself acting on “impulse”?  O  NLINE  E XECUTIVE  F UNCTION  S CALE  711 Results Preliminary analyses of the Webexec scale indicated that it had good internal consistency reliability. Cronbach’s alpha (  N      77) was .785. A principal components analysis indicated that only a single component, which accounted for 48.9% of the variance, had an eigenvalue over 1.0. All items loaded positively on this component, with loadings  between .547 and .820. Although the sample size was small, this analysis gives some confidence that the scale measures a single, coherent construct. The mean score on the scale was 12.13 ( SD    3.02), and scores ranged from 7 to 22. Given the theoretical maximum range of 6 to 24, this suggests a considerable spread in the level of prob-lems people reported experiencing. Associations between Webexec scores and the other measures were evaluated using Pearson’s r and are shown in Table 2.Scores on Webexec were positively correlated with scores on the DEX, indicating that the two measures had much in common. Scores on Webexec were also corre-lated negatively with the objective measures of executive function. These correlations were in the predicted direc-tion: People reporting more problems via Webexec per-formed less well in each of the objective tests. In each case, the correlations with the objective measures were actually stronger for Webexec than for the DEX. Discussion Correlations between Webexec scores, the DEX, and the objective tests of executive function suggest that the Webexec measure has a degree of construct validity. The measure has good internal consistency reliability; how-ever, questions may still be asked about whether its psy-chometric properties will remain satisfactory in a “real” online research context. The present results were obtained using a small, conventional psychology student sample tested under laboratory conditions. Would the psychomet-ric properties be retained with the more heterogeneous volunteer samples who typically participate under much less controlled circumstances in online research projects? Further sources of evidence that will enable this question to be addressed may be found in a reanalysis of data from an earlier online study. STUDY 2 Rodgers et al. (2006) included Webexec as an explor-atory measure in a Web-based study examining self- reported sequelae of using the recreational drug “ecstasy” Cognitive tasks . Participants completed three types of cognitive tasks: reverse digit span, semantic fluency, and semantic fluency with inhibition.  Reverse digit span . Participants heard a list of digits read by the experimenter at a rate of one digit per 2 sec. They were then required to repeat the digits to the experimenter in reverse order. Trials began at the three-digit level, and there were three trials at each level of difficulty. Trials increased in difficulty (number of digits) until the  person failed on two out of three trials at a given level. Scoring was  based on the total number of trials completed correctly. This proce-dure is seen as a good measure of phonological working memory with an executive influence (see, e.g., Rabbitt, 1997). The higher the score, the more efficiently one’s central executive is thought to  be functioning. Semantic fluency . Participants were required to recite as many words as they could from a given category (in this case, animals  beginning with the letter a ) within 1 min. This task involves mul-tiple aspects of executive function (Phillips, 1997), since it involves searching for relevant words from long-term memory, ensuring that items are not repeated, and holding those items in working memory  before verbal presentation. Higher scores are associated with better executive function. Semantic fluency with inhibition . Adapted from Miyake, Fried-man, Emerson, Witzki, and Howerter (2000), this test measured the ability to suppress irrelevant information within a fluency-task paradigm. The procedure mirrored the unconstrained seman-tic task, except that the person had to suppress items containing a particular letter: Participants were instructed to recite as many items of fruit without the letter a  in the descriptor as they could within 1 min. Again, higher scores indicate more efficient execu-tive functioning. DEX . The DEX (Wilson et al., 1996) is a 20-item questionnaire designed to detect everyday signs of executive difficulties. It forms  part of a larger test battery designed to detect executive problems, the Behavioural Assessment of the Dysexecutive Syndrome (BADS; Wilson et al., 1996), and can be administered in either self-report or peer-informant formats. The present investigation used the self-report format, which takes around 10 min to complete. Wilson et al. have shown that the peer-informant format correlates with multiple executive tasks measured by the BADS, whereas Bodenburg and Dopslaff (2008) have reported a reliability coefficient of .85 using the self-report format in brain-injured patients.Although there is evidence that the DEX can be decomposed into multiple factors (see, e.g., Bodenburg & Dopslaff, 2008; Chan,  2001; Wilson et al., 1996), only the global score was used here, for two main reasons. First, the studies above, among others, have shown that the fractionation of executive function is far from clear. Second, our aim was to produce a general measure of executive  problems that might span several domains. Thus, an overall score representing the participant’s overall experience was most suitable for our purposes. Procedure Participants were recruited in psychology laboratory classes at  Northumbria University. They participated one at a time, and all tests were completed in a single session. They first completed the online executive questionnaire using a PC with a Web browser. They then did the reverse digit span task, the measures of semantic flu-ency (constrained and unconstrained), and finally the self-report DEX questionnaire. Total testing time per person was approximately 40 min. Participants Participants were 78 undergraduate psychology students (18 men, 60 women). Ages ranged from 18 to 45, with a mean age of 21.55 ( SD     3.56). One participant left an item blank on the Webexec form, and this person’s scores were excluded from analyses involv-ing that scale. Table 2 Correlations (Pearson’s r  ) Between Self-Report and Objective Measures of Executive Function SemanticSemanticFluency WithReverse DEX Fluency Inhibition Digit Span Webexec.68 ***  .28 *  .35 ***  .42 *** DEX–    .19   .31 ***  .36 *** *  p     .05. ***  p   . 005.  712 B UCHANAN   ET   AL . dents to estimate their level of use of various substances and to an-swer other questions about drug-use patterns and experiences. They were asked to report current frequency of cannabis use on a 4-point scale ( nonuser  , 1–4 times per month , 5–20 times per month , 20   times per month ). For all questions regarding drugs, a “prefer not to answer” option was also included.All of these instruments were presented as interactive forms on a single Web page. Different response formats (clicking on radio  buttons or selecting options from a drop-down menu) were used as appropriate. If participants submitted an incomplete form (i.e., left one or more questions blank), they were automatically informed of this and were asked to supply the missing data and then resubmit the form.Participants also answered a number of demographic questions (age, sex, location, occupation, and education) and questions re-lating to their participation (how they found out about the study, whether they were currently under the influence of any substance, and whether there was any reason their data should not be used in analyses). Analyses pertaining to elements of the questionnaire other than Webexec have been reported elsewhere (Rodgers et al., 2006) and will not be discussed in the present article. Procedure Participants were recruited through a number of methods. These included messages posted on a drug-related bulletin board, links from other online experiments, notices on Web pages and announce-ments in our home institutions, and e-mails to personal contacts. Volunteers participated via an online data-collection Web site. Par-ticipants first saw an informed consent page bearing information about the study and a link to a statement on anonymity and con-fidentiality. They were assured that individual respondents would  be unidentifiable and that they could select “prefer not to answer” options where appropriate. Respondents then saw a page containing  brief instructions; demographic items; the PMQ, Webexec, and drug use questionnaires; and questions about their participation. Once they completed all the items, they clicked on a button labeled “Fin-ished” at the bottom of the page, at which point they were thanked and debriefed. Data Screening Data were screened for multiple responses, with all duplicate sub-missions from the same IP address being excluded. Other exclusion criteria were also applied: instances in which participants indicated that they were under the influence of some substance or that there was some other reason their data should not be used, and implau-sible combinations of demographic data suggestive of mischievous responding (see Rodgers et al., 2006, for a more detailed discussion of this procedure). Following data screening, 417 of the initial 731 submissions remained. Participants Of the 417 participants, 167 were men and 250 were women. The largest proportion (69.3%) came from Europe and had some college or university education (36.9%). The majority (65.0%) reported that they were currently students. Most were age 25 or younger, with 43.2% reporting themselves as falling into the 16–20 age group, and 31.2% reporting themselves as falling into the 21–25 age group. Results and Discussion The mean Webexec score was 11.25 ( SD     3.19). Alpha reliability for this sample was .758, indicating that the scale had acceptable internal consistency. The mean PMQ LT score was 7.43 ( SD     1.02). Alpha reliability for this sample was .833, indicating that the scale had acceptable internal consistency. Among the 415 respondents who an-swered the question on cannabis use, 229 were nonusers, (3,4-methylenedioxymethamphetamine; MDMA) and other substances, including cannabis. This is a prime ex-ample of the type of study for which a short online mea-sure of executive problems may be valuable. Although the main findings from Rodgers et al. and a more detailed description of the methodology have been reported else-where, analyses involving the Webexec measure have not  been published previously and are reported here for the first time.Data from Rodgers et al. (2006) may provide additional information about the validity of the measure, because there is evidence that chronic cannabis use is associated with impairments in executive function. For example, McHale and Hunt (2008) showed that abstinent cannabis users performed less well than controls on laboratory tasks thought to load on executive function. Thus, we predicted that levels of executive impairment reported via Webexec would be positively correlated with use of cannabis.Furthermore, Rodgers et al. (2006) also included a mea-sure of problems experienced with prospective memory (the ability to remember to do things at some point in the future). It has been argued that this ability relies at least in  part on executive functions. Martin, Kliegel, and McDan-iel (2003) found that a compound measure derived from three objective tests (Wisconsin card-sorting test, Stroop task, and Tower of London task) believed to tap executive function was associated with performance on complex  prospective memory tasks. If Webexec scores are asso-ciated with self-reported prospective memory problems, that would comprise further evidence that the Webexec measure behaves in the way that a measure of executive  problems should. Method Materials A Web site was created for the purpose of data acquisition and was hosted on the University of Westminster Web server. The site comprised an informed consent page, followed by a series of ques-tionnaire measures and a debriefing page.Prospective memory was assessed using the Prospective Memory Questionnaire (PMQ). The PMQ is a self-report measure developed  by Hannon, Adams, Harrington, Fries-Dias, and Gipson (1995) that asks people to provide estimates of the frequency with which they have various failures in prospective memory. It has four subscales. The Long-Term Episodic (LT) subscale relates to memory for irreg-ularly scheduled tasks, which require completion some hours or days after a cue to perform them (e.g., remembering to send a birthday card). The Short-Term Habitual (ST) subscale addresses memory for tasks to be completed shortly after the relevant cue that occur on a regular basis (e.g., locking the door when leaving your house). The Internally Cued (IC) subscale addresses memory for tasks that do not have a clear external cue (e.g., forgetting what one is saying halfway through a sentence). The final, somewhat different, subscale is the Techniques to Remember (TR) subscale. This measures the use of strategies to aid prospective memory (e.g., writing notes as reminders of tasks one must do). For the purposes of this study, only the LT subscale of the PMQ was used (Buchanan et al. [2005] found that the ST and IC subscales were not reliable in an online sample, and the TR subscale is not a measure of memory problems per se).Problems experienced with executive function were assessed using Webexec, in the same format described in Study 1. Cannabis use was assessed using a version of the UEL Recreational Drug Use Questionnaire (Parrott, Sisk, & Turner, 2000), which asks respon-  O  NLINE  E XECUTIVE  F UNCTION  S CALE  713 without the additional use of corresponding informant rat-ings. The present validation data were derived from non-clinical populations, and we would counsel against using the measure with more seriously impaired populations in the absence of further validation.Some authors have suggested that self-report measures such as this one may reflect factors other than cognitive impairment. For example, Rabbitt et al. (1995) noted that self-reports may be influenced by factors such as depres-sion and anxiety. Although we would argue that the present data suggest that the people reporting executive problems really do have poorer performance on measures of execu-tive function, we are planning further work to examine the extent to which other factors may contribute to variance in Webexec scores.Further work would also be desirable to elucidate the exact nature of the construct being measured by Webexec. We have argued that the measure reflects participants’ overall experience of problems with a range of executive functions. It would be desirable to ascertain whether it is associated with some functions in particular, or whether it is associated with all equally. This information would help with the interpretation of any findings obtained by stud-ies using Webexec. It would also be desirable to compare Webexec with other cognitive measures (e.g., of attention) in order to rule out the possibility that it actually measures constructs other than executive function. However, the  present data suggest that Webexec is an acceptable tool for online research gathering self-reports of problems experi-enced with executive function in nonclinical populations. AUTHOR NOTE Correspondence concerning this article should be addressed to T. Bu-chanan, Department of Psychology, University of Westminster, 309 Regent Street, London W1B 2UW, England (e-mail: buchant@wmin .ac.uk). REFERENCES Baddeley, A. [D.] (2003). Working memory: Looking back and looking forward.  Nature Reviews Neuroscience ,  4 , 829-839. Baddeley, A. D., & Hitch, G. (1974). Working memory. In G. H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory  (Vol. 8, pp. 47-89). New York: Academic Press. Bodenburg, S., & Dopslaff, N. (2008). The Dysexecutive Question-naire advanced: Item and test score characteristics, 4-factor solution, and severity classification.  Journal of Nervous & Mental Disease ,  196 , 75-78. Buchanan, T., Ali, T., Heffernan, T. M., Ling, J., Parrott, A. C., Rodgers, J., & Scholey, A. B. (2005). Nonequivalence of on-line and paper-and-pencil psychological tests: The case of the prospective memory questionnaire.  Behavior Research Methods ,  37 , 148-154. Chan, R. C. K. (2001). Dysexecutive symptoms among a non-clinical sample: A study with the use of the Dysexecutive Questionnaire.  Brit-ish Journal of Psychology ,  92 , 551-565. Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a Web survey .  Public Opinion Quarterly ,  73 , 349-360. Grace, J., & Malloy, P. F. (2001).  Frontal Systems Behavior Scale (FrSBe) professional manual.  Lutz, FL: Psychological Assessment Resources. Hannon, R., Adams, P., Harrington, S., Fries-Dias, C., & Gipson, M. T. (1995). Effects of brain injury and age on prospective memory self-rating and performance.  Rehabilitation Psychology ,  40 , 289-298. 91 used it 1–4 times per month, 45 used it 5–20 times per month, and 50 used it 20   times per month.Webexec scores were positively correlated with level of cannabis use (Spearman’s        .201, n     415,  p     .0005). Those who used more cannabis reported more executive  problems. This is consistent with McHale and Hunt’s (2008) findings and provides more evidence that self-reported ex-ecutive problems relate to substance use in the same way that “real” objective measures of executive function do.Webexec scores were also positively correlated with scores on the PMQ’s LT prospective memory subscale ( r    .448, n    415,  p   . 0005). This is consistent with the notion that executive function and prospective memory are related, in that people reporting more executive prob-lems also reported more failures of prospective memory. GENERAL DISCUSSION Study 1 demonstrated that the Webexec items form an internally consistent scale that correlates strongly and significantly with the DEX. This indicates that the two measures have much in common, and that if the DEX measures executive problems, it is likely that Webexec does too. Furthermore, Webexec scores correlated nega-tively with three objective measures of executive function (reverse digit span, verbal fluency with constraint, and verbal fluency without constraint): People reporting more  problems displayed poorer performance.Although Webexec and the DEX have similar patterns of associations with the three objective measures, the Web-exec correlations are stronger. One possible reason for this is that Webexec scores may load more strongly than the DEX on those aspects of executive function tapped by the objective tests. However, there are other possible expla-nations: Participants always completed the DEX last, so they could have been tired or inattentive by that stage in the procedure. Alternately, the subjective experience of answering the two questionnaires may have differed, lead-ing to different response sets.Further evidence of validity comes from Study 2, which is more akin to the environment in which we anticipate Webexec will be useful: a less controlled, Web-based proj-ect. Higher Webexec scores were associated with greater use of cannabis (which is believed to be associated with executive problems) and also with a higher level of self- reported failures of prospective memory (which is be-lieved to be associated with executive function).Although these findings point toward the construct validity of the scale, a number of caveats must be borne in mind. An important objection to self-report measures of cognitive function is the notion that people who are cognitively impaired may not be able to give accurate self- reports because of their impairment: Metamemory or metacognition problems impact on their ability to recognize that they have problems (cf. McHale & Hunt, 2008). One implication of this argument is that self-report measures will become less valid as the level of cognitive impairment increases; therefore, measures such as the  present one may have limited utility with clinical samples
Similar documents
View more...
Search Related
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks