Assessing tremor severity.

A clinical rating scale which measured the severity of tremor in 20 patients (12 with essential tremor and 8 with "dystonic" tremor) was assessed at specific anatomical sites for both inter and intra-rater reliability using four raters. The scores obtained with the scale were compared with the results of upper limb accelerometry, an activity of daily living self-questionnaire and estimates of the tremor induced impairment in writing and drawing specimens. The results show that, for the purposes of routine assessment and therapeutic trials, a clinical rating scale can produce reliable results which are a more valid index of tremor induced disability than standard postural accelerometry.     

Unified Parkinson's disease rating scale motor examination: Are ratings of nurses,residents in neurology,and movement disorders specialists interchangeable?

The Unified Parkinson's Disease Rating Scale (UPDRS) is widely used for the clinical evaluation of Parkinson's disease (PD). We assessed the rater variability of the UPDRS Motor examination (UPDRS-ME) of nurse practitioners, residents in neurology, and a movement disorders specialist (MDS) compared to a senior MDS. We assessed the videotaped UPDRS-ME of 50 PD patients. Inter-rater and intra-rater variability were estimated using weighted kappa (kappa(w)) and intraclass correlation coefficients (ICC). Additionally, inter-rater agreement was quantified by calculation of the mean difference between 2 raters and its 95% limits of agreement. Intra-rater agreement was also estimated by calculation of a 95% repeatability limits. The kappa(w) and ICC statistics indicated good to very good inter-rater and intra-rater reliability for the majority of individual UPDRS items and the sum score of the UPDRS-ME in all raters. However, for inter-rater agreement, it appeared that both nurses, residents, and the MDS consistently assigned higher scores than the senior MDS. Mean differences ranged between 1.7 and 5.4 (all differences P < 0.05), with rather wide 95% limits of agreement. The intra-rater 95% repeatability limits were rather wide. We found considerable rater difference for the whole range of UPDRS-ME scores between a senior MDS and nurse practitioners, residents in neurology, and the MDS. This finding suggests that the amount by which raters may disagree should be quantified before starting longitudinal studies of disease progression or clinical trials. Finally, evaluation of rater agreement should always include the assessment of the extent of bias between different raters.     

A novel approach to assess inter-rater reliability in the use of the Overt Aggression Scale-Modified

The ability of raters to apply measures of efficacy accurately and reproducibly in clinical trials of psychotropic medications is vital to the validity of study data. In a large, multi-center trial using the Overt Aggression Scale-Modified (OAS-M), inter-rater reliability was evaluated using standardized patients (SP) in a live 'mock' interview. Thirty raters experienced in the OAS-M each interviewed two actors trained to portray patients with intermittent explosive disorder, borderline and antisocial personality disorder, and/or post-traumatic stress disorder. Inter-rater reliability of OAS-M scores was evaluated using the intra-class correlation coefficient (ICC). The joint reliability of the raters in the study with an expert rater was also assessed by an ICC. The ICC was 0.96 between the raters, and 0.98 between the raters and expert rater. Results suggest that SP can be utilized in rater assessment and that raters administering the OAS-M in this model of rater reliability assessment demonstrate a high level of consistency and reliability. The method described here may be useful in future assessments of inter-rater reliability.     

An evaluation of rating scales utilized for deep brain stimulation for dystonia

The objective of this study was to examine globus pallidus internus deep brain stimulation (GPi-DBS) outcomes in primary and secondary dystonia, derived from blinded ratings using two scales and two raters. Twenty-five patients with variable presentations of dystonia were evaluated with videotaped standardized dystonia rating scales at preoperative baseline and at 6 and 12 months following GPi-DBS implantation. These 75 examination videos were retrospectively evaluated, independently and in random order, by two movement disorder neurologists who were blinded to the treatment status. Both neurologists scored each videotaped evaluation using the Burke–Fahn–Marsden Dystonia Rating Scale-motor part (BFMDRS-M) and the Unified Dystonia Rating Scale (UDRS). A final score for each video was assigned by averaging the raters’ scores. An intra-class correlation coefficient was used to calculate inter-rater reliability. A linear mixed model was fitted to investigate the time effect and its interaction with type of dystonia (primary versus secondary) for each rating scale. Inter-rater reliability was excellent. Intraclass correlation coefficients ranged from 0.994 to 0.997 for both scales at baseline, 6 and 12 months. The average motor improvement scores after GPi DBS for the entire heterogeneous group of dystonia patients after 6 and 12 months of stimulation was 21.32% (p = 0.0010) and 28.95% (p = 0.0017), respectively, when the UDRS score was used. Similar levels of improvement 20.46% (p = 0.0055) at 6 months and 27.39% (p = 0.00197) at 12 months were found using the BFMDRS-M score. Analysis using unblinded scores from our database revealed a 32.99 and 37.27% UDRS improvement at 6 and 12 months, and an improvement in UDRS score of 38.5 and 43.7% when the analysis was limited to only primary dystonia. If the data were further segregated to include only cases of DYT-1 primary generalized dystonia, the UDRS benefit increased to 48.24%. Our primary dystonia group was diluted by the presence of both old- and young-onset patients, as well as focal, segmental and generalized dystonia. In conclusion, (1) evaluating motor outcomes of DBS therapy for dystonia using independent, randomized retrospective rating by blinded raters’ results in lower improvement scores than when outcomes are rated by unblinded treating neurologists. Blinded methodology may be superior and might produce a more realistic assessment of motor outcomes after DBS in patients with dystonia; (2) outcomes were similar whether the BFMDRS-M or UDRS was utilized; (3) GPi-DBS was effective in treating sustained involuntary motor co-contractions in medication refractory dystonia patients, more so in primary dystonia.     

Reliability of a Rating Scale to Assess Post-stroke Psychiatric Symptoms

Background : Various rating scales for post-stroke psychiatric symptoms such as cognitive impairment have long been used in drug efficacy trials in Japan. However, their reliability has not been established. The purpose of the present study was to examine the inter-rater and intra-rater reliability of a rating scale, which has been conventionally used in Japan.
Methods : The most frequent symptoms, including apathy, emotional and cognitive impairment, found in post-stroke patients were assessed using a rating scale comprising 17 items and 4 global assessment scales. Semi-structured interviews of 18 patients with symptoms with various degrees of severity were videotaped. Twelve physicians who were not interviewers independently assessed the severity of symptoms in the 18 patients by replaying the videotapes. This procedure was repeated twice with an interval of two months to examine inter-rater and test-retest reliability.
Results : The results revealed almost satisfactory reliability. Intraclass correlation coefficients were over 0.6 in the inter-rater analysis for most items and the concordance correlation coefficients were over 0.7 in the intra-rater analysis. The rating scale was considered to be reliable, although some items showed relatively low agreement.
Conclusion : The conventional rating scale to assess post-stroke psychiatric symptoms showed satisfactory inter-rater and intra-rater reliability by the videotape method. The validity study should be further investigated.     

The Toronto Western Spasmodic Torticollis Rating Scale: reliability in neurologists and physiotherapists

IntroductionInter-rater reliability for the Toronto Western Spasmodic Torticollis Rating Scale has been determined for neurologists, but not for physiotherapists. We assessed the inter-rater reliability of scoring the clinical severity subscale of the Toronto Western Spasmodic Torticollis Rating Scale by two physiotherapists and two neurologists.MethodsSeven individuals with cervical dystonia were assessed on video a total of 23 times. The assessments were randomized and rated by all raters.ResultsReliability for all raters considered together, assessed by the Intraclass Correlation Coefficient (ICC 2,1) was 0.74 (p < 0.001, 95% CI 0.46–0.88). Reliability for the two clinician groups was not significantly different: neurologists ICC (2,1) = 0.75 (p < 0.001, 95% CI 0.44–0.89), physiotherapists ICC (2,1) = 0.82 (p < 0.001, 95% CI 0.30–0.94).ConclusionThe clinical severity subscale of the Toronto Western Spasmodic Torticollis Rating Scale can be assessed with good inter-rater reliability when physiotherapists and neurologists are represented in the same rater pool.     

Assessment of reliability in the clinical evaluation of depressive symptoms among multiple investigators in a multicenter clinical trial.

The objective of this work was to determine the severity of depressive symptoms when multiple clinical examiners evaluate a single subject, as preparatory to their participation as evaluators in a clinical trial. Using the 17-item Hamilton Depression Rating Scale (HDRS), 37 psychiatrists independently assessed the videotape of a patient with symptoms of depression. A new measure for the detection of multiple examiners not in consensus (DOMENIC) was used to identify scale items with low reliability and raters with low inter-rater reliability, from among the remaining raters. Overall inter-rater agreement on the full HDRS was 'excellent' (97%). All raters but one showed adequate agreement both on individual items and on total scores. Two of the 17 HDRS symptomatology items had unacceptable levels of inter-rater scoring variability (<70% agreement). The use of DOMENIC allows for the detection of items of low inter-rater reliability and identification of raters that deviate from the group's ratings prior to the beginning of a clinical trial.     

A structured interview guide for global impressions: increasing reliability and scoring accuracy for CNS trials

Background

The clinical global impression of severity (CGI-S) scale is a frequently used rating instrument for the assessment of global severity of illness in Central Nervous System (CNS) trials. Although scoring guidelines have been proposed to anchor these scores, the collection of sufficient documentation to support the derived score is not part of any standardized interview procedure. It is self evident that the absence of a standardized, documentary format can affect inter-rater reliability and may adversely affect the accuracy of the resulting data.

Method

We developed a structured interview guide for global impressions (SIGGI) and evaluated the instrument in a 2-visit study of ambulatory patients with Major Depressive Disorder (MDD) or schizophrenia. Blinded, site-independent raters listened to audio recorded SIGGI interviews administered by site-based CGI raters. We compared SIGGI-derived CGI-S scores between the two separate site-based raters and the site-independent raters.

Results

We found significant intraclass correlations (p = 0.001) on all SIGGI-derived CGI-S scores between two separate site-based CGI raters with each other (r = 0.768) and with a blinded, site-independent rater (r = 0.748 and r = 0.706 respectively) and significant Pearson’s correlations between CGI-S scores with all MADRS validity comparisons for MDD and PANSS comparisons for schizophrenia (p- 0.001 in all cases). Compared to site-based raters, the site-independent raters gave identical “dual” CGI-S scores to 67.6% and 68.2% of subjects at visit 1 and 77.1% at visit 2.

Conclusion

We suggest that the SIGGI may improve the inter-rater reliability and scoring precision of the CGI-S and have broad applicability in CNS clinical trials.     

Bilateral pallidotomy for generalized dystonia.

OBJECTIVE: To evaluate the efficacy and safety of bilateral pallidotomies in five patients with generalized dystonia. BACKGROUND: Generalized dystonias are frequently a therapeutic challenge, with poor responses to pharmacological treatment. GPi (globus pallidus internus) pallidotomies for Parkinson's disease ameliorate all kinds of dyskinesias/dystonia, and recent studies reported a marked improvement of refractory dystonias with this procedure. METHODS: Five patients with generalized dystonias refractory to medical treatment were selected; one posttraumatic and four idiopathic. The decision to perform bilateral procedures was based on the predominant axial involvement in these patients. Dystonia severity was assessed with the Burke-Fahn-Marsden Dystonia Scale (BFM). Simultaneous procedures were performed in all but one patient, who had a staged procedure. They were reevaluated with the same scale (BFM) by an unblinded rater at 1, 2, 3, 30, 60, 90, 120 and 180 days post-operatively. RESULTS: The four patients with idiopathic dystonia showed a progressive improvement up to three months; the patient with posttraumatic dystonia relapsed at three months. One patient had a marked improvement, being able to discontinue all the medications. A mean decrease in the BFM scores of 52,58% was noted. One patient had a trans-operative motor seizure followed by a transient hemiparesis secondary to rack hemorrhage; other was lethargic up to three days after the procedure. CONCLUSIONS: Our results show that bilateral GPi pallidotomies may be a safe and effective approach to medically refractory generalized dystonias; it can also be speculated that the posttraumatic subgroup may not benefit with this procedure.     

A short scale for the assessment of motor impairments and disabilities in Parkinson's disease: the SPES/SCOPA

OBJECTIVES: To evaluate the reliability and validity of the Short Parkinson's Evaluation Scale (SPES)/SCales for Outcomes in Parkinson's disease (SCOPA)-a short scale developed to assess motor function in patients with Parkinson's disease (PD). METHODS: Eighty five patients with PD were assessed with the SPES/SCOPA, Unified Parkinson's Disease Rating Scale (UPDRS), Hoehn and Yahr (H&Y) scale, and Schwab and England (S&E) scale. Thirty four patients were examined twice by two different assessors who were blinded to each other's scores and test executions. Additionally, six items of the motor section of the SPES/SCOPA were assessed in nine patients and recorded on videotape to evaluate inter-rater and intra-rater reliability. RESULTS: The reproducibility of the sum scores in the clinical assessments was high for all subscales of the SPES/SCOPA. Inter-rater reliability coefficients for individual items ranged from 0.27-0.83 in the motor impairment section, from 0.58-0.82 in the activities of daily living section, and from 0.65-0.92 in the motor complications section. Inter-rater reliability of the motor items in the video assessments ranged from 0.70-0.87 and intra-rater reliability ranged from 0.81-0.95. The correlation between related subscales of the SPES/SCOPA and UPDRS were all higher than 0.85, and both scales revealed similar correlations with other measures of disease severity. The mean time to complete the scales differed significantly (p<0.001) and measured 8.1 (SD 1.9) minutes for the SPES/SCOPA and 15.6 (SD 3.6) minutes for the UPDRS. CONCLUSION: The SPES/SCOPA is a short, reliable, and valid scale that can adequately be used in both research and clinical practice.     

The use of a rating scale of attentional behaviour

Abstract

In recent years there has been increasing awareness of the limitations of conventional neuropsychological assessment in terms of predicting functional performance in everyday life. In response to the growing recognition of the need for ecological validity in rehabilitation settings, clinicians have begun to develop methods (e.g. rating scales and behaviour checklists) of assessing the impact of neuropsychological impairments on the everyday behaviour of the patient. However, there have been few attempts to investigate the validity or reliability of such scales. The aims of the present study were: (i) to design a rating scale of attentional behaviour, to be completed by therapists treating traumatically head-injured patients who were receiving a remedial intervention for their attentional deficits; (ii) to examine the scale's correlation with neuropsychological measures of attention; (iii) to assess its internal structure and intra-rater reliability; and (iv) to examine the use of the scale by raters in different contexts and by individual raters over time. The scale developed showed some validity as a measure of attentional behaviour. It was quick and simple to administer. It showed modest, but statistically significant correlations with neuropsychological measures of attention, a high level of internal consistency, and excellent intra-rater reliability. The correlations between scores made by different raters in different contexts were much lower. Possible reasons for this finding are discussed, including the influence of context and the frame of reference of the rater. The importance of exploring and attempting to address potential sources of dissonance, both within and between raters, is emphasised.     

Clinical rating of dyskinesias in Parkinson's disease: use and reliability of a new rating scale.

Drug-induced dyskinesias (DID) manifested as hyperkinetic and/or dystonic movements or postures are common problems in Parkinson's disease (PD). Novel therapeutic interventions may offer possibilities to counteract these common adverse effects of an otherwise necessary treatment. To be able to evaluate the effects of such interventions on DID, reliable and relevant clinical assessment tools are needed. We tested the inter- and intrarater reliability of a new clinical dyskinesia rating scale consisting of separate ratings of different body parts, including lateralization and separate ratings of dystonia and hyperkinesias. Interrater reliability was tested both with and without a defined scoring code and clarification of the dystonia section. The nondefined version was also tested for intrarater reliability. Thirteen raters independently reviewed 23 videotape sequences showing PD patients performing standardized motor tests. Inter- and intrarater agreement was significant in all evaluations, and no differences were detected when comparing ratings performed with the defined and nondefined version of the scale. The rationale for, and the role and use of, the present scale are addressed.     

Assessing clinicians' consultation with people with profound learning disability: producing a rating scale

This paper describes the development of a scale for assessing clitiicians’ communication with people with profound learning disability. Four psychiatrists and four nurses were assessed completing three simple non-invasive clinical procedures—blood pressure, pulse and axillary temperature—with people with profound learning disability. Videotaped assessmetit of consultations was performed by three experienced speech and language therapists using a previously designed scale for analysing encounters with people with mild learning disability. This led to the production of a new scale specifically for people with profound learning disability. A significant inter-rater reliability was found between the three speech therapists for total scores (rater a-b, corr = 0·654, P= 0·006; rater a–c, corr = 0·795, P= 0·0001: rater b–c, corr = 0·673, P = 0·004). Significant reliability between raters was also found for the subsections of verbal behaviour and non-verbal behaviour.     

A majority rule approach for region-of-interest-guided streamline fiber tractography

Hand-drawn gray matter regions of interest (ROI) are often used to guide the estimation of white matter tractography, obtained from diffusion-weighted magnetic resonance imaging (DWI), in healthy and in patient populations. However, such ROIs are vulnerable to rater bias of the individual segmenting the ROIs, scan variability, and individual differences in neuroanatomy. In this report, a “majority rule” approach is introduced for ROI segmentation used to guide streamline tractography in white matter structures. DWI of one healthy participant was acquired in ten separate sessions using a 3 T scanner over the course of a month. Four raters identified ROIs within the left hemisphere [Cerebral Peduncle (CPED); Internal Capsule (IC); Hand Portion of the Motor Cortex, or Hand Bump, (HB)] using a group-established standard operating procedure for ROI definition to guide the estimation of streamline tracts within the corticospinal tract (CST). Each rater traced the ROIs twice for each scan session. The overlap of each rater’s two ROIs was used to define a representative ROI for each rater. These ROIs were combined to create a “majority rules” ROI, in which the rule requires that each voxel is selected by at least three of four raters. Reproducibility for ROIs and CST segmentations were analyzed with the Dice Similarity Coefficient (DSC). Intra-rater reliability for each ROI was high (DSCs ≥ 0.83). Inter-rater reliability was moderate to adequate (DSC range 0.54–0.75; lowest for IC). Using intersected majority rules ROIs, the resulting CST showed improved overlap (DSC = 0.82) in the estimated streamline tracks for the ten sessions. Despite high intra-rater reliability, there was lower inter-rater reliability consistent with the expectation of rater bias. Employing the majority rules method improved reliability in the overlap of the CST.     

Pallidal deep brain stimulation relieves camptocormia in primary dystonia

Camptocormia, characterised by a forward flexion of the thoracolumbar spine may occur in various movement disorders, mainly in Parkinson’s disease or in primary dystonia. In severe cases, patients with camptocormia are unable to walk. While treatment options are limited, deep brain stimulation (DBS) with bilateral stimulation of the subthalamic nucleus or globus pallidus internus (GPi) has been proposed as a therapeutic option in refractory cases of Parkinson’s disease. Here we present two patients with severe camptocormia as an isolated form of dystonia and as part of generalised dystonia, respectively, which were both treated with bilateral stimulation of the GPi. Symptoms of dystonia were assessed using the Burke–Fahn–Marsden dystonia rating scale (BFM) before and during deep brain stimulation. In both patients there was a significant functional improvement following long-term bilateral GPi stimulation and both patients gained ability to walk. In the first patient with an isolated dystonic camptocormia the BFM motor subscore for the truncal flexion improved by 75 %. The total BFM motor score in the second patient with a camptocormia in generalised dystonia improved by 45 %, while the BFM score for truncal flexion improved by 87 %. In both patients the effect of the bilateral GPi stimulation on camptocormia was substantial, independent of generalisation of dystonia. Therefore, GPi DBS is a possible treatment option for this rare disease.     

Interrater reliability in clinical trials of depressive disorders

OBJECTIVE: The goal of this study was to review the reported methods of rater training, assessment of interrater reliability, and rater drift in clinical trials of treatments for depressive disorders. METHOD: Two psychiatrists independently identified all original reports of clinical trials relevant to depressive disorders published between 1996 and 2000 in the American Journal of Psychiatry and the Archives of General Psychiatry. Reported methods of rater training, assessment of interrater reliability, and rater drift were systematically summarized. RESULTS: Sixty-three original papers met criteria for inclusion. Only 11 (17%) of the studies reported the number of raters. Only two (9%) of the 22 multicenter and four (10%) of the 41 single-center trials documented rater training. Only nine (22%) of the single-center trials and three (14%) of the multicenter trials reported interrater reliability, despite a median number of five raters (range=2-20). Only three (5%) of the 63 articles reported rater drift. CONCLUSIONS: Few published reports of clinical trials of treatments for depressive disorders document adequately the number of raters, rater training, assessment of interrater reliability, and rater drift.     

Intrathecal baclofen for dystonia: benefits and complications during six years of experience.

Fourteen patients with primary or secondary dystonia received intrathecal baclofen (ITB) through an implanted pump following a trial dose. Patients were selected for ITB trial if they had clinically unsatisfactory responses to oral antidystonic medications, including oral baclofen. Patients were rated using the Burke-Fahn-Marsden rating scale by a blinded rater after the dose of ITB was optimized. Five patients experienced improvement in symptoms as determined by a change in rating scale scores, although only two had a clear clinical benefit. Etiology of dystonia did not determine the efficacy of ITB therapy, as benefit or failure was seen in both primary and secondary dystonia.     

Validity and reliability of a rating scale for the primary torsion dystonias

For quantitative assessment of the primary torsion dystonias, a rating scale is proposed that has two sections--a Movement Scale, based on examination, and a Disability Scale, based on the patient's statements about seven activities of daily living. We assessed the validity of the Movement Scale by comparing scores with a ranking of patients according to dystonia severity and with ratings of the patients on the Disability Scale. In addition, we assessed the inter-and intra-rater reliability of the scale by comparing independent scorings of patients by four examiners and by comparing scorings by the same examiners performed at different times. We found that the Movement Scale was a valid and reliable indicator of the severity of primary torsion dystonia.     

Chronic high-frequency globus pallidus internus stimulation in different types of dystonia: a clinical, video, and MRI report of six patients presenting with segmental, cervical, and generalized dystonia.

The results of deep brain stimulation (DBS) of the globus pallidus internus (Gpi) in six patients with generalized, focal, and segmental dystonia are presented. Pre- and postoperative assessments are given for one patient with generalized inherited dystonia and for five patients with idiopathic segmental or cervical dystonia. Clinical symptoms were evaluated before and 3-12 months after surgery using the Burke-Fahn-Marsden (BFM) dystonia rating scale for primary torsion dystonia and the Tsui scale for cervical dystonia. The Short-Form Health Survey (SF-36) was completed by each patient to document preoperative and postoperative health status. Also, neurological status was documented by video before and during chronic stimulation. Magnetic resonance imaging studies were performed to show the anatomical localization of the electrode leads. Five patients showed a progressive improvement within 7 days. One patient with cervical dystonia and Meige's syndrome showed no improvement for 3 months, but beneficial effects were observed after 12 months. On average, the BFM movement scale scores decreased by 72.5% and Tsui scale scores by 63%. SF-36 showed an improvement in health status by an average of 36% according to eight different health categories. We conclude that chronic high-frequency Gpi stimulation in different types of dystonia is a very effective and safe treatment.     

A videotaped CIBIC for dementia patients: validity and reliability in a simulated clinical trial

BACKGROUND: The global impression of a clinician is an Food and Drug Administration--mandated primary outcome measure for clinical trials in dementia. Reliability and validity of these measures are not well established. METHODS: A videotaped version of the Clinician's Interview Based Impression of Change (CIBIC) was evaluated. Raters were informed that the videotaped interviews were taken at baseline and 6 to 12 months later, when in fact half of the interviews were shown in reverse order. Ratings on "true order" interviews were compared with ratings on "reverse order" interviews. In addition, ratings by neurologists experienced in dementia were compared with those of less experienced raters. RESULTS: Inter-rater reliability of the neurologists was poor when measured by absolute agreement on a 7-point scale (kappa = 0.18). With a less stringent 3-point scale (better, worse, or unchanged), inter-rater reliability was significantly better for the true order videos (kappa = 0.51) than for the reversed order videos (kappa = 0.12). Validity also was reduced in the reverse order group: neurologists rated 90% of subjects correctly in the "true order" group and 63% correctly in the "reversed order" group. The inter-rater reliability of the neurologists was greater than the less experienced raters, but the validity of the neurologists' ratings was only marginally better. CONCLUSIONS: The reliability and validity of the videotape CIBIC are reasonable when patients follow the expected course of gradual decline, but are poor when patients appear to improve. These findings suggest that global assessments should be modified as outcome measures in clinical trials with patients with dementia.