Inter-rater reliability of output measures for a posture matching assessment approach: a pilot study with food service workers

Despite the ongoing health problem of repetitive strain injuries, there are few tools currently available for ergonomic applications evaluating cumulative loading that have well-documented evidence of reliability and validity. The purpose of this study was to determine the inter-rater reliability of a posture matching based analysis tool (3DMatch, University of Waterloo) for predicting cumulative and peak spinal loads. A total of 30 food service workers were each videotaped for a 1-h period while performing typical work activities and a single work task was randomly selected from each for analysis by two raters. Inter-rater reliability was determined using intraclass correlation coefficients (ICC) model 2,1 and standard errors of measurement for cumulative and peak spinal and shoulder loading variables across all subjects. Overall, 85.5% of variables had moderate to excellent inter-rater reliability, with ICCs ranging from 0.30–0.99 for all cumulative and peak loading variables. 3DMatch was found to be a reliable ergonomic tool when more than one rater is involved.     

Inter-rater reliability of output measures for a posture matching assessment approach: a pilot study with food service workers

Despite the ongoing health problem of repetitive strain injuries, there are few tools currently available for ergonomic applications evaluating cumulative loading that have well-documented evidence of reliability and validity. The purpose of this study was to determine the inter-rater reliability of a posture matching based analysis tool (3DMatch, University of Waterloo) for predicting cumulative and peak spinal loads. A total of 30 food service workers were each videotaped for a 1-h period while performing typical work activities and a single work task was randomly selected from each for analysis by two raters. Inter-rater reliability was determined using intraclass correlation coefficients (ICC) model 2,1 and standard errors of measurement for cumulative and peak spinal and shoulder loading variables across all subjects. Overall, 85.5% of variables had moderate to excellent inter-rater reliability, with ICCs ranging from 0.30-0.99 for all cumulative and peak loading variables. 3DMatch was found to be a reliable ergonomic tool when more than one rater is involved.     

Reliability of a new risk assessment method for visual ergonomics

IntroductionThe Visual Ergonomics Risk Assessment Method (VERAM) is a newly developed and validated method to assess visual ergonomics at workplaces. VERAM consists of a questionnaire and an objective evaluation.ObjectiveTo evaluate reliability of VERAM by assessing test-retest reliability of the questionnaire, and intra- and inter-rater reliability of the objective evaluation.MethodsForty-eight trained evaluators used VERAM to evaluate visual ergonomics at 174 workstations. The time interval for test-retest and intra-rater evaluations was 2–3 weeks, and the time interval for inter-rater evaluations was 0–2 days. Test-retest reliability was assessed by intraclass correlation (ICC), the standard error of measurement (SEM) and the smallest detectable change (SDC). Intra- and inter-rater reliability were assessed with weighted kappa coefficients and absolute agreement. Systematic changes were analysed with repeated measures analyses of variance and Wilcoxon sign rank test.ResultsThe ICC of the questionnaire indices ranged from 0.69 to 0.87, while SEM ranged from 7.21 to 10.19 on a scale from 1 to 100, and SDC from 14.42 to 20.37. Intra-rater reliability of objective evaluations ranged from 0.57 to 0.85 (kappa coefficients) and the agreement from 69 to 91%. Inter-rater reliability of objective evaluations ranged from 0.37 to 0.72 (kappa coefficients) and the agreement from 52 to 87%.ConclusionVERAM is a reliable instrument for assessing risks in visual work environments. However, the reliability might increase further by improving the quality of training for evaluators. Complementary evaluations of VERAM's sensitivity to changes in the visual environment are needed.Relevance to industryIt is advantageous to set up a work environment for maximal visual comfort to avoid negative effects on work postures and movements and thus prevent visual- and musculoskeletal symptoms. This method, VERAM, satisfies the need of a valid and reliable tool for determining risks associated with the visual work environment.     

Validity and reliability of the Spineangel® lumbo-pelvic postural monitor

Objective: to determine the reliability and the concurrent validity of the Spineangel^® lumbo-pelvic postural monitoring device. Methods: the dynamic lumbo-pelvic posture of 25 participants was simultaneously monitored by the Spineangel^® and Fastrak^TM devices. Participants performed six different functional tasks in random order. Within-task, within-session and between-day intraclass correlation coefficients (ICC(3,1), ICC(3,5), ICC(2,5), respectively) reliability were calculated for Spineangel^® measurements. Concurrent validity of the Spineangel^® was assessed by means of a Bland and Altman plot and by means of Pearson's correlation coefficient and paired t-test. Results: within-task, within-session and between-day ICC for the Spineangel^® were found to be excellent (>0.93). The Spineangel^® and Fastrak^TM pelvic measurements were found to have a good correlation (R = 0.77). Conclusion: the Spineangel^® is a reliable and valid device for monitoring general lumbo-pelvic movements when clipped on the belt or waistband of workers' clothing during various occupational activities.

Practitioner summary: The Spineangel^® can be used for assessing lumbo-pelvic posture during work or daily-life activities. This device was found to provide reliable and valid measurements for lumbo-pelvic movements. Further research is required to determine whether the use of this device is clinically relevant for patients presenting with low back pain.     

Rater reliability and concurrent validity of the Keyboard Personal Computer Style instrument (K-PeCS)

This paper describes the inter-rater and intra-rater reliability, and the concurrent validity of an observational instrument, the Keyboard Personal Computer Style instrument (K-PeCS), which assesses stereotypical postures and movements associated with computer keyboard use. Three trained raters independently rated the video clips of 45 computer keyboard users to ascertain inter-rater reliability, and then re-rated a sub-sample of 15 video clips to ascertain intra-rater reliability. Concurrent validity was assessed by comparing the ratings obtained using the K-PeCS to scores developed from a 3D motion analysis system. The overall K-PeCS had excellent reliability [inter-rater: intra-class correlation coefficients (ICC)=.90; intra-rater: ICC=.92]. Most individual items on the K-PeCS had from good to excellent reliability, although six items fell below ICC=.75. Those K-PeCS items that were assessed for concurrent validity compared favorably to the motion analysis data for all but two items. These results suggest that most items on the K-PeCS can be used to reliably document computer keyboarding style.     

Development of a fully automated RULA assessment system based on computer vision

The purpose of this study was to develop an automated, RULA-based posture assessment system using a deep learning algorithm to estimate RULA scores, including scores for wrist posture, based on images of workplace postures. The proposed posture estimation system reported a mean absolute error (MAE) of 2.86 on the validation dataset obtained by randomly splitting 20% of the original training dataset before data augmentation. The results of the proposed system were compared with those of two experts’ manual evaluation by computing the intraclass correlation coefficient (ICC), which yielded index values greater than 0.75, thereby confirming good agreement between manual raters and the proposed system. This system will reduce the time required for postural evaluation while producing highly reliable RULA scores that are consistent with those generated by manual approach. Thus, we expect that this study will aid ergonomic experts in conducting RULA-based surveys of occupational postures in workplace conditions.     

The test-retest reliability of a new occupational risk factor questionnaire for outcome studies of low back pain

This study reports the test-retest reliability of a 25-item occupational risk factor questionnaire that can be self-administered in clinical settings and used in outcome studies of low back pain or return to work programs. Subjects were 24 patients (workers on sick leave due to acute low back pain), 29 co-workers on active duty in the same jobs in a utility company, and 53 supervisors. Eighty-six subjects were re-tested within a mean interval of 7-10 days; one group of 20 supervisors was re-tested within 81 days. The questionnaire was self-administered during individual interviews. Reliability was estimated by the kappa statistic as the agreement on the scores within the raters in each group. The agreement ranged from 'slight' (0.15) to 'almost perfect' (0.93) when the re-test interval was less than 43 days. Patients and non-patients were consistent in their assessment of the job demands. rights reserved.     

Investigating the effect of real-time spinal postural biofeedback on seated discomfort in people with non-specific chronic low back pain

A total of 24 participants with non-specific chronic low back pain (NSCLBP) sat for 2 h while their seated posture and low back discomfort (LBD) were analysed. A total of 16 pain developers (PDs), whose LBD increased by at least two points on the numeric rating scale, repeated the procedure 1 week later, while receiving postural biofeedback. PDs were older (p = 0.018), more disabled (p = 0.021) and demonstrated greater postural variability (p < 0.001). The ramping up of LBD was reduced (p = 0.002) on retesting, when sitting posture was less end-range (p < 0.001), and less variable (p = 0.032). Seated LBD appears to be related with modifiable characteristics such as sitting behaviour. Among people with sitting-related NSCLBP, the ramping up of LBD was reduced by modifying their sitting behaviour according to their individual clinical presentation. The magnitude of change, while statistically significant, was small and no follow-up of participants was completed. Further research should examine integrating biofeedback into comprehensive biopsychosocial management strategies for NSCLBP.

Practitioner Summary: The effect of real-time postural biofeedback on LBD was examined among people with LBP. Postural biofeedback matched to the individual clinical presentation significantly reduced LBD within a single session. Further research should examine the long-term effectiveness of postural biofeedback as an intervention for LBP.     

Analysis of an observational rating scale for repetition, posture, and force in selected manufacturing settings

OBJECTIVE: The aim of this work was to quantify the interrater reliability of a set of scales that assess repetition, posture, and force as used on site when examining industrial work. BACKGROUND: Interrater reliability of observational assessment methods can vary depending on the definition of the methods and situations in which they are used. METHOD: In several industries, 846 jobs were assessed using pairs of analysts to rate the repetition, force, and posture of the upper limbs. Twelve analysts with varying experience levels participated. RESULTS: Using an interclass correlation coefficient (ICC), force and repetition had reliability values of .60 and .71 before and .82 and .87 after discussion, respectively. After discussion, peak posture ratings had ICCs of .60 to .83. ICCs for average posture ratings ranged from .31 to .51 initially to .55 to .67 in final ratings. Less experienced analysts changed their initial ratings more than did the senior investigators. CONCLUSION: The high interrater reliability of the repetition and force metrics indicates that a single analyst is appropriate for basic job assessment. Posture ratings benefit greatly from a two-analyst system. Average postures should be assessed across a full range of the scale for interrater reliability assessment. Analyst pairs should be rotated to avoid forming biases. APPLICATION: For basic assessments of forceful exertions and repetitive motions, a single analyst can be used, reducing the resource requirements for both industry and large epidemiological studies.     

Center-of-pressure based postural sway measures: Reliability and ability to distinguish between age,fear of falling and fall history

The purpose of this study was to compare the test-retest reliability of a wide variety of center-of-pressure (CoP) based postural sway measures and their ability to detect the differences between the young and older groups (age comparison), between the older low- and high-fear of falling groups (fear of falling comparison), and between the older non-faller and faller groups (fall history comparison). Forty healthy females (twenty each in both young and older groups) performed three trials of bipedal quiet standing on a force platform, in which eighteen reported CoP based measures were computed from recorded CoP trajectory. Intra-class correlation coefficient (ICC), standard error of measurement (SEM) and %SEM were used to quantify their relative and absolute reliability. Fear of falling and retrospective falls of the older group were recorded using falls efficacy scale-international (FES-I) and the history of falls questionnaire respectively. Experimental results showed that eight measures (RMS distance, RMS distance-AP, mean velocity, mean velocity-AP, mean frequency, mean frequency-AP, phase plane parameter and fractal dimension) had acceptable levels of relative and absolute reliability. Three measures (RMS distance-AP, mean velocity, and phase plane parameter) were sensitive to detect age-group difference and fear of falling under both visual conditions, but no single measure was capable to detect differences between the non-faller and faller groups.Relevance to industryThe results of this research provide useful information on the selection of appropriate center-of-pressure (CoP) based postural sway measures to assess individual's balance ability for preventing the occupational falls.     

Reliability of assessing upper limb postures among workers performing manufacturing tasks

The purpose of this study was to determine the inter- and intra-rater reliability of assessing upper limb postures of workers performing manufacturing tasks. Assessment of neck, shoulder, and wrist postures of 20 manufacturing employees was conducted by two raters observing digital video files using Multimedia Video Task Analysis (MVTA). Generalizability theory was used to estimate the inter- and intra-rater reliability. The results demonstrated good to excellent inter-rater reliability for neck and shoulder postures and fair to excellent inter-rater reliability for wrist postures. Intra-rater posture assessment demonstrated good to excellent reliability for both raters in all postures of the neck, shoulder, and wrist. This study demonstrated that posture assessment of manufacturing workers using MVTA is a reliable method.     

Cumulative postural exposure measured by a novel device: a preliminary study

The aim of the present study was to examine the within-day reliability of the Spineangel? postural monitoring device and to measure cumulative lumbo-pelvic posture exposure of health care workers. Twenty-one workers from an aged-care residential home wore the Spineangel, attached to the belt or waistband of their normal work apparel, during a period of the work shift. To assess the within-day reliability of measurements, 11 workers performed two sets of three lumbo-pelvic forward flexion, sustaining them for five 5 s each, at the beginning and at the end of the work shift. Different thresholds for cumulative postural exposure were measured. The reliability was found to be excellent (ICC = 0.81). On average, a threshold of 30° of lumbo-pelvic forward flexion was exceeded 1069 times/h (SD 2157.1); at 45°, 121 times/h (SD 223.8); and at 60°, 8 times/h (SD 21.8). The use of Spineangel is thus likely to be a useful device for monitoring work posture. STATEMENT OF RELEVANCE: The Spineangel? is capable of providing reliable postural measurements in the workplace. Different cumulative postural exposure thresholds were established considering three domains of cumulative exposure: magnitude (range of motion), frequency and duration. The implementation of such domains for cumulative exposure allowed us to explore interesting forms of monitoring posture exposure.     

Inter-rater reliability of query/probe-based techniques for measuring situation awareness

Query- or probe-based situation awareness (SA) measures sometimes rely on process experts to evaluate operator actions and system states when used in representative settings. This introduces variability of human judgement into the measurements that require inter-rater reliability assessment. However, the literature neglects inter-rater reliability of query/probe-based SA measures. We recruited process experts to provide reference keys to SA queries in trials of a full-scope nuclear power plant simulator experiment to investigate the inter-rater reliability of a query-based SA measure. The query-based SA measure demonstrated only ‘moderate’ inter-rater reliability even though the queries were seemingly direct. The level of agreement was significantly different across pairs of experts who had different levels of exposure to the experiment. The results caution that inter-rater reliability of query/probe-based techniques for measuring SA cannot be assumed in representative settings. Knowledge about the experiment as well as the domain is critical to forming reliable expert judgements.     

Development of Human Posture Simulation Method for Assessing Posture Angles and Spinal Loads

Video‐based posture analysis employing a biomechanical model is gaining a growing popularity for ergonomic assessments. A human posture simulation method of estimating multiple body postural angles and spinal loads from a video record was developed to expedite ergonomic assessments. The method was evaluated by a repeated measures study design with three trunk flexion levels, two lift asymmetry levels, three viewing angles, and three trial repetitions as experimental factors. The study comprised two phases evaluating the accuracy of simulating self‐ and other people's lifting posture via a proxy of a computer‐generated humanoid. The mean values of the accuracy of simulating self‐ and humanoid postures were 12° and 15°, respectively. The repeatability of the method for the same lifting condition was excellent (～2°). The least simulation error was associated with side viewing angle. The estimated back compressive force and moment, calculated by a three‐dimensional biomechanical model, exhibited a range of 5% underestimation. The posture simulation method enables researchers to quantify simultaneously body posture angles and spinal loading variables with accuracy and precision comparable to on‐screen posture‐matching methods.     

An investigation of the reliability of Rapid Upper Limb Assessment (RULA) as a method of assessment of children's computing posture

Rapid Upper Limb Assessment (RULA) is a quick observation method of posture analysis. RULA has been used to assess children's computer-related posture, but the reliability of RULA on a paediatric population has not been established. The purpose of this study was to investigate the inter-rater and intra-rater reliability of the use of RULA with children. Video recordings of 24 school children were independently viewed by six trained raters who assessed their postures using RULA, on two separate occasions. RULA demonstrated higher intra-rater reliability than inter-rater reliability although both were moderate to good. RULA was more reliable when used for assessing the older children (8-12 years) than with the younger children (4-7 years). RULA may prove useful as part of an ergonomic assessment, but its level of reliability warrants caution for its sole use when assessing children, and in particular, younger children.     

The investigation of work-related musculoskeletal disorders among female workers in a hazelnut factory: Prevalence,working posture,work-related and psychosocial factors

BackgroundWork-related musculoskeletal disorders (WMSDs) are the most common occupational health problems in the industrialized world. The identification of WMSDs specific to occupation is essential to plan and implement preventative programs. This study investigated the prevalence of WMSDs and determined the factors related to the severity of the most common disorders among female workers in a hazelnut factory.MethodsA total of 114 female workers were examined in this cross-sectional study using the Nordic Musculoskeletal Questionnaire, the Dutch Musculoskeletal Questionnaire, and an author-developed questionnaire (for socio-demographic characteristics and psychosocial factors). Working posture, craniovertebral angle, thoracic kyphosis angle (TKA), and lumbar lordosis angle were evaluated based on a photogrammetry method using the ImageJ program.ResultsThe prevalence of WMSDs in at least one body part during the last 12 months was 92.1%. The highest prevalence of WMSDs was in the lower back (61.4%), neck (57.9%), shoulder (53.6%), and upper back (45.6%). The least affected part was the elbow. Lower back, shoulder, and neck disorders were the most common causes of activity limitation. The severity of lower back pain was associated with stress in the workplace and work-related factors. The neck pain severity was related to craniovertebral angle and work-related factors. The shoulder pain severity increased with TKA and employment duration. TKA was related to upper back pain severity.ConclusionsWMSDs are common among female workers in a hazelnut factory. Reorganizing of working posture, work-related factors, and stress at work may be beneficial to decrease the prevalence of WMSDs and pain severity.Relevance to Industry; The female workers in the hazelnut factory are vulnerable to the work-related musculoskeletal disorders, particularly in the spinal region. The employers should endeavor to improve the health of the workers by the adjustment of the working posture and environmental factors.     

Intra-rater and inter-rater reliability of the rapid entire body assessment (REBA) tool

BackgroundErgonomics researchers and practitioners use many techniques to assess risk. The Rapid Entire Body Assessment (REBA) is a common tool used to facilitate the measurement and evaluation of the risks associated with working postures as a part of ergonomic workload. However, little work has been reported regarding the reliability of REBA reporting.ObjectiveThis study assesses the reliability of this commonly used tool for research and practice.MethodsThe study was conducted as part of the larger Safe Workload Ergonomic Exposure Project (SWEEP), which is a University of Minnesota research initiative for custodians. For this effort, a secondary data analysis was conducted on data collected during a study of custodians’ exposures to risks of musculoskeletal disorders. Eight observers used the REBA tool to sequentially evaluate tasks performed two times in succession by the same individual.ResultsThis study reports high intra-rater reliability (ICC = 0.925) for REBA raw scores and moderate inter-rater reliability (IRR) (Fleiss kappa = 0.54) for a categorical scoring of REBA.ConclusionA moderate amount of IRR was found, and a standardized training and calibration protocol is proposed as a potential means to improve intra- and inter-rater reliability.     

Initial assessment of reliability of a self-administered web-based neuropsychological test battery

IntroductionWeb-based neuropsychological testing can be an important tool in meeting the increasing demands for neuropsychological assessment in the clinic and in large research studies. The primary aim of this study was to investigate practice effects and reliability of self-administered web-based neuropsychological tests in Memoro. Due to lack of consistent analysis and reporting of reliability in the literature, especially intraclass correlation coefficients (ICC), we highlight how using different ICC measures results in different estimates of reliability.Method61 (31 females) participants (mean age 53.3 years) completed the Memoro tests twice with a median of 14 days between testing.ResultsPractice effects were detected for all cognitive measures (d = 0.32–0.61), most pronounced for memory measures. Reliability estimated using two-way random effects single measure absolute agreement ICC(2,1) were between 0.55 and 0.74. Two-way mixed effects average measure consistency ICC(3,2), ranged from 0.79 to 0.89. Reliability was highest for the processing speed task and lower for the memory tasks.ConclusionsMemoro tests had test-retest reliability similar to that of traditional, computerized and web-based test batteries used clinically and in research. It is important to carefully choose and specify the ICC implemented, as ICC(2,1) and ICC(3,2) give different results and reflect reliability of different measures.     

Three-dimensional spinal motion and risk of low back injury during sheep shearing

Sheep shearers are known to work in sustained flexed postures and have a high prevalence of low back pain (LBP). As sustained posture and spinal movement asymmetry under substantial loads are known risk factors for back injury our aim was to describe the 3D spinal movement of shearers while working. We hypothesised that thoraco-lumbar and lumbo-sacral movement would be tri-axial, asymmetric, and task specific. Sufficient retro-reflective markers were placed on the trunk of 12 shearers to define thoraco-lumbar and lumbo-sacral 3D motion during three tasks. Thoraco-lumbar movement consistently involved flexion, left lateral flexion, and right rotation. Lumbo-sacral movement consistently involved right lateral flexion in flexion with minimal rotation. Shearers therefore work in sustained spinal flexion where concurrent, asymmetric spinal movements into both lateral flexion and rotation occur. These asymmetric movements combined with repetitive loading may be risk factors leading to the high incidence of LBP in this occupational group.     

Inter-rater reliability of an inertial measurement unit sensor-based posture-matching method: A pilot study

Posture quantification is important for analyzing trunk and upper extremity loading and estimating the risk of work-related musculoskeletal disorders. This study introduced an inertial measurement unit sensor-based posture-matching (ISPM) method. The ISPM method is the concept that involves observers wearing sensors and simulating postures in photos or videos obtained at job sites to assess the postural risks for the trunk and upper extremity. We tested the inter-rater reliability of the method in 4 novice observers. For the trunk, shoulder/elbow, and wrist joint angle category, weighted kappa scores were 0.89, 0.52, and 0.26, respectively. The results of the ISPM method indicated moderate or high inter-rater reliability for trunk flexion, shoulder flexion/extension, shoulder abduction/adduction, and elbow flexion. The ISPM method was more reliable than conventional observation-based posture assessment tools used for trunk posture analyses. However, the assessment of upper extremity angles indicated that the reliability of ISPM was lower than that of conventional observation-based posture assessment tools. This was a proof-of-concept study conducted using a few samples. Therefore, further testing is necessary to support the findings.Relevance to industryThe ISPM method requires a minimal level of training. It uses conventional video recording and imposes almost no interruptions to the workers’ performance. The ISPM method showed moderate inter-rater reliability for the trunk, shoulder flexion/extension, shoulder abduction/adduction, and elbow flexion during the joint angle analysis.