PSP Metrics specializes in the assessment of candidates for system operator positions in electric utilities and ISOs/RTOs. During the past 50 years, we have conducted 6 largescale validation studies comparing the actual job performance of hundreds of system operators to PSP Metrics test results. Consistently, PSP has found that applicants need to have a particular set of aptitudes, interests, and work behaviors in order to succeed on the job.
While specific success factors have changed over the years, as the job itself as changed, PSP has been able to continuously improve its accuracy rate in system operator selection. At present, we believe that our selection program is the most accurate, thorough, and defensible one of its kind available in North America. The methods used by PSP meet or surpass all professional and governmental standards for employee selection procedures.
Selecting successful system operators is more difficult than many employee selection problems. First, situations do not repeat frequently in the control room setting. Second, it is impossible to predict all of the problems that can arise on the job. Third, usually a relatively small number of system operators work in any one control center. In addition to these difficulties, the job itself has a long learning curve that often involves expensive and extensive training programs. Thus, it takes a great deal of time
before a manager knows for certain if an operator will succeed. This time often exceeds 12 months. In selecting system operators, most managers want to answer four basic questions:
- Can the candidate do the job?
- Will the candidate do the job?
- Will the candidate get along with others?
- Is there room for the candidate to grow?
PSP’s assessment process helps to answer these important questions accurately and objectively, using our proprietary database of successful system operators for benchmarking. These benchmarks were updated in 2013 and include separate standards for distribution and transmission system operators.
Procedures for 2013 Studies
A total of 325 current system operators working at 25 different control centers throughout North America participated in the 2013 validation studies. The control center size varied from 5 to 44 operators. Approximately half of the operators worked in distribution, while the other half worked in transmission control centers. All operators previously had taken PSP’s system operator screening test. System operations managers provided job performance ratings on technical performance and overall work performance for each participant in the studies. Technical performance focused on handling the technical demands of the job itself. Overall performance included the ability to get along with others in a team environment, to communicate effectively, to handle job stress and pressure, and to display good work habits. Manager ratings were compared to PSP test results for each operator in the studies. Statistical analyses were conducted according to scientific and legal standards. Separate analyses were conducted for distribution and transmission groups.
2013 STUDIES: GENERAL FINDINGS
The overall predictive power of the PSP system operator assessment program was demonstrated again in 2013, as it had been in 5 previous decades. Statistically significant differences were found between system operators and unsuccessful applicants in terms of aptitudes, work interests, and work behaviors. Moreover, these success factors were found to include differences from the factors obtained in PSP’s 2003 validation study. The changes in success factors reflect changes in the job itself over this period. These changes are consistent with anecdotal reports from managers and supervisors who have noticed that several additional work interests and work behaviors improve work performance in the current business environment.
The selected group of 325 operators is elite in several aspects. First, all had worked on the job for at least 12 months so that reliable performance records could be established. Second, participants in the studies were much stronger intellectually than the candidate population from which they had been chosen. Third, because they were selected for system operator jobs before the current studies, applicants with poor work habits, poor people skills, low frustration tolerance, or incompatible work interests were weeded out ahead of time. So, while the 2013 group is representative of system operators, it also is a cut above the general population of applicants across industries.
2013 STUDIES: DETAIL RESULTS
Transmission System Operators
Aptitudes – Among transmission system operators, 4 PSP aptitude tests were especially powerful indicators of job performance. Statistically significant PSP aptitude tests included analytical reasoning, critical thinking, verbal skills, and quantitative reasoning. These findings make intuitive sense, as the job requires sequential thinking, multivariate problem solving, and the ability to think on one’s feet in new or first-time situations. The job also requires good verbal skills for the three-way communication process in the control room setting. In addition, mathematical reasoning is becoming increasingly important in understanding the complex metrics of electric power transmission.
Work Interests – Transmission system operators also can be differentiated from unselected applicants on the basis of work interests. The 2013 studies demonstrated that work interest in understanding the scientific/technical aspects of the electric power system is important. In addition, successful transmission system operators enjoy using numbers as tools in analyzing data and solving problems. They are more focused on technically-oriented work activities than they are on socially-oriented endeavors. Finally, transmission system operators appear more interested than the general population when it comes to job-related reading activities.
Work Behaviors – In previous validation studies, highly ranked system operators demonstrated above average frustration tolerance coupled with a very positive outlook. This trend continues in the 2013 studies. Additionally, transmission system operators have a natural optimism and a willingness to give others the benefit of the doubt. They also appear to be non-defensive people who are willing to admit and learn from their mistakes without blaming others.
One additional behavior emerged in the 2013 studies. Work energy appears to be an important predictor of success on the job. This is something of a surprise in that system operations work is considered a sedentary occupation. However, multitasking is increasingly called for as job responsibilities have expanded. In addition, during major outages, transmission system operators’ work tempo necessarily needs to increase. So, when job responsibilities and working conditions are taken into account, work energy makes sense as a predictor of success.
Work energy and the ability to multi-task are increasingly important.
Compared to PSP’s benchmarks for the general working public, transmission system operators excel in two additional ways. First, they tend to be more self-confident and more decisive than the norm. Second, they appear to be more open-minded and more adaptable in thinking than the population at large. In sum, there are several major behavioral attributes which distinguish transmission system operators from unsuccessful applicants and from non-utility employee groups.
Distribution System Operators
Distribution system operators share a number of key aptitudes, work interests, and work behaviors with their transmission counterparts. Distribution system operators need to have strong aptitudes in analytical reasoning and critical thinking, along with verbal and mathematical skills. They enjoy performing data analyses and understanding the scientific/technical workings of the electric power system. They have a positive outlook, handle frustration/pressure well, and demonstrate a good work energy level.
However, the 2013 studies show that additional factors contribute to success at the distribution level that apparently are not needed to the same degree in transmission system operators. For example, distribution system operators appear to be an extroverted group of people. The desire to socialize, to be outgoing, appears to be more important in the distribution level than it is with transmission system operators. In view of the increased communication and teamwork skills needed both inside and outside the control center, it is not surprising that extroversion emerges as an indicator of success.
In comparing distribution system operators to PSP’s benchmarks for the population at large, distribution system operators stand out in several ways. They appear to be brighter than the general working population, more self-confident, more resilient/frustration tolerant, more optimistic, and more technically oriented. Distribution system operators appear to enjoy the intellectual challenge of problem solving, particularly in real-time situations, more than other employees in general. Distribution system operators also welcome feedback on their performance and are more willing to learn than working adults at large.
PSP’s 2013 validation studies for success factors in system operator selection demonstrate that transmission and distribution system operators are more alike than different from each other. To be successful, both groups need to have similar aptitudes and interests. Work behaviors needed for success also overlap in transmission and distribution system operators. Differences between these groups are primarily in the magnitude required of each factor in order to be successful on the job.
Both transmission and distribution system operators, however, are significantly different from the general population in terms of aptitudes, work interests, and work behaviors. In addition, system operators in both categories achieved higher PSP Metrics test scores in 2013 than they did in 2003.
Clearly, it takes more to succeed as a system operator today than it did 10 years ago. This is true for both transmission and distribution system operators. As a result, PSP’s standards for system operator selection have now been updated and revised based on the 2013 studies to reflect the aptitudes, work interests, and work behaviors needed for on-the-job success. Armed with this latest research, PSP will continue to provide objective and accurate screening tools for the electric power industry.
Comments are closed.