FMCSA Proposes Changes to the CSA’s SMS

Posted March 02, 2023

On Feb. 15, 2023, the Federal Motor Carrier Safety Administration (FMCSA) published a notice of proposed changes to the safety management system (SMS). The notice also requests comments and input on the FMCSA’s system to identify motor carriers for safety interventions.

The National Research Council of the National Academy of Sciences (NAS) recommended on June 27, 2017, that FMCSA develop and test a new statistical model. This Compliance Bulletin reviews the FMCSA’s analysis and its proposed changes to the SMS.

Comments must be received on or before May 16, 2023.

SMS Overview

In December 2010, the FMCSA implemented the SMS to identify high-risk motor carriers for investigations. The SMS reviews a minimum of four months of motor carrier performance data to identify the highest-risk carriers. The SMS then assigns carrier prioritization status in each Behavior Analysis and Safety Improvement Category (BASIC) based on the carrier’s on-road performance or investigation results. In addition, SMS  provides motor carriers and other stakeholders with updated monthly safety performance data. To complete this analysis and appraisal, the SMS quantifies the safety performance of motor carriers using data available from the FMCSA’s Motor Carrier Management Information System (MCMIS).

NAS Study

Under Section 5221 of the Fixing America’s Surface Transportation(FAST) Act, the NAS was required to conduct a study of the FMCSA’s Compliance, Safety, Accountability (CSA) program and SMS. Specifically, the FAST Act required the NAS to analyze:

  • The accuracy with which the BASICs:
    • Identify high-risk carriers; and
    • Predict or are correlated with future crash risk, crash severity or other safety indicators for motor carriers, including the highest-risk carriers;
  • The methodology used to calculate BASIC percentiles and identify carriers for enforcement, including the weights assigned to particular violations and the tie between crash risk and specific regulatory violations, with respect to accurately identifying and predicting future crash risk for motor carriers;
  • The relative value of inspection information and roadside enforcement data;
  • Any data collection gaps or data sufficiency problems that may exist and the impact of those gaps and problems on the efficacy of the CSA program;
  • The accuracy of safety data, including the use of crash data from crashes in which a motor carrier was free from fault;
  • The determination of whether BASIC percentiles for motor carriers of passengers should be calculated separately from motor carriers of freight;
  • The differences in the rates at which safety violations are reported to the FMCSA for inclusion in the SMS by various enforcement authorities, including States, territories, and Federal inspectors; and
  • Members of the public’s use of the SMS and the effect making the SMS information public has had on reducing crashes and eliminating unsafe motor carriers from the industry.

In addition, the NAS was to consider:

  • If the SMS provides comparable precision and confidence, through SMS alerts and percentiles, for the relative crash risk of individual large and small motor carriers;
  • If alternatives to the SMS would identify high-risk carriers more accurately; and
  • The recommendations and findings of the comptroller general of the United States and the inspector general of the Department of Transportation and independent review team reports issued before the date of enactment of this Act.

On June 27, 2017, the NAS published a report titled ‘‘Improving Motor Carrier Safety Measurement.’’ The NAS report concluded that the SMS, in its current form, is structured in a reasonable way, and its method of identifying motor carriers for alert status is defensible. The NAS agreed that the FMCSA’s overall approach, based on crash prevention rather than prediction, is sound. The NAS provided the FMCSA with six recommendations to improve the system, one of which is to develop an Item Response Theory (IRT), which is what the FMCSA published in this notice. The FMCSA will update and address all NAS recommendations after it reviews the comments on this particular proposal. The FMCSA’s original corrective action plan to the six NAS recommendations can be found on its website.

IRT Overview

The FMCSA conducted a full IRT and found many limitations and practical challenges with using an IRT model. As a result, the FMCSA has concluded that IRT modeling does not perform well for the agency’s use in identifying motor carriers for safety interventions and, therefore, does not improve overall safety. The IRT is overly complex, and adopting the IRT model would reduce transparency without improving safety. As a result, the FMCSA will not replace the SMS with an IRT model. Instead, the FMCSA continues its commitment to continuously improving the SMS to identify motor carriers that present the highest crash risk through a transparent and effective system.

Proposed SMS Changes

The FMCSA conducted analyses during the IRT modeling study that revealed areas where the SMS could be improved to better identify high-risk carriers for intervention without the complications inherent in adopting an IRT model. As a result, the FMCSA is proposing the following combined improvements to the SMS:

  • Reorganizing and updating safety categories, including new segmentation;
  • Consolidating violations;
  • Simplifying violation severity weights;
  • Using proportionate percentiles instead of safety event groups;
  • Improving intervention thresholds;
  • Increasing focus on recent violations; and
  • Updating the utilization factor.

Reorganize and Update Safety Categories

The FMCSA is proposing reorganizing the controlled substances/alcohol, unsafe driving and vehicle maintenance safety categories as described below. The FMCSA also proposes to segment the driver fitness and hazardous materials (HM) compliance safety categories to account for differences in carrier operations. The new safety categories would be:

  • Unsafe driving;
  • Crash indicator;
  • HOS compliance;
  • Vehicle maintenance;
  • Vehicle maintenance: driver observed;
  • HM compliance; and
  • Driver fitness.

Controlled Substances/Alcohol

The FMCSA conducted an exploratory factor analysis (EFA) to identify potential new groupings of violations by highlighting statistical relationships between the violations in each BASIC.

The Controlled Substances/Alcohol group has the fewest violations of any BASIC, and those violations are also cited relatively infrequently. The EFA demonstrated that controlled substances and alcohol violations were strongly associated with the unsafe driving BASIC and supported removing the Controlled Substances/Alcohol category as a standalone BASIC. The new Unsafe Driving Safety Category now includes the drug and alcohol violations that were previously captured in the Controlled Substances/Alcohol BASIC.

Vehicle Maintenance

Vehicle maintenance is the largest BASIC in terms of both the number of violation identifiers included in the BASIC and the number of violations cited during inspections. The EFA results showed that breaking this category into two separate categories would provide greater specificity to help carriers improve and enforcement officials to conduct targeted investigations.

Therefore, vehicle maintenance violations would be divided into two separate categories:

  • Vehicle maintenance: driver observed, which includes violations that may be identified by a driver during a pre- or post-trip inspection and/or while operating the vehicle; and
  • Vehicle maintenance, which includes all other vehicle maintenance violations.

Segmentation in Driver Fitness and HM Compliance

The SMS accounts for differences in carrier operations in the unsafe driving and crash indicator BASICs by segmenting carriers according to whether they primarily operate combination vehicles (i.e., more than 70% of their total power units [PU]) or straight vehicles. Carriers that are not considered combination carriers are considered straight carriers. This segmentation ensures that carriers are compared to other carriers with fundamentally similar exposure to crash risk when operating their vehicles.

The FMCSA tested whether applying segmentation to other safety categories would improve the identification of the highest-risk carriers in those categories. Based on its analysis, the FMCSA is proposing to segment the Driver Fitness and HM Compliance Safety categories to more effectively pinpoint safety issues relating to each operation type. The FMCSA determined that “segmenting HOS Compliance, Vehicle Maintenance: Driver Observed, and Vehicle Maintenance would not improve those safety categories.”

In the Driver Fitness BASIC, carriers that operate straight trucks and similar vehicles have much higher violation rates than motor carriers that operate combination vehicles. Segmenting the Driver Fitness BASIC into straight and combination segments more effectively identifies carriers with higher crash rates in both segments. Although fewer carriers were prioritized for intervention in the Driver Fitness BASIC, the carriers that were removed from prioritization had a lower crash rate, which allows the FMCSA to better focus on those carriers that pose a higher risk to public safety.

The current HM Compliance BASIC compares cargo tank carriers to non-cargo tank carriers, but these carriers have fundamentally different operations. A carrier is categorized as a cargo tank carrier for purposes of segmentation if more than 50% of its inspections indicate the vehicles were cargo tanks. The FMCSA’s analysis found that segmenting carriers as cargo tank carriers and non-cargo tank carriers in the HM Compliance safety category in conjunction with adjusting the HM compliance threshold from the 80th to 90th percentile identifies a group of carriers that has an:

  • HM inspection violation rate that is 22% higher; and
  • HM acute critical (A/C) violation rate that is 46% higher than carriers identified for intervention under the current HM compliance BASIC.

Consolidate Violations

The FMCSA’s analysis during IRT modeling confirmed that similar violation provisions could be consolidated to mitigate differences that result from inspectors citing different violation codes. Grouping similar violations together would also allow motor carriers and enforcement officials to identify and address specific safety issues more easily.

Severity Weights

The SMS assigns each violation a specific severity weight that is intended to correlate with the crash risk associated with that violation. The SMS assigns severity weights to violations on a scale of 1 through 10. This approach has been criticized as overly subjective. The FMCSA tested many different models to improve the severity weights attached to violations in the SMS, including models that applied regression analysis and IRT.

The FMCSA is proposing to simplify violation severity weights by assigning each consolidated violation group a weight of either 1 or 2. Out-of-service (OOS) violations and violations in the Unsafe Driving Safety category that are disqualifying offenses under 49 CFR 383.51 would be assigned a weight of 2, and all other violations would be assigned a weight of 1.

If an OOS violation is combined with a non-OOS violation in the consolidated violation grouping, the consolidated group would be assigned the higher weight of 2. The FMCSA’s evaluation found that simplifying the severity weights identifies carriers with higher crash rates. This change would maintain the safety focus on those violations severe enough to result in an OOS Order while removing the subjectivity and complications of distinguishing each violation by severity on a scale of 1 through 10.

Proportionate Percentile

The FMCSA places motor carriers into safety event groups in the SMS based on their number of inspections and crashes. The FMCSA is now proposing to use a new method of ‘‘proportionate percentiles’’ that will remove sudden jumps in percentiles, which can occur when a carrier moves into a different safety event group. By removing those percentile jumps, the FMCSA would be able to more accurately evaluate whether a carrier’s safety performance is improving or declining from month to month.

The proportionate percentile approach would use safety event groups only to calculate the benchmark median value of each grouping, which would be calculated periodically. A carrier’s proportionate percentile would be calculated from a weighted average of percentiles based on those benchmark medians. After the benchmark run has been established, any changes to a carrier’s percentile would be based solely on the carrier’s own safety performance and would not be impacted by the safety performance of other carriers.

Improved Intervention Thresholds

The FMCSA prioritizes carriers for safety interventions when their SMS percentiles reach or exceed preestablished levels called intervention thresholds.Since higher percentiles represent worse safety performance, a lower intervention threshold in a BASIC represents a more stringent safety criterion.

The FMCSA’s effectiveness test (ET) found that the Unsafe Driving, Crash Indicator, and HOS Compliance BASICs have the strongest correlation to crash risk. Therefore, those BASICs have lower intervention thresholds than the other BASICs, at 65% for property carriers, 60% for HM carriers and 50% for passenger carriers. The intervention thresholds for the vehicle maintenance, controlled substances/ alcohol, and driver fitness BASICs currently are set at:

  • 80% for property carriers,
  • 75% for HM carriers,
  • 65% for passenger carriers; and
  • 80% for all carriers for the HM compliance intervention thresholds.

The FMCSA examined whether adjusting the intervention thresholds for the Driver Fitness, HM Compliance, Vehicle Maintenance, and Vehicle Maintenance: Driver Observed safety categories could improve the agency’s focus on carriers with the highest crash risk.

The FMCSA’s updated ET continues to show that the Driver Fitness and HM Compliance safety categories have the lowest correlation to crash risk. The FMCSA believes raising the intervention thresholds in those safety categories, as shown in the table below, would allow it to focus on populations with a greater safety risk. The FMCSA also considered lowering the intervention thresholds in the Vehicle Maintenance and Vehicle Maintenance: Driver Observed safety categories. However, because the FMCSA is now proposing to split vehicle maintenance into two safety categories, it has determined that more carriers would be prioritized for vehicle maintenance issues by applying the current intervention thresholds to the new Vehicle Maintenance and Vehicle Maintenance: Driver Observed safety categories than are prioritized in the current Vehicle Maintenance BASIC. Therefore, the FMCSA does not propose changing the intervention thresholds for the Vehicle Maintenance safety categories.

Recent Violations

The SMS currently assigns percentiles in the HOS compliance, Vehicle Maintenance and Driver Fitness BASICs if the last inspection in the past two years resulted in a violation. Under this standard, a carrier may be prioritized for intervention even if the carrier has no recent violation. The FMCSA is proposing to sharpen the focus on carriers with more recent violations by assigning percentiles only to carriers that have at least one violation in the safety category in the past 12 months. This change means that if all a carrier’s violations in a particular safety category are 12 months or older, the carrier will not be assigned a percentile in that category.

Updated Utilization Factors

The utilization factor in the SMS helps to account for a carrier’s exposure in the unsafe driving and crash indicator BASICs. Carriers with higher-than-average exposure to safety events, as measured by vehicle miles traveled (VMT) per PU, receive an adjustment in those BASICs. The utilization factor currently covers carriers that drive up to 200,000 VMT per PU per year. The FMCSA is proposing to extend the utilization factor to carriers that drive up to 250,000 VMT per PU in the unsafe driving and crash indicator safety categories to more accurately account for carriers with increased exposure.