The Forecasting Algorithm Is Usually Not the Problem

Executive Summary

Forecasting conversations often gravitate toward algorithms: Which model should we use? Should we add machine learning? Can AI improve accuracy? Those are legitimate questions, but they are rarely the first questions that determine whether a forecast will be useful.

In practice, forecast performance is frequently constrained by problems that sit outside the core statistical model. Sparse and intermittent demand creates noise. Historical shipments can be distorted by stockouts or unusual events. New products and product transitions lack a clean historical baseline. Pricing, promotions, customer changes, and distribution shifts introduce information that history cannot anticipate. And even a technically sound forecast can fail if it is produced at the wrong level of detail for the decision being made.

The implication is straightforward: better forecasting is not simply a model-selection problem. It is a signal-management and decision-design problem. Organizations improve forecasting when they identify what is true signal, explicitly treat known sources of distortion, integrate forward-looking business knowledge, and match forecasting granularity to the decisions that depend on it.

Forecast where the signal exists; plan where the decision needs to be made.

This paper examines five recurring challenge areas and outlines practical responses that can improve forecast usefulness before an organization spends more time tuning algorithms.

1. Why Algorithms Get Too Much Attention

Modern planning platforms provide an expanding library of statistical methods, demand-sensing techniques, machine-learning models, and AI-enabled capabilities. Yet no algorithm can reliably compensate for inputs that misrepresent reality or for a planning design that asks the model to solve the wrong problem.

Consider a few examples. A forecasting model cannot know that a sales spike came from a one-time project unless that event is identified. It cannot infer that shipments were low because inventory was unavailable rather than because customers wanted less. It cannot know that Sales just won a major customer, that Marketing is launching a campaign, or that a competitor is introducing a substitute product. Nor can a model create a stable signal where the data is too sparse at a very detailed level.

This is why the forecast algorithm should be viewed as one component of a broader forecasting operating model. The statistical engine matters, but the quality of the signal presented to that engine—and the way the result is translated into decisions—often matters more.

The five challenge areas

Challenge	What goes wrong	Primary response
Noise	Small-volume detail creates random-looking patterns.	Find the level where a usable signal exists; segment demand and smooth selectively.
Distorted history	Past shipments or orders may not represent true market demand.	Identify abnormal events explicitly and cleanse history using defined policies.
Portfolio change	Products evolve faster than the historical baseline.	Use NPI, supersession, lifecycle, and end-of-life logic rather than waiting for history.
Commercial conditions	Future business actions can overwhelm historical patterns.	Integrate Sales, Marketing, Pricing, and customer knowledge through collaboration.
Granularity	Detailed forecasts become volatile, difficult to reconcile, and impossible to review.	Match forecast level and horizon to where decisions are actually made.

2. Challenge One: Noise

At detailed levels, many demand streams simply do not behave like smooth time series. A SKU-customer-location combination may sell nothing for several periods and then receive one large order. Slow-moving items, aftermarket parts, project-based demand, and customer order batching can all produce histories dominated by zeros and spikes.

When that data is pushed directly into a forecasting model, the organization can end up chasing randomness. Week-to-week actual variation becomes week-to-week forecast variation, even when the underlying business has not meaningfully changed.

What to do instead

• Segment demand based on forecastability rather than applying one method to every item-location-customer combination.

• Clean legitimate historical anomalies where there is a documented business reason to do so.

• Use smoothing or aggregation where the detailed series does not contain a stable signal.

• Avoid forcing a statistical forecast at a level where the data behaves randomly simply because the hierarchy technically allows it.

  
  

The objective is not to hide variability. It is to distinguish variability that contains information from variability that is simply noise.

3. Challenge Two: Distorted History

Historical data is often treated as objective truth, but shipment and order history can be a poor representation of actual market demand. Stockouts suppress shipments. Shipping delays shift demand across periods. Split orders can create false volatility. Strikes, weather events, unusual customer projects, competitor actions, and pre-buying ahead of a price increase can contaminate the baseline.

History cleansing can make the problem worse when planners remove spikes simply because they look unusual. Without root-cause analysis, a cleansing routine can erase legitimate demand or preserve events that should not be repeated in the future.

What to do instead

• Create explicit policies for identifying and treating abnormal events.

• Document the assumption behind every material history adjustment so the organization can distinguish fact from judgment.

• Separate the forecasting algorithm from the history-cleansing process; do not expect a core statistical model to infer why an event was abnormal.

• Flag stockouts and constrained sales so the baseline reflects likely demand, not merely what the supply chain was able to ship.

  
  

A strong forecasting process makes abnormal history visible and explainable. The algorithm should receive the best representation of demand the business can reasonably construct.

4. Challenge Three: Portfolio Change

Forecasting becomes especially difficult when history does not exist—or when the history that does exist is disappearing. New product introductions, product transitions, substitutions, discontinuations, and competitive launches all weaken the value of a simple historical baseline.

A transition from Product A to Product B is rarely isolated. The new item may cannibalize another product, create a halo effect across the portfolio, or move through a soft transition in one channel and a hard transition in another. Shelf timing, customer conversion, and supersession rules can all change the demand pattern.

End-of-life planning deserves equal attention. Organizations often focus heavily on forecasting the launch while neglecting the exit, which can lead to excess inventory, markdowns, obsolescence, and margin erosion.

What to do instead

• Use analog products, predecessor items, families, and lifecycle curves when direct history is unavailable.

• Model supersession explicitly, including halo effects, cannibalization, soft versus hard transitions, and channel differences.

• Build NPI and end-of-life forecasting into the standard process rather than treating them as exceptions handled informally.

• Document commercial and operational assumptions so the forecast can be revised as launch evidence appears.

  
  

5. Challenge Four: Changing Commercial Conditions

History is backward-looking. Business plans are not. Customer wins and losses, new channels, distribution expansion, pricing actions, promotions, and marketing activity can change demand faster than historical patterns can absorb the new information.

Pricing is particularly important and is often underrepresented in forecasting conversations. A future price increase can cause customers to forward-buy, temporarily inflating demand before the change and depressing it afterward. Competitor pricing can create a second-order effect that is not visible in a company’s own history.

Promotions and marketing also need to be separated conceptually. Promotions are generally time-bound and tied to a specific incentive or event. Marketing is broader demand creation or awareness and can be much harder to attribute to a discrete lift. Combining the two under a single “commercial adjustment” can make assumptions difficult to measure or learn from.

What to do instead

• Integrate Sales, Marketing, Pricing, and customer/channel changes into a collaborative business planning process.

• Distinguish between promotions and broader marketing effects so assumptions can be measured appropriately.

• Guard against double counting when a commercial initiative begins to appear in actual history while still remaining in a forward-looking override.

• Treat overrides as hypotheses supported by business intelligence, not as unstructured optimism.

  
  

The statistical forecast should provide a disciplined baseline. The business planning process should then incorporate information the history cannot know yet.

6. Challenge Five: Granularity

A forecast can be mathematically more accurate at an aggregate level and still be useless for the decision that needs to be made. The opposite is also true: forcing forecasts to an extremely detailed level can destroy signal, create volatility, and overwhelm planners with combinations that no one can review.

The right forecasting level depends on the decision and the horizon. Distribution replenishment may require DC-level detail. Production planning may require item-level demand. Capacity decisions may work at a product-family level. Procurement may need a material-family view. Sales, Operations, and Finance may also organize the same business through different hierarchies.

The challenge therefore is not to choose one permanent level and “let it ride.” It is to design aggregation, disaggregation, reconciliation, and horizon logic so that a common demand view can support different decisions at different points in time.

What to do instead

• Forecast at the level where a credible signal exists.

• Disaggregate intelligently when downstream decisions require more detail than the statistically stable forecast level.

• Reconcile customer, product, location, and financial hierarchies rather than allowing each function to operate from an unrelated forecast.

• Vary the planning level by horizon when appropriate; the level needed 120 days out may differ from the level needed 30 days out.

• Use segmentation to focus human review on the combinations where judgment can create value.

Granularity, hierarchy, horizon, and segmentation together create a kind of “3D chess.”  The goal of an elegant process design is to simplify it into a repeatable operating model.

7. A Better Mental Model: Convert Noise into Signal

One useful way to frame operational forecasting is as a continuous effort to define signal over noise. Some demand behavior is already explainable. Other variability initially sits in a general noise bucket because the organization does not yet understand what is driving it.

Improvement comes from systematically decomposing that noise. If stockouts explain part of the variance, treat them explicitly. If customer batching explains another part, model or segment it. If promotions, price moves, or lifecycle transitions explain still more, convert those effects from unexplained variation into managed assumptions.

Forecast improvement, in this view, is not a one-time search for the perfect algorithm. It is an iterative learning process in which more of the business becomes explainable over time. The result is a cleaner baseline, more disciplined overrides, and a better connection between the forecast and the decisions it supports.

8. What a Strong Forecasting Operating Model Looks Like

Organizations that want better forecasting should broaden the agenda beyond model tuning. A mature operating model combines statistical rigor with business context, process discipline, and decision alignment.

1. Segment before you optimize.

Different demand streams have different levels of predictability. Separate stable, seasonal, intermittent, new, promotional, and lifecycle-driven demand before selecting methods.

2. Make history cleansing policy-driven.

Known anomalies should be treated consistently and documented, not removed simply because they look inconvenient.

3. Treat NPI and EOL as core forecasting processes.

Portfolio change is normal business activity, not an exception to the forecasting process.

4. Separate baseline from business intelligence.

Use the statistical forecast as a disciplined starting point, then add forward-looking knowledge through a controlled collaborative process.

5. Design granularity around decisions.

Forecast where the data contains signal and translate that forecast to the level required by replenishment, production, capacity, procurement, and financial planning.

6. Measure learning, not just error.

Accuracy and bias still matter, but so does whether the organization is reducing unexplained variation, improving assumptions, and preventing repeated causes of forecast error.

9. The Role of Technology and AI

Technology can make this operating model easier to execute, but only when it reinforces the process. Forecasting engines can automate model selection and statistical generation. Segmentation tools can help identify where a forecast is likely to be predictable. Workflow and collaboration platforms can capture assumptions, connect commercial inputs, maintain decision history, and support reconciliation across functions.

AI can also accelerate analysis: identifying patterns that deserve investigation, summarizing changes, surfacing anomalies, or helping planners evaluate scenarios. But AI should not become a substitute for business definition. A system still needs to know which events are abnormal, which assumptions are approved, which hierarchy matters, and which decision the forecast is intended to support.

The most valuable use of technology is therefore not to hide the forecasting process inside a black box. It is to standardize the repeatable work, make assumptions visible, and give planners more time to investigate the parts of demand that remain unexplained.

Conclusion

Forecast accuracy will always attract attention, and statistical models will continue to improve. But organizations should resist the assumption that the next algorithm will solve a forecasting problem rooted in noisy data, distorted history, portfolio churn, missing commercial context, or poor planning design.

The more productive question is: What is preventing the organization from seeing and using the true demand signal? That question shifts the focus from model selection to the broader disciplines that make a forecast useful.

When companies segment demand intelligently, cleanse history with policy, manage portfolio transitions explicitly, integrate forward-looking business knowledge, and align granularity to decisions, the algorithm becomes what it should be: a repeatable analytical engine inside a much stronger decision process.

The objective is not a perfect forecast. It is a forecast that is explainable, appropriately detailed, and good enough to drive better business decisions.