This put up is co-written by Goktug Cinar, Michael Binder, and Adrian Horvath from Bosch Heart for Synthetic Intelligence (BCAI).
Income forecasting is a difficult but essential activity for strategic enterprise selections and monetary planning in most organizations. Typically, income forecasting is manually carried out by monetary analysts and is each time consuming and subjective. Such guide efforts are particularly difficult for large-scale, multinational enterprise organizations that require income forecasts throughout a variety of product teams and geographical areas at a number of ranges of granularity. This requires not solely accuracy but in addition hierarchical coherence of the forecasts.
Bosch is a multinational company with entities working in a number of sectors, together with automotive, industrial options, and client items. Given the affect of correct and coherent income forecasting on wholesome enterprise operations, the Bosch Heart for Synthetic Intelligence (BCAI) has been closely investing in the usage of machine studying (ML) to enhance the effectivity and accuracy of economic planning processes. The purpose is to alleviate the guide processes by offering cheap baseline income forecasts through ML, with solely occasional changes wanted by the monetary analysts utilizing their business and area data.
To attain this purpose, BCAI has developed an inside forecasting framework able to offering large-scale hierarchical forecasts through custom-made ensembles of a variety of base fashions. A meta-learner selects the best-performing fashions based mostly on options extracted from every time collection. The forecasts from the chosen fashions are then averaged to acquire the aggregated forecast. The architectural design is modularized and extensible by way of the implementation of a REST-style interface, which permits steady efficiency enchancment through the inclusion of extra fashions.
BCAI partnered with the Amazon ML Options Lab (MLSL) to include the most recent advances in deep neural community (DNN)-based fashions for income forecasting. Current advances in neural forecasters have demonstrated state-of-the-art efficiency for a lot of sensible forecasting issues. In comparison with conventional forecasting fashions, many neural forecasters can incorporate extra covariates or metadata of the time collection. We embody CNN-QR and DeepAR+, two off-the-shelf fashions in Amazon Forecast, in addition to a customized Transformer mannequin educated utilizing Amazon SageMaker. The three fashions cowl a consultant set of the encoder backbones usually utilized in neural forecasters: convolutional neural community (CNN), sequential recurrent neural community (RNN), and transformer-based encoders.
One of many key challenges confronted by the BCAI-MLSL partnership was to supply sturdy and cheap forecasts beneath the affect of COVID-19, an unprecedented world occasion inflicting nice volatility on world company monetary outcomes. As a result of neural forecasters are educated on historic knowledge, the forecasts generated based mostly on out-of-distribution knowledge from the extra unstable durations might be inaccurate and unreliable. Subsequently, we proposed the addition of a masked consideration mechanism within the Transformer structure to deal with this difficulty.
The neural forecasters might be bundled as a single ensemble mannequin, or included individually into Bosch’s mannequin universe, and accessed simply through REST API endpoints. We suggest an method to ensemble the neural forecasters by way of backtest outcomes, which gives aggressive and sturdy efficiency over time. Moreover, we investigated and evaluated quite a few classical hierarchical reconciliation strategies to make sure that forecasts combination coherently throughout product teams, geographies, and enterprise organizations.
On this put up, we display the next:
- How you can apply Forecast and SageMaker customized mannequin coaching for hierarchical, large-scale time-series forecasting issues
- How you can ensemble customized fashions with off-the-shelf fashions from Forecast
- How you can scale back the affect of disruptive occasions similar to COVID-19 on forecasting issues
- How you can construct an end-to-end forecasting workflow on AWS
Challenges
We addressed two challenges: creating hierarchical, large-scale income forecasting, and the affect of the COVID-19 pandemic on long-term forecasting.
Hierarchical, large-scale income forecasting
Monetary analysts are tasked with forecasting key monetary figures, together with income, operational prices, and R&D expenditures. These metrics present enterprise planning insights at totally different ranges of aggregation and allow data-driven decision-making. Any automated forecasting resolution wants to supply forecasts at any arbitrary stage of business-line aggregation. At Bosch, the aggregations might be imagined as grouped time collection as a extra basic type of hierarchical construction. The next determine reveals a simplified instance with a two-level construction, which mimics the hierarchical income forecasting construction at Bosch. The overall income is cut up into a number of ranges of aggregations based mostly on product and area.
The overall variety of time collection that must be forecasted at Bosch is on the scale of thousands and thousands. Discover that the top-level time collection might be cut up by both merchandise or areas, creating a number of paths to the underside stage forecasts. The income must be forecasted at each node within the hierarchy with a forecasting horizon of 12 months into the longer term. Month-to-month historic knowledge is obtainable.
The hierarchical construction might be represented utilizing the next type with the notation of a summing matrix S (Hyndman and Athanasopoulos):
On this equation, Y equals the next:
Right here, b represents the underside stage time-series at time t.
Impacts of the COVID-19 pandemic
The COVID-19 pandemic introduced vital challenges for forecasting as a result of its disruptive and unprecedented results on nearly all features of labor and social life. For long-term income forecasting, the disruption additionally introduced surprising downstream impacts. For example this downside, the next determine reveals a pattern time collection the place the product income skilled a major drop at first of the pandemic and regularly recovered afterwards. A typical neural forecasting mannequin will take income knowledge together with the out-of-distribution (OOD) COVID interval because the historic context enter, in addition to the bottom fact for mannequin coaching. Consequently, the forecasts produced are not dependable.
Modeling approaches
On this part, we talk about our varied modeling approaches.
Amazon Forecast
Forecast is a fully-managed AI/ML service from AWS that gives preconfigured, state-of-the-art time collection forecasting fashions. It combines these choices with its inside capabilities for automated hyperparameter optimization, ensemble modeling (for the fashions offered by Forecast), and probabilistic forecast technology. This lets you simply ingest customized datasets, preprocess knowledge, prepare forecasting fashions, and generate sturdy forecasts. The service’s modular design additional allows us to simply question and mix predictions from extra customized fashions developed in parallel.
We incorporate two neural forecasters from Forecast: CNN-QR and DeepAR+. Each are supervised deep studying strategies that prepare a worldwide mannequin for your complete time collection dataset. Each CNNQR and DeepAR+ fashions can absorb static metadata details about every time collection, that are the corresponding product, area, and enterprise group in our case. Additionally they robotically add temporal options similar to month of the yr as a part of the enter to the mannequin.
Transformer with consideration masks for COVID
The Transformer structure (Vaswani et al.), initially designed for pure language processing (NLP), lately emerged as a preferred architectural alternative for time collection forecasting. Right here, we used the Transformer structure described in Zhou et al. with out probabilistic log sparse consideration. The mannequin makes use of a typical structure design by combining an encoder and a decoder. For income forecasting, we configure the decoder to straight output the forecast of the 12-month horizon as a substitute of producing the forecast month by month in an autoregressive method. Based mostly on the frequency of the time collection, extra time associated options similar to month of the yr are added because the enter variable. Extra categorical variables describing the meta info (product, area, enterprise group) are fed into the community through a trainable embedding layer.
The next diagram illustrates the Transformer structure and the eye masking mechanism. Consideration masking is utilized all through all of the encoder and decoder layers, as highlighted in orange, to forestall OOD knowledge from affecting the forecasts.
We mitigate the affect of OOD context home windows by including consideration masks. The mannequin is educated to use little or no consideration to the COVID interval that incorporates outliers through masking, and performs forecasting with masked info. The eye masks is utilized all through each layer of the decoder and encoder structure. The masked window might be both specified manually or by way of an outlier detection algorithm. Moreover, when utilizing a time window containing outliers because the coaching labels, the losses should not back-propagated. This consideration masking-based technique might be utilized to deal with disruptions and OOD instances introduced by different uncommon occasions and enhance the robustness of the forecasts.
Mannequin ensemble
Mannequin ensemble usually outperforms single fashions for forecasting—it improves mannequin generalizability and is healthier at dealing with time collection knowledge with various traits in periodicity and intermittency. We incorporate a collection of mannequin ensemble methods to enhance mannequin efficiency and robustness of forecasts. One widespread type of deep studying mannequin ensemble is to combination outcomes from mannequin runs with totally different random weight initializations, or from totally different coaching epochs. We make the most of this technique to receive forecasts for the Transformer mannequin.
To additional construct an ensemble on high of various mannequin architectures, similar to Transformer, CNNQR, and DeepAR+, we use a pan-model ensemble technique that selects the top-k greatest performing fashions for every time collection based mostly on the backtest outcomes and procure their averages. As a result of backtest outcomes might be exported straight from educated Forecast fashions, this technique allows us to reap the benefits of turnkey companies like Forecast with enhancements gained from customized fashions similar to Transformer. Such an end-to-end mannequin ensemble method doesn’t require coaching a meta-learner or calculating time collection options for mannequin choice.
Hierarchical reconciliation
The framework is adaptive to include a variety of strategies as postprocessing steps for hierarchical forecast reconciliation, together with bottom-up (BU), top-down reconciliation with forecasting proportions (TDFP), atypical least sq. (OLS), and weighted least sq. (WLS). All of the experimental outcomes on this put up are reported utilizing top-down reconciliation with forecasting proportions.
Structure overview
We developed an automatic end-to-end workflow on AWS to generate income forecasts using companies together with Forecast, SageMaker, Amazon Easy Storage Service (Amazon S3), AWS Lambda, AWS Step Features, and AWS Cloud Growth Package (AWS CDK). The deployed resolution gives particular person time collection forecasts by way of a REST API utilizing Amazon API Gateway, by returning the ends in predefined JSON format.
The next diagram illustrates the end-to-end forecasting workflow.
Key design concerns for the structure are versatility, efficiency, and user-friendliness. The system ought to be sufficiently versatile to include a various set of algorithms throughout improvement and deployment, with minimal required adjustments, and might be simply prolonged when including new algorithms sooner or later. The system must also add minimal overhead and assist parallelized coaching for each Forecast and SageMaker to scale back coaching time and procure the most recent forecast sooner. Lastly, the system ought to be easy to make use of for experimentation functions.
The tip-to-end workflow sequentially runs by way of the next modules:
- A preprocessing module for knowledge reformatting and transformation
- A mannequin coaching module incorporating each the Forecast mannequin and customized mannequin on SageMaker (each are working in parallel)
- A postprocessing module supporting mannequin ensemble, hierarchical reconciliation, metrics, and report technology
Step Features organizes and orchestrates the workflow from finish to finish as a state machine. The state machine run is configured with a JSON file containing all the required info, together with the placement of the historic income CSV recordsdata in Amazon S3, the forecast begin time, and mannequin hyperparameter settings to run the end-to-end workflow. Asynchronous calls are created to parallelize mannequin coaching within the state machine utilizing Lambda features. All of the historic knowledge, config recordsdata, forecast outcomes, in addition to intermediate outcomes similar to backtesting outcomes are saved in Amazon S3. The REST API is constructed on high of Amazon S3 to supply a queryable interface for querying forecasting outcomes. The system might be prolonged to include new forecast fashions and supporting features similar to producing forecast visualization studies.
Analysis
On this part, we element the experiment setup. Key elements embody the dataset, analysis metrics, backtest home windows, and mannequin setup and coaching.
Dataset
To guard the monetary privateness of Bosch whereas utilizing a significant dataset, we used an artificial dataset that has comparable statistical traits to a real-world income dataset from one enterprise unit at Bosch. The dataset incorporates 1,216 time collection in complete with income recorded in a month-to-month frequency, protecting January 2016 to April 2022. The dataset is delivered with 877 time collection on the most granular stage (backside time collection), with a corresponding grouped time collection construction represented as a summing matrix S. Every time collection is related to three static categorical attributes, which corresponds to product class, area, and organizational unit in the actual dataset (anonymized within the artificial knowledge).
Analysis metrics
We use median-Imply Arctangent Absolute Proportion Error (median-MAAPE) and weighted-MAAPE to guage the mannequin efficiency and carry out comparative evaluation, that are the usual metrics used at Bosch. MAAPE addresses the shortcomings of the Imply Absolute Proportion Error (MAPE) metric generally utilized in enterprise context. Median-MAAPE offers an outline of the mannequin efficiency by computing the median of the MAAPEs calculated individually on every time collection. Weighted-MAAPE studies a weighted mixture of the person MAAPEs. The weights are the proportion of the income for every time collection in comparison with the aggregated income of your complete dataset. Weighted-MAAPE higher displays downstream enterprise impacts of the forecasting accuracy. Each metrics are reported on your complete dataset of 1,216 time collection.
Backtest home windows
We use rolling 12-month backtest home windows to match mannequin efficiency. The next determine illustrates the backtest home windows used within the experiments and highlights the corresponding knowledge used for coaching and hyperparameter optimization (HPO). For backtest home windows after COVID-19 begins, the result’s affected by OOD inputs from April to Could 2020, based mostly on what we noticed from the income time collection.
Mannequin setup and coaching
For Transformer coaching, we used quantile loss and scaled every time collection utilizing its historic imply worth earlier than feeding it into Transformer and computing the coaching loss. The ultimate forecasts are rescaled again to calculate the accuracy metrics, utilizing the MeanScaler carried out in GluonTS. We use a context window with month-to-month income knowledge from the previous 18 months, chosen through HPO within the backtest window from July 2018 to June 2019. Extra metadata about every time collection within the type of static categorical variables are fed into the mannequin through an embedding layer earlier than feeding it to the transformer layers. We prepare the Transformer with 5 totally different random weight initializations and common the forecast outcomes from the final three epochs for every run, in complete averaging 15 fashions. The 5 mannequin coaching runs might be parallelized to scale back coaching time. For the masked Transformer, we point out the months from April to Could 2020 as outliers.
For all Forecast mannequin coaching, we enabled automated HPO, which may choose the mannequin and coaching parameters based mostly on a user-specified backtest interval, which is about to the final 12 months within the knowledge window used for coaching and HPO.
Experiment outcomes
We prepare masked and unmasked Transformers utilizing the identical set of hyperparameters, and in contrast their efficiency for backtest home windows instantly after COVID-19 shock. Within the masked Transformer, the 2 masked months are April and Could 2020. The next desk reveals the outcomes from a collection of backtest durations with 12-month forecasting home windows ranging from June 2020. We are able to observe that the masked Transformer persistently outperforms the unmasked model.
We additional carried out analysis on the mannequin ensemble technique based mostly on backtest outcomes. Particularly, we examine the 2 instances when solely the highest performing mannequin is chosen vs. when the highest two performing fashions are chosen, and mannequin averaging is carried out by computing the imply worth of the forecasts. We examine the efficiency of the bottom fashions and the ensemble fashions within the following figures. Discover that not one of the neural forecasters persistently out-perform others for the rolling backtest home windows.
The next desk reveals that, on common, ensemble modeling of the highest two fashions offers one of the best efficiency. CNNQR gives the second-best consequence.
Conclusion
This put up demonstrated learn how to construct an end-to-end ML resolution for large-scale forecasting issues combining Forecast and a customized mannequin educated on SageMaker. Relying on your online business wants and ML data, you should use a totally managed service similar to Forecast to dump the construct, prepare, and deployment strategy of a forecasting mannequin; construct your customized mannequin with particular tuning mechanisms with SageMaker; or carry out mannequin ensembling by combining the 2 companies.
If you want assist accelerating the usage of ML in your services, please contact the Amazon ML Options Lab program.
References
Hyndman RJ, Athanasopoulos G. Forecasting: rules and observe. OTexts; 2018 Could 8.
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Consideration is all you want. Advances in neural info processing programs. 2017;30.
Zhou H, Zhang S, Peng J, Zhang S, Li J, Xiong H, Zhang W. Informer: Past environment friendly transformer for lengthy sequence time-series forecasting. InProceedings of AAAI 2021 Feb 2.
In regards to the Authors
Goktug Cinar is a lead ML scientist and the technical lead of the ML and stats-based forecasting at Robert Bosch LLC and Bosch Heart for Synthetic Intelligence. He leads the analysis of the forecasting fashions, hierarchical consolidation, and mannequin mixture strategies in addition to the software program improvement group which scales these fashions and serves them as a part of the interior end-to-end monetary forecasting software program.
Michael Binder is a product proprietor at Bosch International Providers, the place he coordinates the event, deployment and implementation of the corporate large predictive analytics utility for the large-scale automated knowledge pushed forecasting of economic key figures.
Adrian Horvath is a Software program Developer at Bosch Heart for Synthetic Intelligence, the place he develops and maintains programs to create predictions based mostly on varied forecasting fashions.
Panpan Xu is a Senior Utilized Scientist and Supervisor with the Amazon ML Options Lab at AWS. She is engaged on analysis and improvement of Machine Studying algorithms for high-impact buyer purposes in a wide range of industrial verticals to speed up their AI and cloud adoption. Her analysis curiosity consists of mannequin interpretability, causal evaluation, human-in-the-loop AI and interactive knowledge visualization.
Jasleen Grewal is an Utilized Scientist at Amazon Internet Providers, the place she works with AWS clients to resolve actual world issues utilizing machine studying, with particular concentrate on precision medication and genomics. She has a robust background in bioinformatics, oncology, and scientific genomics. She is captivated with utilizing AI/ML and cloud companies to enhance affected person care.
Selvan Senthivel is a Senior ML Engineer with the Amazon ML Options Lab at AWS, specializing in serving to clients on machine studying, deep studying issues, and end-to-end ML options. He was a founding engineering lead of Amazon Comprehend Medical and contributed to the design and structure of a number of AWS AI companies.
Ruilin Zhang is an SDE with the Amazon ML Options Lab at AWS. He helps clients undertake AWS AI companies by constructing options to deal with widespread enterprise issues.
Shane Rai is a Sr. ML Strategist with the Amazon ML Options Lab at AWS. He works with clients throughout a various spectrum of industries to resolve their most urgent and revolutionary enterprise wants utilizing AWS’s breadth of cloud-based AI/ML companies.
Lin Lee Cheong is an Utilized Science Supervisor with the Amazon ML Options Lab group at AWS. She works with strategic AWS clients to discover and apply synthetic intelligence and machine studying to find new insights and clear up complicated issues.
