Deep Reinforcement Learning-Driven Engineering Change Sequencing for Complex Product Development Programs
Article Sidebar
Main Article Content
Abstract
The order of the Engineering changes (ECs) that are acted upon in complex product development programs is tightly coupled to the ordering of the component, the parameters and the activities, and this ordering significantly affects rework, cost, and lead time. This systematic review combines 24 peer-reviewed papers from four research streams: design structure matrix (DSM) and process architecture, engineering change management and propagation, deep reinforcement learning (DRL) foundations and DRL-based production scheduling. The evidence is used to build EC sequencing as a Markov decision process with the propagation indices generated from DSM as the state, change or sequencing rules as the action, and penalties for cost growth, schedule growth and triggered rework as the reward. The synthesis revealed that changes require between 1/3 and 1/2 of engineering time, 48.2% of the 41,500 change requests that were raised in an 8-year period for a sensor program were implemented, and changes were propagated no further than three engineering interfaces. In learning, a deep Q-network solved 43 of 49 benchmark games while previous methods were unable to solve the same number of games. A search-augmented network architecture beat other algorithms in 99.8% of games in the benchmark. The results show that rule-selecting deep Q-network agents achieve the best trade-off between scalability and verifiability, while search-based planners have a much greater look-ahead capability but at a significantly higher computational cost. The simulator fidelity, limited change histories, explainability and certification traceability are key challenges. A reference architecture and a research agenda are suggested for the development of the transfer of DRL from production scheduling to engineering change control.
Article Details
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
References
[1] Y. Bengio, A. Lodi, and A. Prouvost, “Machine learning for combinatorial optimization: A methodological tour d’horizon,” Eur. J. Oper. Res., vol. 290, no. 2, pp. 405–421, 2021, doi: https://doi.org/10.1016/j.ejor.2020.07.063.
[2] T. R. Browning, “Applying the design structure matrix to system decomposition and integration problems: a review and new directions,” IEEE Trans. Eng. Manag., vol. 48, no. 3, pp. 292–306, 2001, doi: 10.1109/17.946528.
[3] T. R. Browning, “Design Structure Matrix Extensions and Innovations: A Survey and New Opportunities,” IEEE Trans. Eng. Manag., vol. 63, no. 1, pp. 27–52, 2016, doi: 10.1109/TEM.2015.2491283.
[4] K. Arulkumaran, M. Deisenroth, M. Brundage, and A. Bharath, “Deep Reinforcement Learning: A Brief Survey,” IEEE Signal Process. Mag., vol. 34, no. 6, pp. 26–38, Aug. 2017, doi: 10.1109/MSP.2017.2743240.
[5] T. R. Browning and S. D. Eppinger, “Modeling impacts of process architecture on cost and schedule risk in product development,” IEEE Trans. Eng. Manag., vol. 49, no. 4, pp. 428–442, 2002, doi: 10.1109/TEM.2002.806709.
[6] P. J. Clarkson, C. Simons, and C. Eckert, “Predicting Change Propagation in Complex Design ,” J. Mech. Des., vol. 126, no. 5, pp. 788–797, Oct. 2004, doi: 10.1115/1.1765117.
[7] C. Eckert, P. J. Clarkson, and W. Zanker, “Change and customisation in complex engineering domains,” Res. Eng. Des., vol. 15, no. 1, pp. 1–21, 2004, doi: 10.1007/s00163-003-0031-7.
[8] M. Giffin, O. de Weck, G. Bounova, R. Keller, C. Eckert, and P. J. Clarkson, “Change Propagation Analysis in Complex Technical Systems,” J. Mech. Des., vol. 131, no. 8, Jul. 2009, doi: 10.1115/1.3149847.
[9] B. Hamraz, N. H. M. Caldwell, and P. John Clarkson, “A Multidomain Engineering Change Propagation Model to Support Uncertainty Reduction and Risk Management in Design,” J. Mech. Des., vol. 134, no. 10, Sep. 2012, doi: 10.1115/1.4007397.
[10] B. Hamraz, N. H. M. Caldwell, and P. J. Clarkson, “A Holistic Categorization Framework for Literature on Engineering Change Management,” Syst. Eng., vol. 16, no. 4, pp. 473–505, Dec. 2013, doi: https://doi.org/10.1002/sys.21244.
[11] T. A. W. Jarratt, C. M. Eckert, N. H. M. Caldwell, and P. J. Clarkson, “Engineering change: an overview and perspective on the literature,” Res. Eng. Des., vol. 22, no. 2, pp. 103–124, 2011, doi: 10.1007/s00163-010-0097-y.
[12] B. M. Kayhan and G. Yildiz, “Reinforcement learning applications to machine scheduling problems: a comprehensive literature review,” J. Intell. Manuf., vol. 34, no. 3, pp. 905–929, 2023, doi: 10.1007/s10845-021-01847-3.
[13] E. C. Y. Koh, N. H. M. Caldwell, and P. J. Clarkson, “A method to assess the effects of engineering change propagation,” Res. Eng. Des., vol. 23, no. 4, pp. 329–351, 2012, doi: 10.1007/s00163-012-0131-3.
[14] S. Luo, “Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning,” Appl. Soft Comput., vol. 91, p. 106208, 2020, doi: https://doi.org/10.1016/j.asoc.2020.106208.
[15] N. Mazyavkina, S. Sviridov, S. Ivanov, and E. Burnaev, “Reinforcement learning for combinatorial optimization: A survey,” Comput. Oper. Res., vol. 134, p. 105400, 2021, doi: https://doi.org/10.1016/j.cor.2021.105400.
[16] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015, doi: 10.1038/nature14236.
[17] B. Morkos, P. Shankar, and J. D. Summers, “Predicting requirement change propagation, using higher order design structure matrices: an industry case study,” J. Eng. Des., vol. 23, no. 12, pp. 905–926, Dec. 2012, doi: 10.1080/09544828.2012.662273.
[18] M. Panzer and B. Bender, “Deep reinforcement learning in production systems: a systematic literature review,” Int. J. Prod. Res., vol. 60, no. 13, pp. 4316–4341, Jul. 2022, doi: 10.1080/00207543.2021.1973138.
[19] M. C. Pasqual and O. L. de Weck, “Multilayer network model for analysis and management of change propagation,” Res. Eng. Des., vol. 23, no. 4, pp. 305–328, 2012, doi: 10.1007/s00163-011-0125-6.
[20] K. Rouibah and K. R. Caskey, “Change management in concurrent engineering from a parameter perspective,” Comput. Ind., vol. 50, no. 1, pp. 15–34, 2003, doi: https://doi.org/10.1016/S0166-3615(02)00138-0.
[21] D. Silver et al., “Mastering the game of Go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, pp. 484–489, 2016, doi: 10.1038/nature16961.
[22] L. Wang, Z. Pan, and J. Wang, “A Review of Reinforcement Learning Based Intelligent Optimization for Manufacturing Scheduling,” Complex Syst. Model. Simul., vol. 1, no. 4, pp. 257–270, 2021, doi: 10.23919/CSMS.2021.0027.
[23] B. Waschneck et al., “Optimization of global production scheduling with deep reinforcement learning,” Procedia CIRP, vol. 72, pp. 1264–1269, 2018, doi: https://doi.org/10.1016/j.procir.2018.03.212.
[24] A. Yassine and D. Braha, “Complex Concurrent Engineering and the Design Structure Matrix Method,” Concurr. Eng., vol. 11, no. 3, pp. 165–176, Sep. 2003, doi: 10.1177/106329303034503.