The Role of GitLab Runners in CI/CD Pipelines: Configuring EC2, Docker, and Kubernetes Build Environments
Article Sidebar
Main Article Content
Abstract
This research studies GitLab Runners optimization in CI/CD pipelines across the EC2, Docker, and Kubernetes environment configurations. It shows that such key strategies for enhancing build performance and resource utilization would reduce build time by 65 percent and resource costs by 40 percent. Practical recommendations for configuring runners to achieve optimal efficiency are presented in the context of analyzing 200 enterprise pipelines. The key optimization techniques are autoscaling based on real-time metrics, advanced caching to minimize the rebuilds, and tuning the resource allocation to avoid over-provision. The study further looks into the capability of machine learning models to optimize the number of runners dynamically, predict the hit or not on the cache, and automatically pick up the execution environment. When these innovations are applied, CI / CD pipeline performance will improve by reducing idle resources, building time, and optimizing resource utilization. The paper shows that experts can achieve very good availability and cost efficiency by adapting the configuration of GitLab Runners. The research also discusses the evolution of automated environment selection and machine learning-based performance tuning. This framework serves as the base for organizations to increase their CI/CD pipeline development rate and facilitates a faster, more reliable, and cheaper software delivery.
Article Details
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
References
A. A. Zeeshan, A. A., & Zeeshan, “Securing Build Systems for DevOps. DevSecOps for. NET Core: Securing Modern Software Applications,” 163–214, 2020.
D. C. Winn, “Cloud Foundry: the cloud-native platform. ‘ O’Reilly Media, Inc.,’” 2016.
S. Nyati, “Revolutionizing LTL carrier operations: A comprehensive analysis of an algorithm-driven pickup and delivery dispatching solution. International Journal of Science and Research (IJSR), 7(2), 1659-1666. Retrieved from,” 2018.
A. R. Zhao, N., Tarasov, V., Albahar, H., Anwar, A., Rupprecht, L., Skourtis, D., ... & Butt, “Large-scale analysis of docker images and performance implications for container storage systems. IEEE Transactions on Parallel and Distributed Systems,” 32(4), 918–930, 2020.
G. Sharif, M., Janto, S., & Lueckemeyer, “Coaas: Continuous integration and delivery framework for hpc using gitlab-runner. In Proceedings of the 2020 4th International Conference on Big Data and Internet of Things (pp. 54-58).”
S. Chinamanagonda, “Automating Infrastructure with Infrastructure as Code (IaC). Available at SSRN 4986767,” 2019.
R. L. D. Santos, “Deploying and managing network services over programmable virtual networks,” 2018.
S. P. Matthias, K., & Kane, “Docker: Up & Running: Shipping Reliable Containers in Production. ‘ O’Reilly Media, Inc.,’” 2015.
B. Babar, M. A., & Ramsey, “Evaluating Security of Containerised Technologies for Building Private Cloud,” 2017.
J. Piscaer, “Kubernetes in the Enterprise,” URL: https://platform9. com/resource/the-gorilla-guide-to-kubernetes-in-theenterprise, 2018.
J. Cook, “Docker for data science: building scalable and extensible data infrastructure around the Jupyter notebook server,” 2017.
A. Crankshaw, D., Sela, G. E., Mo, X., Zumar, C., Stoica, I., Gonzalez, J., & Tumanov, “InferLine: latency-aware provisioning and scaling for prediction serving pipelines. In Proceedings of the 11th ACM Symposium on Cloud Computing (pp. 477-491).”
H. Adolfsson, “Comparison of auto-scaling policies using docker swarm,” 2019.
K. Rashmi, K. V., Chowdhury, M., Kosaian, J., Stoica, I., & Ramchandran, “{EC-Cache}:{Load-Balanced},{Low-Latency} Cluster Caching with Online Erasure Coding. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) (pp. 401-417),” 2016.
I. Di Natali, “Deploying a scalable API management platform in an enterprise Kubernetes-based environment (Doctoral dissertation, Politecnico di Torino),” 2020.
W. C. Mehmood, A., Muhammad, A., Khan, T. A., Rivera, J. J. D., Iqbal, J., Islam, I. U., & Song, “Energy-efficient auto-scaling of virtualized network function instances based on resource execution pattern. Computers & Electrical Engineering, 88, 106814,” 2020.
A. Kumar, “The convergence of predictive analytics in driving business intelligence and enhancing DevOps efficiency. International Journal of Computational Engineering and Management, 6(6), 118-142. Retrieved from,” 2019.
A. Orozco-GómezSerrano, “Adaptive Big Data Pipeline,” 2020.
Y. H. Liu, J., Chai, Y. P., Qin, X., & Liu, “Endurable SSD-based read cache for improving the performance of selective restore from deduplication systems. Journal of computer science and technology,” 33, 58–78, 2018.
Z. Li, Y., Zhang, J., Jiang, C., Wan, J., & Ren, “PINE: Optimizing performance isolation in container environments. IEEE Access, 7, 30410-30422,” 2019.
G. Herrera, J., & Moltó, “Toward bio-inspired auto-scaling algorithms: An elasticity approach for container orchestration platforms. IEEE Access, 8, 52139-52150,” 2020.
Z. Marahatta, A., Pirbhulal, S., Zhang, F., Parizi, R. M., Choo, K. K. R., & Liu, “Classification-based and energy-efficient dynamic task scheduling scheme for virtualized cloud data center. IEEE Transactions on Cloud Computing,” 9(4), 1376–1390, 2019.
R. Jung, “Platform and Methodology for Developing Modern Systems in Restricted Enterprise Environments, using Elixir/Erlang, Docker, CI/CD and Microservices,” 2018.
N. Shalev, “Improving system security and reliability with OS help. Research Thesis,” 2018.
A. Pihlak, “Continuous Docker Image Analysis and Intrusion Detection Based On Open-Source Tools,” 2020.
J. Ghasemisharif, M., Ramesh, A., Checkoway, S., Kanich, C., & Polakis, “O single {Sign-Off}, where art thou? an empirical analysis of single {Sign-On} account hijacking and session management on the web. In 27th USENIX Security Symposium (USENIX Security 18) (pp. 1475-1492),” 2018.
M. Jawed, “Continuous security in DevOps environment: Integrating automated security checks at each stage of continuous deployment pipeline (Doctoral dissertation, Wien),” 2019.
X. Chen, W., Rao, J., & Zhou, “Preemptive, low latency datacenter scheduling via lightweight virtualization. In 2017 USENIX Annual Technical Conference (USENIX ATC 17) (pp. 251-263),” 2017.
M. Moilanen, “Deploying an application using Docker and Kubernetes,” 2018.
M. G. Imdoukh, M., Ahmad, I., & Alfailakawi, “Machine learning-based auto-scaling for containerized applications. Neural Computing and Applications,” 32(13), 9745–9760, 2020.
P. Alonso, M., Coll, S., Martínez, J. M., Santonja, V., & López, “Power consumption management in fat-tree interconnection networks. Parallel computing,” 48, 59–80, 2015.
R. Zhong, Z., & Buyya, “A cost-efficient container orchestration strategy in kubernetes-based cloud computing infrastructures with heterogeneous resources. ACM Transactions on Internet Technology (TOIT), 20(2), 1-24,” 2020.
A. Schwanke, “Faculty Informatics Bachelor of Science–Business Information Systems,” 2019.
S. Grubor, “Deployment with Docker: Apply continuous integration models, deploy applications quicker, and scale at large by putting Docker to work. Packt Publishing Ltd,” 2017.
B. Alkadi, O., Moustafa, N., & Turnbull, “A review of intrusion detection and blockchain applications in the cloud: approaches, challenges and solutions. IEEE Access, 8, 104893-104917,” 2020.
R. Paganetti, “Building a Compliance Model: A Delphi Study of Managed Security Service Providers Governing Regulatory Compliance Successfully (Doctoral dissertation, Capella University),” 2020.
I. Nishihara, R., Moritz, P., Wang, S., Tumanov, A., Paul, W., Schleier-Smith, J., ... & Stoica, “Real-time machine learning: The missing pieces. In Proceedings of the 16th workshop on hot topics in operating systems (pp. 106-110).”
L. Erik, S., & Emma, “The Future of Software Development: AI-Driven Testing and Continuous Integration for Enhanced Reliability. International Journal of Trend in Scientific Research and Development, 2(4), 3082-3096,” 2018.
C. Karamitsos, I., Albarhami, S., & Apostolopoulos, “Applying DevOps practices of continuous automation for machine learning. Information, 11(7), 363,” 2020.
M. Karslioglu, “Kubernetes-A Complete DevOps Cookbook: Build and manage your applications, orchestrate containers, and deploy cloud-native services. Packt Publishing Ltd,” 2020.
A. Bansal, “System to redact personal identified entities (PII) in unstructured data. International Journal of Advanced Research in Engineering and Technology, 11(6), 133,” 2020.
C. W. Fuller, “Continuous Integration/Continuous Delivery Pipeline for Air Force Distributed Common Ground System (AF DCGS),” 2020.