Enhancing IDS performance through a comparative analysis of Random Forest, XGBoost, and Deep Neural Networks

Sow, Thierno Hamidou ORCID: https://orcid.org/0009-0004-0875-8649 et Mehdi, Adda (2025). Enhancing IDS performance through a comparative analysis of Random Forest, XGBoost, and Deep Neural Networks. Machine Learning with Applications, 22 (100738).

PDF Licence d'utilisation : CC BY-NC-ND 4.0. Télécharger (3MB) | Prévisualisation

Résumé

Intrusion Detection Systems (IDS) face major challenges in network security, notably the need to combine a high detection rate with reliable performance. This reliability is often affected by class imbalances and inadequate hyperparameter optimization. This article addresses the issue of improving the detection rate of IDS by evaluating and comparing three machine learning algorithms: Random Forest (RF), XGBoost, and Deep Neural Networks (DNN), using the NSL-KDD dataset. In our methodology, we integrate SMOTE (Synthetic Minority Oversampling Technique) to tackle the unbalanced nature of the data, ensuring a more balanced representation of the different classes. This approach helps optimize model performance, reduce bias, and enhance robustness. Additionally, hyperparameter optimization is performed using Optuna, ensuring that each algorithm operates at its optimal level. The results show that our model, using the Random Forest algorithm, achieves an accuracy of 99.80%, surpassing the performance of XGBoost and Deep Neural Networks (DNN). This makes our approach a true asset for intrusion detection methods in computer networks.

Type de document :	Article
Validation par les pairs :	Non
Mots-clés :	Cybersécurité ; Système de détection d'intrusion (IDS) ; Apprentissage automatique ; Apprentissage profond ; NSL-KDD ; Smote / Cybersecurity ; Intrusion Detection System (IDS) ; Machine learning ; Deep learning ; NSL-KDD ; Smote.
Départements et unités départementales :	Département de mathématiques, informatique et génie
Date de dépôt :	02 avr. 2026 18:54
Dernière modification :	02 avr. 2026 18:54
URI :	https://semaphore.uqar.ca/id/eprint/3533