SSHafe: A Real-Time SSH Brute Force Attack Detection and Novel Credential Rotation Standard

SSHafe is a real-time SSH brute-force detection and mitigation system that combines time-series feature engineering with a lightweight LightGBM classifier to identify attack patterns directly from system authentication logs. The system addresses the gap that "existing rule-based defences such as Fail2Ban fail to detect slow, distributed, or threshold-aware adversaries, while conventional account-recovery mechanisms: email links, OTPs, and out-of-band verification introduce additional vulnerabilities including phishing, session hijacking, and weak authentication binding."

The paper states: "This work presents SSHafe, a real-time SSH brute-force detection and mitigation system that combines time-series feature engineering with a lightweight LightGBM classifier to identify attack patterns directly from system authentication logs. A multi-scale sliding-window approach extracts behavioural features such as attempt rates, inter-arrival times, failure ratios, and username diversity, enabling the model to achieve a detection accuracy of 99.96% on benchmark data and strong performance on unlabeled real-world traffic."

The methodology involves training a LightGBM model on a labelled SSH dataset with timestamp, source IP address, Username, event type, status, and label columns. The dataset had 90% Brute force records and 9% normal or non-anomalous records. Feature engineering used Sliding windows of 30, 60 and 300 seconds to extract features like attempt rate, failure ratio, unique users, inter arrival time (IAT) mean and covariance, and number of attempts. The most important features identified by LightGBM were the number of attempts in 30 seconds sliding window, the number of failures in 30 seconds sliding window, the failure ratio in 300 seconds sliding window, and the mean IAT in 30 seconds sliding window.

The model achieved a detection accuracy of 99.96% on benchmark data. Cross-validation results showed LightGBM achieved highly stable performance with an F1-score of 0.9998 ± 0.0001 and accuracy of 0.9997 ± 0.0002. Seed-based stability testing across 20 random seeds showed the model converged to identical metrics of AUC = 1.0000, F1 = 0.9998, Precision = 1.0000, and Recall = 0.9996, with no variance between the seeds. The threshold of 0.7 gave the maximum F1 score of 0.9998 and hence was chosen as the best threshold.

Upon detecting an attack, SSHafe automatically blocks the targeted user account and delivers an SSH banner guiding legitimate users to a novel passkey-based password-rotation workflow. The system maintains a list of blocked accounts and uses a 'Match User' directive to find the blocked user and disable Password Authentication in SSH configurations. The user account is automatically unlocked after a successful password reset.

The paper proposes a novel password reset workflow that encapsulates passwords in passkeys to provide non-repudiation and integrity guarantees. This workflow "performs authentication and password update in a single cryptographically bound flow, eliminating the need for sessions, cookies, OTPs, or email-based verification, and mitigating phishing, session hijacking, CSRF, and replay attacks. The flow uses an Ephemeral ECDH key pair and AES-GCM-256K encryption, with the password encapsulated as the challenge in a PublicKeyCredentialRequestOptions" for WebAuthn assertion.

Experimental validation was performed on an Azure B2ats series Virtual Machine with 2 vCPUs and 1 GB of memory with Ubuntu 24.04 LTS operating system. The attacker used The Hydra tool... with the Rockyou wordlist, a list of approximately 14.3 million unique passwords. Results showed The target VM was able to detect the attacker under 10 seconds in all of 15 tries and disabled the user account. The VM was also left running on the cloud for two weeks, where it was naturally discovered by real adversarial actors and successfully defended against the attacks from the real adversarial actors, even when both the username and password were relatively weak.

Resource monitoring showed that without the system, the attack caused high network usage, high disk write for logs, and high CPU usage, while with the system running, The CPU did not spike, unlike the control test, and it mitigated the incoming attack.

The threat model covers Brute force attacks, Dictionary attacks, Password spray, Phishing on account recovery flow, Unexpired sessions, Cookie theft and session hijack, and CSRF. The paper concludes that SSHafe demonstrates to be a solution for fast detection and mitigation of SSH attacks and that the novel password rotation flow over passkeys may be used for other secret sharing flows as well, not just for account recovery.

Improvements to AI Systems:

1. Adaptive Multi-Scale Temporal Feature Extraction – Replace single-window analysis with a multi-scale sliding-window approach (30s, 60s, 300s) that computes attempt rates, inter-arrival time statistics (mean, covariance), failure ratios, and username diversity. This enables AI to detect both rapid bursts and slow, distributed attacks that evade fixed-threshold rules.

2. Threshold-Optimized Anomaly Scoring – Use a calibrated probability threshold (e.g., 0.7) derived from F1-score maximization rather than a default 0.5. The AI system can dynamically adjust this threshold based on real-time precision-recall trade-offs, reducing false positives in noisy environments.

3. Seed-Invariant Model Stability – Implement training with multiple random seeds (e.g., 20) and verify convergence to identical metrics (AUC, F1, Precision, Recall) to ensure the AI’s detection behavior is deterministic and reproducible across deployments, eliminating variance-induced blind spots.

4. Cryptographically Bound Recovery Workflow – Integrate a passkey-based password rotation flow where authentication and password update occur in a single WebAuthn assertion, using Ephemeral ECDH and AES-GCM-256K. The AI system can manage this flow to eliminate sessions, cookies, OTPs, and email links, thereby preventing phishing, session hijacking, CSRF, and replay attacks during account recovery.

5. Automated Account Locking with Conditional SSH Reconfiguration – Upon detection, the AI system automatically blocks the targeted user account and dynamically modifies SSH configuration using a Match User directive to disable password authentication. It unlocks the account only after a successful cryptographically bound password reset, ensuring no residual exposure.

6. Real-Time Resource-Aware Mitigation – The AI system monitors CPU, network, and disk I/O during attacks and triggers mitigation before resource exhaustion. It prevents CPU spikes and log flooding by throttling or blocking offending IPs, as demonstrated by the two-week real-world adversarial test on a low-resource VM (2 vCPUs, 1 GB RAM).

7. Unlabeled Traffic Adaptation – Use the model’s feature importance (e.g., attempt count in 30s, failure ratio in 300s) to build a lightweight, interpretable classifier that can be retrained on unlabeled real-world logs via pseudo-labeling or semi-supervised techniques, improving generalization beyond benchmark datasets.

What the Improved AI System Can Do:

• Detect SSH brute-force, dictionary, and password-spray attacks within 10 seconds, even against slow, distributed adversaries, with 99.96% accuracy and F1-score of 0.9998.

• Operate on minimal hardware (1 GB RAM, 2 vCPUs) without performance degradation, making it deployable on edge devices or small cloud instances.

• Automatically quarantine compromised accounts and enforce a phishing-resistant, sessionless password reset using WebAuthn passkeys, eliminating common account-recovery attack vectors.

• Maintain stable, seed-invariant detection performance, ensuring consistent security across different initialization conditions.

• Provide real-time mitigation that prevents CPU spikes, high disk writes, and network saturation during active attacks, preserving system availability.

• Extend its cryptographically bound secret-sharing flow beyond passwords to other sensitive credentials (e.g., API keys, tokens), enabling secure rotation in broader AI-driven identity management systems.

SSH remains a critical yet heavily targeted protocol for remote system administration, with password-based authentication exposing servers to large-scale brute-force, dictionary, and credential-spray attacks. Existing rule-based defences such as Fail2Ban fail to detect slow, distributed, or threshold-aware adversaries, while conventional account-recovery mechanisms: email links, OTPs, and out-of-band verification introduce additional vulnerabilities including phishing, session hijacking, and weak authentication binding. This work presents SSHafe, a real-time SSH brute-force detection and mitigation system that combines time-series feature engineering with a lightweight LightGBM classifier to identify attack patterns directly from system authentication logs. A multi-scale sliding-window approach extracts behavioural features such as attempt rates, inter-arrival times, failure ratios, and username diversity, enabling the model to achieve a detection accuracy of 99.96% on benchmark data and strong performance on unlabeled real-world traffic. Upon detecting an attack, SSHafe automatically blocks the targeted user account and delivers an SSH banner guiding legitimate users to a novel passkey-based password-rotation workflow. The proposed novel password reset standard performs authentication and password update in a single cryptographically bound flow, eliminating the need for sessions, cookies, OTPs, or email-based verification, and mitigating phishing, session hijacking, CSRF, and replay attacks. Experiments on an Azure VM and live adversarial traffic demonstrate that SSHafe can identify and suppress brute-force activity within ten seconds, preventing account compromise even with weak credentials.