Abstract
Recent advances in large language models (LLMs) and their integration into user-facing applications have introduced novel forms of cyber threat surfaces beyond traditional software vulnerabilities. This paper explores a conceptual framework for understanding adaptive composition attacks a class of cybersecurity threats that emerge not from individual system weaknesses, but from the unintended interactions between secure components. Specifically, the paper examines scenarios where LLMs, equipped with execution permissions and access to peripheral systems such as email clients or operating system interfaces, are manipulated to iteratively improve social engineering attacks through a feedback loop mechanism. This phenomenon is modeled as a self-adaptive hacking loop, where one instance of an LLM assists in the generation, evaluation, and refinement of attack vectors that target another LLM or the same system recursively. Existing literature has addressed phishing generation by LLMs (Begou et al., 2023; Heiding et al., 2024) and prompt injection vulnerabilities (Wang et al., 2023), yet current frameworks fail to account for complex interactions where permission delegation, system trust composition, and model self-coordination coalesce into emergent attack behaviors. This paper introduces a theoretical construct for composition-based threat modeling and outlines the potential for LLMs to act not merely as passive generators of malicious content, but as coordinated attackers capable of exploiting their own operational environments. The study further identifies the absence of experimental environments that simulate these interdependent dynamics and highlights the need for new defense paradigms resilient to emergent compositional threats. The findings advocate for a shift from isolated security assessments to a holistic analysis of AI-integrated ecosystems, emphasizing the role of adaptive behavior and internal coordination in future cybersecurity challenges
Introduction
The field of artificial intelligence (AI) has undergone rapid evolution in the past decade, driven largely by the emergence of large language models (LLMs) such as OpenAI's GPT series, Google's Gemini, Meta's LLaMA, and Anthropic's Claude. These models, trained on massive datasets and designed to understand and generate human-like language, have significantly expanded the capabilities of AI systems in both consumer and enterprise applications. The increasing sophistication of LLMs has enabled them to perform complex tasks including code generation, document summarization, data extraction, and conversational reasoning across various domains (Brown et al., 2020; OpenAI, 2023) Beyond natural language processing, LLMs are now being integrated into automation pipelines with system-level access, enabling users to execute real-world operations such as sending emails, managing files, accessing cloud APIs, and interfacing with operating systems. This integration often facilitated through agents or tools like Microsoft’s Copilot, ChatGPT's Code Interpreter, and Anthropic’s MCP (Model-Controlled Programming) architecture represents a significant shift from passive dialogue systems to interactive execution agents. In such systems, LLMs are not only reasoning engines but also control surfaces for computational tasks, effectively becoming mediators between user intent and system execution This development has simultaneously introduced a new dimension of cybersecurity risks. Whereas traditional phishing attacks rely on human-crafted deception, recent studies have shown that LLMs can autonomously generate persuasive phishing emails and tailor them to specific user contexts (Begou et al., 2023; Heiding et al., 2024). Furthermore, AI-generated phishing content has been demonstrated to match or even exceed the effectiveness of human-crafted attacks in both general and targeted scenarios. Such findings highlight a growing concern regarding the misuse of generative AI in social engineering attacks and threat amplification
However, the problem extends beyond generative capabilities. A more complex and underexplored threat emerges when LLMs are embedded within multi-component systems that grant them varying degrees of control over other services and software layers. In such environments, LLMs may interact with each other or with system resources in unpredictable ways. Of particular concern is the scenario where one AI agent is used to craft a malicious payload—such as a phishing email or executable command—while another AI agent, acting within the same system or across linked services, interprets and acts on that payload. This interaction can form a self-adaptive hacking loop, whereby the attacking agent iteratively refines the exploit based on observed behavior or feedback from the target agent, leading to a recursive optimization of attack vectors This paper posits that such threats are not the result of flaws in individual software components, but arise from the compositional vulnerabilities inherent in modern AI-integrated ecosystems. These vulnerabilities are often invisible to conventional threat models, which typically analyze components in isolation. By contrast, the threat landscape is now shaped by dynamic interaction patterns between intelligent agents and system environments. The core hypothesis of this study is that AI systems, under certain conditions of trust, permission, and system access, may be manipulated intentionally or inadvertently into assisting in the compromise of their own operating context Through a theoretical analysis, this paper seeks to explore the structure, dynamics, and potential consequences of such compositional threats. It introduces a new framework for evaluating AI-powered attack compositions and proposes a shift in cybersecurity paradigms one that emphasizes not just the security of individual tools, but the emergent risks arising from their coordinated use within complex, AI-enabled environments