tag: Llm-Security · 36 items
- Engineer — Learn: Novel research showing that maintaining pre/post unlearning model snapshots creates a recoverable side channel — relevant if you operate LLMs with approximate unlearning workflows. No patch exists; redesign signal is to avoid retaining both snapshots together or treat unlearning as non-binding from a data-exposure standpoint.
- SOC/IR — Skip
- Leader — Learn: Organizations using approximate LLM unlearning to satisfy GDPR right-to-erasure or similar data removal obligations may be overstating compliance guarantees — this research undermines the privacy claim of that approach. Worth flagging to DPO and AI governance leads if the company relies on unlearning in customer-data contexts.
- Engineer — Plan: If you deploy any of the named guards (Shieldstral, ShieldGemma, Llama Guard 3, Qwen3Guard, gpt-oss-safeguard) in a pipeline, audit whether your input preprocessing allows appended forged structure; the mitigation—retraining guards to score no lower than their most harmful fragment—cuts attack success from 75% to under 2%, so track upstream patch releases for these models.
- SOC/IR — Learn: Target rebinding is a structurally novel evasion class against LLM safety layers, but the paper reports no in-the-wild exploitation and the attack leaves no distinct log artifact to hunt; worth understanding as AI-application attacks become a detection domain.
- Leader — Learn: Research demonstrates that widely used LLM safety guards can be systematically bypassed, which is relevant if your organization ships AI features relying on these guards for policy enforcement; no immediate action required, but factor into your AI risk posture reviews this quarter.
- Engineer — Learn: Academic research on embedding robust provenance markers in LLM-generated code that survive identifier renaming and refactoring — relevant background if your team is evaluating AI code attribution or supply-chain traceability tooling, but no current system changes required.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Novel automated approach to reversing undocumented binary protocols using LLM reasoning plus taint-flow analysis — worth tracking for teams that need to analyze proprietary or malware-related protocol formats, but no shipping tool or immediate engineering action yet.
- SOC/IR — Learn: The framework’s demonstrated utility against real-world botnet C2 protocols is relevant to analysts who reverse-engineer malware traffic, but this is academic research without a released tool or ready-to-use detection artifacts today.
- Leader — Skip
- Engineer — Plan: Any team running open-weight LLMs in agent harnesses with shell access is in scope for this attack class. Before next sprint, enforce DNS egress blocking (or DNS query monitoring) on all LLM inference containers and require signed provenance for open-weight models pulled from public hubs.
- SOC/IR — Plan: This paper introduces a detection gap: base32-chunked subdomains in DNS queries from ML inference pods are the exfil channel. Write a detection rule (Sigma/KQL) for anomalous high-entropy subdomain queries originating from container workloads, and add LLM agent namespaces as a monitored DNS log source.
- Leader — Learn: Novel supply-chain risk class showing that downloaded AI model weights can carry persistent malware independent of prompts or sandboxing. Useful framing for an AI/LLM usage policy that requires internal vetting of open-weight models before agent deployment.
- Engineer — Learn: Novel black-box attack class against LLM inference systems using shared KV caches for reasoning trace reuse — engineers building or deploying such systems should audit whether their architecture exposes shared cache state across users or sessions, and treat this as a design constraint before enabling reasoning-cache reuse features.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: This research demonstrates that LLM backdoors can corrupt reasoning gradually across interaction stages, making accuracy-based monitoring insufficient — teams deploying fine-tuned or third-party models should evaluate whether their model-validation pipelines can detect staged behavioral drift, not just terminal outputs.
- SOC/IR — Learn: No IOCs or active campaign; the attack requires model-level access to plant, placing it outside typical SIEM/EDR detection scope today, but understanding this gradual-activation pattern is useful context if investigating anomalous LLM output in AI-assisted workflows.
- Leader — Learn: Academic research with no current exploitation signals, but the finding that reasoning corruption can precede visible accuracy drops is worth noting for AI governance discussions — particularly for organizations using externally fine-tuned models or LLM APIs where model provenance is opaque.
- Engineer — Learn: Research describing how attackers can decompose and aggregate requests to evade LLM safety classifiers is relevant for teams building AI safety pipelines; no patch or configuration action is indicated by the available signals.
- SOC/IR — Learn: Interesting evasion technique, but no IOCs, ATT&CK-mapped TTPs, or active campaign signals are present — nothing to hunt or tune detections against right now.
- Leader — Learn: Useful context on fundamental limitations of LLM safety controls for organizations deploying AI products; relevant to AI governance policy discussions but does not require near-term action.
- Engineer — Act: If you run any publicly exposed LLM or AI inference servers, audit them now for unauthorized crypto miner processes and verify no unauthenticated endpoints are reachable from the internet; 3,400+ confirmed infections indicates active, ongoing exploitation of this attack surface.
- SOC/IR — Plan: Build or tune detections for anomalous CPU/GPU utilization and suspicious outbound connections from AI/ML servers tied to the Canto Incognito campaign; no IOCs are published yet, but the behavioral signature of crypto miner deployment on LLM infrastructure is worth adding to your hunt library.
- Leader — Learn: AI and LLM infrastructure is now an active target for financially motivated botnets at scale — worth noting when reviewing security controls around AI deployments, but no immediate board or regulatory action required absent confirmed exposure.
- Engineer — Learn: A credible Linux kernel maintainer’s perspective on how LLMs affect software security and supply-chain trust is worth understanding for engineers integrating AI into development workflows, even without immediate action.
- SOC/IR — Skip
- Leader — Learn: GKH’s framing of LLM-era security risks from an authoritative open-source voice can inform AI-use policy discussions with leadership, though no immediate action is required.
- Engineer — Learn: Novel architectural approach that assigns ring IDs to tokens by origin, creating hard trust boundaries inside the model — relevant for teams building or evaluating LLM pipelines, but no deployable artifact or patch exists yet.
- SOC/IR — Learn: Reinforces that indirect prompt injection is a structural problem in current LLM deployments, useful context for analysts building detection logic around agentic or RAG-based systems, but no IOCs or hunt opportunities here.
- Leader — Learn: Confirms that LLM systems lack native separation between authoritative and non-authoritative inputs — useful framing when developing AI governance policy or evaluating vendor security claims around agentic products.
- Engineer — Learn: Research finding that fine-tuned LLMs exhibit false negative rates when validating proposed network intents against security policy — a design-relevant caution for teams evaluating LLM-based network automation or IBN tooling.
- SOC/IR — Skip
- Leader — Learn: Early evidence that LLM agents used for network policy automation can silently miss policy violations; useful context when evaluating AI-assisted infrastructure governance or setting guardrails for agentic tooling adoption.
- Engineer — Learn: New benchmark reveals that LLMs score well on recall and code tasks (~95-99%) but poorly on security judgment justification (~53%), relevant context if your team uses LLMs to audit cryptographic implementations or IoT firmware — don’t over-trust LLM-generated security verdicts without human review.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: If you pipe internal data through OpenAI or Anthropic APIs, this stateless proxy pattern is worth evaluating as a DLP layer; no active exploit pressure, but the architecture addresses a real secret-leakage risk in LLM-integrated pipelines.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Research framework for constructing auditable infrastructure dependency graphs using multiple LLMs with provenance tracking — worth evaluating if your team models CI/CD or cloud dependency chains, but no tooling is ready for production use today.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Honeypot research shows that untrusted ‘free’ LLM backends receive full coding-agent context — filesystem paths, conversation history, tool manifests — before any response is sent. Audit every LLM endpoint configured in your coding agents and ensure all traffic goes to verified, first-party providers.
- SOC/IR — Learn: Demonstrates a passive exfiltration path: coding agents silently send working paths and tool manifests to whatever endpoint they’re pointed at. No IOCs or active campaign here, but useful context for future detections around unexpected outbound HTTPS from dev tools to novel LLM API hosts.
- Leader — Plan: Employees using unofficial ‘free’ AI coding tools may be routing sensitive codebase context and filesystem details to unverified third parties; establish or enforce an approved-LLM-provider policy for coding agents this quarter before an incident forces a reactive response.
- Engineer — Learn: Academic framework for detecting model drift after deployment using privacy-preserving proofs; no running systems to patch today, but the black-box token-probe approach is worth tracking as LLM supply-chain integrity tooling matures.
- SOC/IR — Skip
- Leader — Learn: Offers a governance-relevant framing: proprietary LLMs can be silently altered post-approval, and cryptographic audit frameworks are emerging to address that gap — useful context for AI risk discussions with the board or auditors.
- Engineer — Learn: If your product integrates GPT-4o, Gemini, or similar multimodal models, this research shows existing content-safety wrappers are brittle against adaptive attackers; no patch exists yet, but it motivates evaluating your VLM endpoints against adaptive prompt-injection test suites.
- SOC/IR — Skip
- Leader — Learn: Research demonstrating high-success jailbreaks against GPT-4o and Gemini is useful framing for board-level AI risk discussions and for questioning AI vendor safety assurance claims when procuring or expanding VLM-based tooling.
- Engineer — Learn: Research identifies internal attention heads and MLP pathways responsible for safety bypass in LLaMA-2-7B — useful context when evaluating LLM safeguard architectures, but no operational change needed today and findings are on one specific model.
- SOC/IR — Skip
- Leader — Learn: Findings suggest current LLM safety alignment has exploitable structural weaknesses; relevant context when assessing risk posture of internally deployed LLM products, but no immediate action required.
- Engineer — Learn: Novel defense technique for federated fine-tuning pipelines; relevant if you run distributed LLM training with sensitive data, but no patch or configuration action needed today — research-stage only.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: If you rely on DP guarantees to protect training data in ML pipelines, this research shows that controlling memorization and controlling extraction are formally separate — a model can be memorized yet unextractable, or vice versa. Revisit your threat model assumptions, but no system change is required today.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Reinforces the design principle that LLM safety filters alone are insufficient; architecture decisions should place external guardrails (input/output validation, prompt firewalls) outside the model layer rather than trusting built-in refusals.
- SOC/IR — Skip
- Leader — Learn: Supports the case for defense-in-depth policy around AI deployments: if safety refusals are fragile by design, any AI system handling sensitive data needs external controls beyond the model’s built-in guardrails — useful framing for board or audit conversations about AI risk.
- Engineer — Learn: The findings — that safety alignment increases over-refusal (safety tax), privacy is near-orthogonal to other trustworthiness dimensions, and distillation degrades robustness — are useful mental models for engineers selecting or evaluating LLMs in their stack, though no immediate system changes are required.
- SOC/IR — Skip
- Leader — Learn: The finding that strong alignment does not protect privacy, and that distilled models suffer robustness collapse, provides empirical grounding for AI governance decisions and risk conversations with leadership about LLM adoption — useful for future board decks but no same-week action needed.
- Engineer — Learn: Novel architecture for parameter-level capability gating in MoE models (tested on Qwen3-30B and DeepSeek-V2-Lite); worth tracking if your team deploys or fine-tunes MoE-based models and needs verifiable separation between capability tiers — no production tooling yet.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Identifies a real supply-chain risk for teams consuming third-party LoRA adapters: a backdoored adapter can alter model output on hidden triggers without modifying base model weights. LoRAScan’s inference-time monitoring approach is worth evaluating if your ML pipelines pull adapters from untrusted registries or Hugging Face.
- SOC/IR — Learn: No active exploitation, IOCs, or ATT&CK-mappable TTPs to act on; this is foundational research on a threat class. Worth filing as context if your org is building detections around AI/ML pipeline integrity, but no hunt or rule work warranted today.
- Leader — Learn: Surfaces an emerging supply-chain risk category for AI workloads—untrusted fine-tuned adapters as a malware vector—useful background for shaping AI vendor-risk policy before it becomes a control requirement.
- Engineer — Learn: The hybrid deterministic-plus-LLM pipeline (regex/AST/topology plus LLM refinement) that roughly doubles vulnerability coverage over static rules alone is worth tracking as a design pattern for AppSec tooling, though the automotive ECU focus makes it directly applicable only in that niche.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Novel attack class: hidden payloads in pre-filled AI deep links can alter LLM memory without user awareness. No exploitation signals or PoC, but engineers building AI-integrated features should audit any ‘Ask AI’ button implementations for unsanitized prompt passthrough.
- SOC/IR — Learn: No IOCs, ATT&CK mapping, or active campaign indicators are present. Worth tracking as AI assistant adoption grows, but there is no detection surface or hunt query to act on today.
- Leader — Plan: This attack class is relevant to any enterprise deploying AI assistants with memory or context features; factor it into AI acceptable-use policy and vendor evaluation criteria before broader rollout.
- Engineer — Learn: Research shows that single-turn ASR benchmarks overstate real-world robustness of GUI agent guardrails, with 4-turn escalation chains recovering ~20 points of attack success across all tested models. Teams building or deploying GUI agents should treat static prompt-level alignment as insufficient and evaluate multi-turn threat scenarios in their safety testing.
- SOC/IR — Skip
- Leader — Learn: If your organization is piloting or deploying AI GUI agents, this research illustrates that current safety guardrails are weaker than benchmark numbers suggest under realistic multi-turn user interaction — useful context for AI deployment policies and vendor capability reviews, but no immediate action required.
- Engineer — Learn: Interesting research combining code slicing with LLM analysis to detect reentrancy and overflow in ERC-721 contracts, but no tooling release or actionable change to running systems today.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Novel prompt-suffix attack degrades speculative decoding throughput without corrupting outputs, affecting any deployment using draft-target inference acceleration (vLLM, TGI, etc.). No patch or mitigation exists yet; file this when designing LLM serving infrastructure to justify input validation and rate controls at the prompt layer.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Research shows output-only filters like Llama-Guard 3 are insufficient against reasoning-layer attacks; teams building AI applications should evaluate reasoning context, not just final outputs, when designing safety architectures.
- SOC/IR — Skip
- Leader — Learn: Finding that reasoning-capable models are 2x+ more vulnerable and standard output safeguards regularly fail has implications for enterprise AI risk posture; useful context for AI usage policies and vendor safety attestation reviews.
- Engineer — Learn: Research shows specific syntactic elements (constraints, guards, conditions) placed in prompts consistently reduce insecure code generation from open LLMs — useful input for teams building internal coding assistants or prompt templates for developer tooling.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: If you deploy LLM agents with skill files or tool orchestration, this research quantifies a real risk class: agents routinely violate preconditions and constraints, producing privacy leaks and unsafe config changes. No patch action today, but the SLGuard scaffold approach is worth evaluating if you build skill-guided agents.
- SOC/IR — Skip
- Leader — Learn: Academic evidence that LLM agents fail safety constraints at high rates is useful background for AI governance discussions, but there is no immediate vendor exposure or regulatory trigger here — file for the next AI risk policy review.
- Engineer — Skip
- SOC/IR — Learn: SherAgent demonstrates a 31–64% improvement in automated attack investigation success rates using LLM-driven provenance graph backtracking — useful context for teams evaluating or building AI-assisted triage workflows, though no production tool or IOCs are released here.
- Leader — Learn: Research from a real SOC environment shows LLM-assisted alert triage meaningfully reduces the manual investigation backlog; relevant background for leaders assessing AI tooling investments in detection and response.
- Engineer — Learn: Novel research showing an LLM-agentic pipeline that improves directed fuzzer crash-trigger rates by generating semantically aware seed corpora; worth evaluating if your team runs fuzzing campaigns against internal C/C++ codebases, but no immediate change to running systems is required.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Academic research on grounded agentic reasoning for malware behavior reconstruction; no immediate engineering action, but the tri-grounding approach (domain, semantics, knowledge) is worth noting when evaluating LLM-assisted code-analysis tooling.
- SOC/IR — Learn: Malaika’s behavior-reconstruction framing — connecting sparse program evidence to auditable behavioral conclusions — could inform how teams structure LLM-assisted malware triage workflows, though no detection or hunt action is available from this paper alone.
- Leader — Skip