tag: Llm-Security · 21 items
- Engineer — Learn: Honeypot research shows that untrusted ‘free’ LLM backends receive full coding-agent context — filesystem paths, conversation history, tool manifests — before any response is sent. Audit every LLM endpoint configured in your coding agents and ensure all traffic goes to verified, first-party providers.
- SOC/IR — Learn: Demonstrates a passive exfiltration path: coding agents silently send working paths and tool manifests to whatever endpoint they’re pointed at. No IOCs or active campaign here, but useful context for future detections around unexpected outbound HTTPS from dev tools to novel LLM API hosts.
- Leader — Plan: Employees using unofficial ‘free’ AI coding tools may be routing sensitive codebase context and filesystem details to unverified third parties; establish or enforce an approved-LLM-provider policy for coding agents this quarter before an incident forces a reactive response.
- Engineer — Learn: Academic framework for detecting model drift after deployment using privacy-preserving proofs; no running systems to patch today, but the black-box token-probe approach is worth tracking as LLM supply-chain integrity tooling matures.
- SOC/IR — Skip
- Leader — Learn: Offers a governance-relevant framing: proprietary LLMs can be silently altered post-approval, and cryptographic audit frameworks are emerging to address that gap — useful context for AI risk discussions with the board or auditors.
- Engineer — Learn: If your product integrates GPT-4o, Gemini, or similar multimodal models, this research shows existing content-safety wrappers are brittle against adaptive attackers; no patch exists yet, but it motivates evaluating your VLM endpoints against adaptive prompt-injection test suites.
- SOC/IR — Skip
- Leader — Learn: Research demonstrating high-success jailbreaks against GPT-4o and Gemini is useful framing for board-level AI risk discussions and for questioning AI vendor safety assurance claims when procuring or expanding VLM-based tooling.
- Engineer — Learn: Research identifies internal attention heads and MLP pathways responsible for safety bypass in LLaMA-2-7B — useful context when evaluating LLM safeguard architectures, but no operational change needed today and findings are on one specific model.
- SOC/IR — Skip
- Leader — Learn: Findings suggest current LLM safety alignment has exploitable structural weaknesses; relevant context when assessing risk posture of internally deployed LLM products, but no immediate action required.
- Engineer — Learn: Novel defense technique for federated fine-tuning pipelines; relevant if you run distributed LLM training with sensitive data, but no patch or configuration action needed today — research-stage only.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: If you rely on DP guarantees to protect training data in ML pipelines, this research shows that controlling memorization and controlling extraction are formally separate — a model can be memorized yet unextractable, or vice versa. Revisit your threat model assumptions, but no system change is required today.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Reinforces the design principle that LLM safety filters alone are insufficient; architecture decisions should place external guardrails (input/output validation, prompt firewalls) outside the model layer rather than trusting built-in refusals.
- SOC/IR — Skip
- Leader — Learn: Supports the case for defense-in-depth policy around AI deployments: if safety refusals are fragile by design, any AI system handling sensitive data needs external controls beyond the model’s built-in guardrails — useful framing for board or audit conversations about AI risk.
- Engineer — Learn: The findings — that safety alignment increases over-refusal (safety tax), privacy is near-orthogonal to other trustworthiness dimensions, and distillation degrades robustness — are useful mental models for engineers selecting or evaluating LLMs in their stack, though no immediate system changes are required.
- SOC/IR — Skip
- Leader — Learn: The finding that strong alignment does not protect privacy, and that distilled models suffer robustness collapse, provides empirical grounding for AI governance decisions and risk conversations with leadership about LLM adoption — useful for future board decks but no same-week action needed.
- Engineer — Learn: Novel architecture for parameter-level capability gating in MoE models (tested on Qwen3-30B and DeepSeek-V2-Lite); worth tracking if your team deploys or fine-tunes MoE-based models and needs verifiable separation between capability tiers — no production tooling yet.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Identifies a real supply-chain risk for teams consuming third-party LoRA adapters: a backdoored adapter can alter model output on hidden triggers without modifying base model weights. LoRAScan’s inference-time monitoring approach is worth evaluating if your ML pipelines pull adapters from untrusted registries or Hugging Face.
- SOC/IR — Learn: No active exploitation, IOCs, or ATT&CK-mappable TTPs to act on; this is foundational research on a threat class. Worth filing as context if your org is building detections around AI/ML pipeline integrity, but no hunt or rule work warranted today.
- Leader — Learn: Surfaces an emerging supply-chain risk category for AI workloads—untrusted fine-tuned adapters as a malware vector—useful background for shaping AI vendor-risk policy before it becomes a control requirement.
- Engineer — Learn: The hybrid deterministic-plus-LLM pipeline (regex/AST/topology plus LLM refinement) that roughly doubles vulnerability coverage over static rules alone is worth tracking as a design pattern for AppSec tooling, though the automotive ECU focus makes it directly applicable only in that niche.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Novel attack class: hidden payloads in pre-filled AI deep links can alter LLM memory without user awareness. No exploitation signals or PoC, but engineers building AI-integrated features should audit any ‘Ask AI’ button implementations for unsanitized prompt passthrough.
- SOC/IR — Learn: No IOCs, ATT&CK mapping, or active campaign indicators are present. Worth tracking as AI assistant adoption grows, but there is no detection surface or hunt query to act on today.
- Leader — Plan: This attack class is relevant to any enterprise deploying AI assistants with memory or context features; factor it into AI acceptable-use policy and vendor evaluation criteria before broader rollout.
- Engineer — Learn: Research shows that single-turn ASR benchmarks overstate real-world robustness of GUI agent guardrails, with 4-turn escalation chains recovering ~20 points of attack success across all tested models. Teams building or deploying GUI agents should treat static prompt-level alignment as insufficient and evaluate multi-turn threat scenarios in their safety testing.
- SOC/IR — Skip
- Leader — Learn: If your organization is piloting or deploying AI GUI agents, this research illustrates that current safety guardrails are weaker than benchmark numbers suggest under realistic multi-turn user interaction — useful context for AI deployment policies and vendor capability reviews, but no immediate action required.
- Engineer — Learn: Interesting research combining code slicing with LLM analysis to detect reentrancy and overflow in ERC-721 contracts, but no tooling release or actionable change to running systems today.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Novel prompt-suffix attack degrades speculative decoding throughput without corrupting outputs, affecting any deployment using draft-target inference acceleration (vLLM, TGI, etc.). No patch or mitigation exists yet; file this when designing LLM serving infrastructure to justify input validation and rate controls at the prompt layer.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Research shows output-only filters like Llama-Guard 3 are insufficient against reasoning-layer attacks; teams building AI applications should evaluate reasoning context, not just final outputs, when designing safety architectures.
- SOC/IR — Skip
- Leader — Learn: Finding that reasoning-capable models are 2x+ more vulnerable and standard output safeguards regularly fail has implications for enterprise AI risk posture; useful context for AI usage policies and vendor safety attestation reviews.
- Engineer — Learn: Research shows specific syntactic elements (constraints, guards, conditions) placed in prompts consistently reduce insecure code generation from open LLMs — useful input for teams building internal coding assistants or prompt templates for developer tooling.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: If you deploy LLM agents with skill files or tool orchestration, this research quantifies a real risk class: agents routinely violate preconditions and constraints, producing privacy leaks and unsafe config changes. No patch action today, but the SLGuard scaffold approach is worth evaluating if you build skill-guided agents.
- SOC/IR — Skip
- Leader — Learn: Academic evidence that LLM agents fail safety constraints at high rates is useful background for AI governance discussions, but there is no immediate vendor exposure or regulatory trigger here — file for the next AI risk policy review.
- Engineer — Skip
- SOC/IR — Learn: SherAgent demonstrates a 31–64% improvement in automated attack investigation success rates using LLM-driven provenance graph backtracking — useful context for teams evaluating or building AI-assisted triage workflows, though no production tool or IOCs are released here.
- Leader — Learn: Research from a real SOC environment shows LLM-assisted alert triage meaningfully reduces the manual investigation backlog; relevant background for leaders assessing AI tooling investments in detection and response.
- Engineer — Learn: Novel research showing an LLM-agentic pipeline that improves directed fuzzer crash-trigger rates by generating semantically aware seed corpora; worth evaluating if your team runs fuzzing campaigns against internal C/C++ codebases, but no immediate change to running systems is required.
- SOC/IR — Skip
- Leader — Skip
- Engineer — Learn: Academic research on grounded agentic reasoning for malware behavior reconstruction; no immediate engineering action, but the tri-grounding approach (domain, semantics, knowledge) is worth noting when evaluating LLM-assisted code-analysis tooling.
- SOC/IR — Learn: Malaika’s behavior-reconstruction framing — connecting sparse program evidence to auditable behavioral conclusions — could inform how teams structure LLM-assisted malware triage workflows, though no detection or hunt action is available from this paper alone.
- Leader — Skip