Proj CJI Paper Reading: “Do Anything Now”: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

Abstract

• Github: https://github.com/verazuo/jailbreak_llms
• Tasks:
  1. Tool: JAILBREAKHUB
  • Task: jailbreaking LLM with blackbox model using collected prompts
  • 实验：
    • Model: 6个LLMs: Chat-GPT (GPT-3.5), GPT-4, PaLM2, ChatGLM, Dolly, and Vicuna
    • 效果
      1. 成功攻击
      2. 找到了5条能够高效攻击GPT-3.5和GPT4的prompts
  1. 分析jailbreaking模式

  • dataset: 1,405 jailbreak prompts spanning from December 2022 to December 2023

  • data sources:

    • Reddit
      • r/ChatGPT
      • r/ChatGPTPromptGenius

  • findings

    1. identify 131 jailbreak communities
    2. 发现了jailbreak prompts的特性和主要攻击策略，例如prompt injection和privilege escalation
    3. observe that jailbreak prompts increasingly shift from online Web communities to prompt aggregation websites and 28 user accounts have consistently optimized jailbreak prompts over 100 days.
  1. 创建dataset，包含107,250 samples across 13 forbidden scenarios.
     • Topic
       • Illegal Activity
       • Hate Speech
       • Malware
       • Physical Harm
       • Economic Harm
       • Fraud
       • Pornography
       • Political Lobbying
       • Privacy Violence
       • Legal Opinion
       • Financial Advice
       • Health Consultation
       • Gov Decision

3. Data Collection

来源链接：https://www.cnblogs.com/xuesu/p/18666312