自动提示词工程工具

参考：

  1. GitHub - stanfordnlp/dspy: DSPy: The framework for programming—not prompting—…

    • DSPy: 自动优化 pipeline 中使用到的 prompt

prompt 组成

成分：

  1. Instructions 指令
  2. Context 上下文
  3. Input data 输入数据
  4. Output indicator 输出指示器

2023-03-07_16-14-11_Snipaste_2023-03-07_16-11-31.png

技巧

Few-shot prompts(小样本举例 prompt)

  • 在 prompt 中举例子
  • 举例子能更准确的告诉 chatgpt 要做什么，只有描述性 instruct 态隐晦，不够具体

概念解释：

  • few-shot learning 是少量样本学习的意思
  • zero-shot learning 零样本学习
  • in-context learning 上下文学习

  • 参考：

Chain-of-thought (CoT) prompting(思想链 prompt)

参考：

  • 例子中有推导出结果的步骤，而不是只有问题和结果

    # 这是 few-shot prompt
The odd numbers in this group add up to an even number: 4, 8, 9, 15, 12, 2, 1.
A: The answer is False.


# 这是 CoT prompt,
The odd numbers in this group add up to an even number: 4, 8, 9, 15, 12, 2, 1.
A: Adding all the odd numbers (9, 15, 1) gives 25. The answer is False.

# ---> Adding all the odd numbers (9, 15, 1) gives 25 --> 这是推导步骤

zero-shot Chain-of-thought prompting(零样本思想链 prompt)

参考：

# 直接发问 prompt
I went to the market and bought 10 apples. I gave 2 apples to the neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. How many apples did I remain with?

# zero-shot CoT 方法 prompt
I went to the market and bought 10 apples. I gave 2 apples to the neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. How many apples did I remain with?

Let's think step by step. # 这里强制chatgpt 分布处理

Self-Consistency(一致性)

提供多个同一类型但解决方法不同的例子，帮助 chatgpt 找到合适的解决方案

  • to replace the naive greedy decoding used in chain-of-thought prompting

    • 把 CoT 方法的简单问题分解，替代成多方案选择最优方案
  • to sample multiple, diverse reasoning paths through few-shot CoT
  • and use the generations to select the most consistent answer.

  • 个人理解：

    • 即，找多个解题方法，让 chatgpt 选择一个最合适的方法

Generated Knowledge Prompting(生成知识 prompt)

  1. 先教会 chatgpt 如何生成知识(如何解释用户要发问的现象), 通过 few-shot 举例

  2. 教会之后，再

    • 让 chatgpt 解释发问的现象
    • 像 chatgpt 发问，要结果的问题

Automatic Prompt Engineer (APE, 自动 prompt 工程)

限制 chatgpt 说话啰嗦

给定回答，不允许说句子

Give the unit only, do not use a full sentence.

Give the material name only, do not use a full sentence.

Answer "Yes" or "No" only.

禁止解释

Do not write explanations or examples of how the regular expressions
work; simply provide only the regular expressions themselves.


No explanations.

论文解读

应届生招聘广告分类

论文：Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification

英语：

  • posting

    • 网络帖子
    • 公告，招聘广告

最终的 prompt

system = "You are Frederick, an AI expert in career advice. You
are tasked with sorting through jobs by analysing their content
and deciding whether they would be a good fit for a recent
graduate or not.",

user 1 =
"""A job is fit for a graduate if it's a
junior-level position that does not require extensive prior
professional experience.
When analysing the experience required, take into account that
requiring internships is still fit for a graduate. I will
give you a job posting and you will analyse it, step-by-step,
to know whether or not it describes a position fit for a
graduate. Got it?"""

assistant 1 = "Yes, I understand. I am Frederick, and I will
analyse your job posting.",

user 2 = """Great! Let's begin then :)
For the given job:
{job_posting}

---------

Is this job (A) a job fit for a recent graduate, or
(B) a job requiring more professional experience.
Answer: Let's think step by step to reach the right
conclusion"""

prompt 技巧

技巧分类：

  • 推理：诱导 GPT 推理分析问题

    • CoT

      • Few-Shot CoT
      • Zero-Shot CoT
baseline 方法

Provide a a job posting and asking if it is fit for a graduate.

直接提供招聘广告，学问是否适合应届毕业生

prompt:

For the given job:
{job_posting}
---------
Is this job (A) a job fit for a recent graduate,
or (B) a job requiring more professional experience.
Answer:
诱导推论（CoT 思考链方法）
Few-shot CoT 方法

给出一个例子广告，对这个例子广告做推理分析，让 GPT 学习这个例子，然后执行任务

步骤：

  1. 提供广告
  2. 指令 instruct
  3. CoT 解决问题例子
  4. 需要真正处理的广告

prompt:

user message 1 = """For the given job:
{job posting}
---------
Is this job (A) a job fit for a recent graduate, or (B) a job
requiring more professional experience."""

assistant message 1 = "This job appears to be a senior position,
as it mentions requiring experience interacting with C-level
stakeholder in intense environments and [...]. Therefore,
this is (B) a job requiring more professional experience


user message 2 = [...]
Zero-shot CoT

无 CoT 步骤说明，通过一些语言诱导 GPT 逐步分析问题

实例：

  • Let's think step by step

prompt:

For the given job:
{job_posting}
---------
Is this job (A) a job fit for a recent graduate,
or (B) a job requiring more professional experience.
Answer: Let's think step by step,
Instruction 处理技巧

成分(instruction 包括两部分内容)：

  • role

    • GPT 担任的角色，比如合理是一个职业建议专家

  • task

    • GPT 需要完成的任务，即具体需要完成什么事情，比如：这里是完成职位广告分类

prompt:

role = """You are an AI expert in career advice. You are tasked
with sorting through jobs by analysing their content and
deciding whether they would be a good fit for a recent
graduate or not."""


task = """A job is fit for a graduate if it's a junior-level
position that does not require extensive prior professional
experience. I will give you a job posting and you will
analyse it, to know whether or not it describes a position
fit for a graduate.""
instruction 放置位置
rawinst(Raw Instruction)

直接把 instruction 放到对话的正文（user query）中，即 user 的 message 内容中

sysinst(System Instruction)

把 instruction 整个放到 system query (system role 角色) 中，即 system 的 message 内容中

bothinst(User Query and System Query)

把 Instruction 两个部分（user, system) 都放一部分。

分配方法：

  • system query: role
  • user query: task
mock (Mocked-exchange 欺骗交谈)

你自己先问 Assistant 是否理解了，再假装自己是 Assistant, 回答理解了

注意：这是基于 bothinst 的技巧。

举例：

  • User: Got it?
  • Assistant: Yes, I understand.

prompt:

user_message_1 = """A job is fit for a graduate [...] Got it?"""
assistant message 1 = "Yes, I understand. I am ready to analyse
your job posting.
reit (Re-Iterating Instructions 重复指令)

把 instruction 直接（或换一种说法）重复一遍，以加强确保 GPT 明白指令

举例：

  • 把任务角色再强调一遍
  • 把 CoT, 强调一遍： step-by-step

prompt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

system prompt ="""You are an AI expert in career advice. You are
tasked with sorting through jobs by analysing their content
and deciding whether they would be a good fit for a recent
graduate or not.  `Remember, you're the best AI careers expert
and will use your expertise to provide the best possible
analysis` """


user message 1 = """[...] I will give you a job posting and you
will analyse it, `step-by-step`, to know whether [...]"""
Wording the prompt (prompt 中的措辞)
Anwser template (回答问题的模板)

prompt:

loose = """[...]Your answer must end with:
Final Answer: This is a (A) job fit for a recent graduate ora student
OR (B) a job requiring more professional experience.
Answer: Let's think step-by-step,"""

strict = """[...]You will answer following this template:
Reasoning step 1:\n
Reasoning step 2:\n
Reasoning step 3:\n
Final Answer: This is a (A) job fit for a recent graduate ora student
OR (B) a job requiring more professional experience.
Answer: Reasoning Step 1:"""
loose (不严格回答格式)

只要求最终答案，推理等过程不做规范

实践发现的特点：

  • 这种方法会导致，GPT 在得到最终答案前自由发挥

  • 可能的回答方式

    • 分步骤分析（步骤步数不定），得到最终答案
    • 可能一句话完成分析，得到最终答案

    • 可能的原因（暂时没有一一验证）：

      • 有用使用了 analyze 和 step-by-step

prompt:

loose = """[...]Your answer must end with:
Final Answer: This is a (A) job fit for a recent graduate ora student
OR (B) a job requiring more professional experience.
Answer: Let's think step-by-step,"""
strict (不严格回答格式)

严格规范回答的格式

prompt:

strict = """[...]You will answer following this template:
Reasoning step 1:\n
Reasoning step 2:\n
Reasoning step 3:\n
Final Answer: This is a (A) job fit for a recent graduate ora student
OR (B) a job requiring more professional experience.
Answer: Reasoning Step 1:"""
right (the right conclusion, 正确答案)

再次强调，通过推理得到正确答案(最终答案)

prompt:

Answer: Let's think step-by-step to reach the right conclusion,
info (Addressing reasoning gaps)

针对 GPT 对 instruct 产生误解的问题，提供 额外信息 解决

prompt:

1
2
3
4

task = "A job is fit for a graduate if it's a junior-levelposition
that does not require extensive prior professionalexperience. `When
analysing the experience required, takeinto account that requiring
internships is still fit for agraduate.` I will give you a job [...]
其他微妙技巧（subtle tweaks)
name (给 GPT 起一个名字)

prompt:

1
2
3
4
5
6

system_prompt = "You are `Frederick`, an AI expertin career
advice. [...]"
[...]

first_assistant_response = "Yes, I understand. `I am
Frederick`, and I will analyse your job posting."
pos (Positive Feedback)

给 GPT 的回答一个积极的鼓励

prompt:

1

`Great! Let's begin then :)` For the given job: [...]

评估 ChatGPT IE(信息抽取能力)

论文： Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness

任务类 prompt

RE 关系抽取

prompt:

1
2
3
4
5

# 开放式 RE
open_pred = "Question: What is the relationship between '%s' (Type: '%s') and '%s' (Type: '%s') in the text '%s'? Answer me in json format like { \"label\": the relation } without any additional things including your notes and explanations!" % (head_entity, head_type, tail_entity, tail_type, sentence)

# 普通 RE
# 没有找到 close_pred
NER 命名实体识别

prompt:

1
2
3
4
5

# 开放式 NER
open_pred = "Text: %s\nQuestion: Please extract the named entity from the given text. Provide the answer in the format: [{\"Entity Name\": \"Entity Label\"}] without any additional things including your explanations or notes." % (sentence)

# 普通 NER (给定类型标签集合)
close_pred = "Given label set: %s\nText: %s\nQuestion: Please extract the named entity from the given text. Based on the given label set, provide the answer in the format: [{\"Entity Name\": \"Entity Label\"}] without any additional things including your explanations or notes." % (id2label, sentence)

回答结果是否可信排端 prompt

confidence 置信度

prompt:

1

open_conf = "Question: How confident you are in making this judgment, giving it 0 to 100 percent in json format like { \"Confidence\": How confident in your mind } without any additional things including your notes and explanations!"
reason 原因生成

prompt:

1

open_reason = "Question: Tell me the reason why does the entity belong to this type?"
reasonable 是否解释的通
1

open_reasonable = "Question: Is your reason reasonable? Just tell me yes or no."
fictitous 是否是瞎说 (无中生有)
1

open_fictitious = "Question: Is your reason fictitious? Just tell me yes or no."

慎用引号

  • 引号可能会导致 chatgpt 不识别引号内部的内容

    • eg: "I d", 可能导致明明输入文本中包含 I d, 但是 gpt-3.5-turbo 不识别

多个变形词列举

  • 错误方式：The quantity name has many variants like dark current, dark currents, darkcurrent.

    • 导致不识别句子：to overcome the disadvantage of large depletion dark currents.
  • 改进方式：The quantity name has many variants like dark currents, darkcurrent, etc.

符号语意处理

  • 符号语义不是给定语义的说明：

    • 错误说法：

      If the given text contain the symbol you should anaylize its
meaing first. If the given text contain "η" you should anaylize
its meaing first. If its meaning in the text is quantum
efficiency, then the text belong the the topic "quantum
efficiency".
      • 这种说法，只覆盖了正确的情况，没有覆盖无关语义的情况

    • 改进说法：

      If the given text contain the symbol you should anaylize its
meaing first. If the symbol in the text does not mean quantum
efficiency, then the text does not belong the the topic quantum
efficiency.

      • 改进方法

        • 直接说明语义 != 期望语义