How to write a perfect prompt?

Prompting is the only way to talk to modern LLMs. The concept of prompting goes back to 1970, but it became widely used only with GPT-2 and GPT-3. A recent study, "The Prompt Report," showed that slight changes in the ordering of examples in the prompt could affect accuracy by tens of per cent, so writing good prompts can be quite challenging.

At least 200 different prompt techniques exist, including multilingual and multimodal support. The study divides all these techniques into Zero-Shot, Few-Shot, Thought Generation, Ensembling, Self-Criticism, and Decomposition. The recommendation here is to focus on the problem(understand how domain experts solve it right now), start from a simple approach, monitor and iterate.

There is a better alternative - generate prompts by LLM itself. Just a month ago, Anthropic released a tool for prompt generation and shared how the tool works internally with the actual prompt. The prompt uses different techniques, such as chain of thought, role prompting, style prompting, few-shot prompting, and others. Now, we can just write a basic prompt ourselves and ask LLM to optimize it.

The APEER(Automatic Prompt Engineering Enhances Large Language Model Reranking) paper reveals more details on how automated prompt generation works. APEER's main focus is Information Retrieval tasks. The optimization process has two steps: Feedback optimization and Preference optimization.

The first step is Feedback optimization. We take an initial prompt and execute it. The initial prompt and results are passed to the feedback generation prompt. The feedback is processed with help of refinement prompt. This is how the refined prompt is generated. To generate the new prompt we need the second step - Preference optimization.

The refined prompt is used as a parameter, together with positive and negative examples, to generate the new prompt. The new prompt is an improved version of the initial prompt. As described, the APEER requires three meta prompts: feedback, refinement and preference optimization prompts. The negative and positive examples are also generated with the help of LLM.

References:

1. https://arxiv.org/abs/2406.06608 - The Prompt Report: A Systematic Survey of Prompting Techniques

2. https://www.anthropic.com/news/prompt-generator - Generate better prompts in the developer console

3. https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview - Prompt engineering overview

4. https://topai.tools/s/LLM-prompt-optimizer - 70 Top AI LLM Prompt Optimizer Tools

5. https://colab.research.google.com/drive/1SoAajN8CBYTl79VyTwxtxncfCWlHlyy9 - Prompt Generator [MAKE A COPY].ipynb

6. https://github.com/keirp/automatic_prompt_engineer - Large Language Models Are Human-Level Prompt Engineers (code)

7. https://arxiv.org/pdf/2211.01910 - Large Language Models Are Human-Level Prompt Engineers

8. https://arxiv.org/abs/2406.14449 - APEER: Automatic Prompt Engineering Enhances Large Language Model Reranking

9. https://www.promptingguide.ai/techniques/ape - Automatic Prompt Engineer (APE)