4 stages of LLM app value creation and development.
From Prompt Engineering to Agent Pipeline
Stage 1 - Prompt Engineering. When a prototype of a new LLM app is under development, it usually starts from a wrapper around a prompt. We have a simple prompt that we call via an API and return a response to the user. This is the basic approach which allows us to test value for the user. The problem of this stage is that it's pretty easy to replicate by competitors.
Stage 2 - Prompt Architecture. Chain of Thought, Self-consistency prompting, Step-back prompting and Chain of Verification are examples of advanced prompt techniques. These techniques require multi-step execution. The basic idea is to take the user's request and process it via a reasoning sequence.
Stage 3 - Knowledge Management Platform. In this stage, we add RAG, memory and Knowledge Base. We enhance our app by providing external and/or real-time information with the ability to remember previous interactions. One might think of it as a stateful LLM app, but not only this. By creating and curating content for Knowledge Base, we create a high-level value for the user and a barrier to entry for competitors.
Stage 4 - Agent Pipeline. LLM Agent is a system which uses reasoning to define which actions to execute and which tools to use to solve a task at hand. There is a massive difference between role-based prompt engineering ("You are a lawyer. Please answer this question - ") and Agents. From what I see around me, to solve a task we usually need to build several agents and organize them in the way of a processing pipeline. A good analogy of such a pipeline might be a web request processing pipeline with middleware.
As an example, let's build "talk to txt file" app, stages:
1. "Here is a document: {DOCUMENT}. Answer user's question - {QUESTION}"
2. "Here is a document: {DOCUMENT}. {STEPBACK_CONTEXT} Answer user's question - {QUESTION}", STEPBACK_CONTEXT is an additional call to LLM to generate step-back question from user's question and embed it as a context
3. "Here is a document: {DOCUMENT}. Here is the additional context: {MEMORY}. {STEPBACK_CONTEXT} Answer user's question - {QUESTION}" - here, we add MEMORY, which can be additional information from the Knowledge base or a summary of previous conversations with the user.
4. "Here is a list of tools: WEB_SEARCH, DOCUMENT, …. . Choose tools and propose an algorithm to answer the user's question - {QUESTION}"
A few articles on this subject:
https://cobusgreyling.medium.com/large-language-model-llm-disruption-of-chatbots-8115fffadc22 - Large Language Model (LLM) Disruption of Chatbots
https://towardsdatascience.com/prompt-architecture-bd8a07117dab - Is it Time to Start Talking About Prompt Architecture in LLMs?
https://www.ionio.ai/blog/what-is-llm-agent-ultimate-guide-to-llm-agent-with-technical-breakdown - What is LLM Agent? Ultimate Guide to LLM Agent [With Technical Breakdown]