What Is Prompt Engineering and Why Does It Matter?

TL;DR

Prompt engineering is the iterative process of refining the text input you give a large language model, studying the response, and repeating until the output matches your goal. It matters because models are stochastic (the same prompt can yield different answers) and can fabricate facts, so tuning prompts, parameters, and system personas produces more reliable results.

Transcript

hi welcome to Lesson Four of generative AI for beginners my name is Nan Nan I'm a senior Cloud Advocate on the AI advocacy team and I'm excited to talk to you about prompt engineering fundamentals So today we're going to first start off with a quick recap if you've been following the series you know most of the terms we'll just look at them really ... Read More

Key Insights

• A prompt is the natural language text input a user provides to a large language model, effectively programming the model to do what the user wants, while the response is the content the model generates to support that request.
• Prompt engineering is the process of iterating on your text input, studying the model's response, and repeating until the response looks like what you want it to be.
• Models are stochastic, meaning there is no guarantee the same request will return the same response, that the response will be correct, or that it matches what you expected.
• The same prompt sent to different providers (Hugging Face, OpenAI, Azure OpenAI) and different models (GPT-3.5 Turbo vs GPT-4 Turbo) generates different responses, even though Azure OpenAI uses OpenAI under the covers.
• The temperature parameter controls creativity: increasing it makes the model return more varied, creative responses to the same prompt, while resubmitting an identical prompt can still produce different output.
• The system persona shapes response quality: the default is 'you're a helpful assistant,' but changing it to something like a creative assistant that only speaks in limerick changes how the model answers.
• Fabrication (also called hallucination) happens because a model does not understand meaning, it just predicts the next token based on global knowledge, so responses can look realistic but not be grounded in truth.
• GPT-3.5 Turbo has a September 2021 cutoff date and GPT-4 Turbo has a September 2023 cutoff, and asking about the 2025 Oscar led GPT-3.5 Turbo to invent a movie premise, demonstrating fabrication.

Questions & Answers

Q: What is prompt engineering?

Prompt engineering is the process where you iterate on the prompt, the text input that you give to the model, study the response, and keep going until the response looks like what you want it to be. To do it well you need to understand how the model works, starting from basic chat completion where you provide text input and the model returns generated content.

Q: What is a prompt in generative AI?

A prompt is the natural language input, the text input, that the user provides to a large language model. You can almost think of it as the way the user programs the model to do what they want. The response is then what the model generates to support the user's request, producing content based on that text input.

Q: Why does prompt engineering matter?

Prompt engineering matters because models are stochastic, meaning there is no guarantee the model gives the same response every time, no guarantee the response is correct, and no guarantee it is what you thought it would be. By adjusting variables like prompt templates, system context, and model parameters, you work to get the best possible response for your prompt.

Q: What does it mean that language models are stochastic?

Stochastic means that if you give the model a text input, there is no guarantee it returns the same response every time. In the video, the same 'tell me about the element gallium' prompt produced different responses across Hugging Face, OpenAI, and Azure OpenAI, across GPT-3.5 Turbo and GPT-4 Turbo, and even on repeated runs of the same model with the same request.

Q: How does the temperature parameter affect responses?

Temperature is a model parameter that controls creativity. In the demonstration, making the temperature bigger told the model to get more creative, so when the same prompt was submitted the model returned something completely different, mentioning uses from everyday electronics to medical imaging. Changing parameters like temperature, along with changing the model or the text, produces different responses.

Q: What is fabrication in large language models?

Fabrication, also heard by the term hallucination, is when the model generates a response that may not be rooted in fact. It happens because a model does not really understand or know the meaning of what you are asking, it is just predicting the next token, the next word, based on its global knowledge. Responses can look realistic but may not be true, and you may not even know.

Q: How can a system persona change a model's response?

The system persona sets the model's behavior and by default says you're a helpful assistant. In the video, changing the persona to a creative assistant that only speaks in limerick made the model return its answer about gallium as a limerick. This shows how you can change the quality and style of the response by tuning parameters like the system persona.

Q: What providers can beginners use to try prompt engineering?

The course lets you get started with one of three providers: OpenAI, Hugging Face, or Azure AI, where you can use Azure OpenAI or Azure AI Studio. A getting started folder shows how to set up. For exercises you have two choices: a no-code sandbox that needs no coding at all, or provided notebooks set up with your credentials for interactive programming exercises.

Summary & Key Takeaways

• Lesson Four of Generative AI for Beginners recaps core terms: a prompt is the natural language input that programs the model, the response is generated content, fabrication is unfounded output, a base LLM is a general-purpose foundational model, and instruction-tuned LLMs specialize in tasks like summarization, translation, or code generation.

• Prompt engineering iterates on text input, studies the response, and repeats until the output is right. Beyond the basic chat completion loop, you can adjust prompt templates, conversation history, retrieval augmented generation, system context, and model parameters. Learners can start with OpenAI, Hugging Face, or Azure AI, using either no-code sandboxes or notebooks.

• Two challenges motivate prompt engineering. Models are stochastic, so the same 'tell me about the element gallium' prompt gives different answers across providers, models, and repeated runs, and temperature and system persona shift results further. Models also fabricate: asked who won the 2025 Oscar, GPT-3.5 Turbo invented a movie premise.