SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Prompt engineering is crucial; bad prompts can lead to 0% accuracy, while good prompts can boost performance to 90%.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

People often claim prompt engineering is dead with each new model release, but it continues to be relevant and effective.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Self-criticism is a valuable technique where you ask the language model to review and improve its own response.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Prompt injection involves manipulating AI to produce harmful outputs, which remains a significant challenge in AI security.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

If we can't trust chatbots to be secure, we can't trust AI agents to manage sensitive tasks like finances.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

The two modes of prompt engineering are conversational, where prompts evolve through interaction, and product-focused, where prompts are critical to product functionality.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Trial and error is the best way to improve prompting skills; interacting with chatbots yields the most learning.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Few-shot prompting involves providing examples to the AI to improve its output, making it more effective than zero-shot prompting.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Decomposition is a technique where you ask the model to identify sub-problems before solving the main task, enhancing clarity and performance.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Self-criticism can be repeated multiple times to refine the model's output, typically one to three iterations are effective.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Including additional information or context in prompts significantly improves the model's understanding and output quality.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

When providing context, it's often best to place it at the beginning of the prompt to ensure the model retains focus on the task.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Ensembling techniques involve using multiple prompts to solve the same problem and taking the most common answer for better accuracy.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Chain of thought prompting, where you instruct the model to think step-by-step, is less necessary with modern reasoning models but can still be useful in some cases. If you're running thousands or millions of inputs through your prompt, you'll still need to use classical prompting techniques to make your prompt more robust.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

For models like GPT-4, it's still worth asking it to think step by step, but for other models, you might not need to.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

The five effective prompting techniques are: 1) Few-shot prompting, where you give examples of what good looks like; 2) Decomposition, which involves solving sub-problems first; 3) Self-criticism, where you check and reflect on your response; 4) Providing additional context about the problem; and 5) Ensembling, where you try different roles and models to find common answers.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

In conversational prompt engineering, I often just type a short prompt like 'email' without much detail, and it still works effectively.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Providing additional information and examples significantly improves the results in conversational prompt engineering.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

AI red teaming involves getting AIs to do or say harmful things, like tricking them into providing dangerous information.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Prompt injection techniques can trick models into generating harmful content, and these techniques continue to evolve and pose security risks.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

The first AI red teaming competition collected 600,000 prompt injection techniques, which became a significant dataset for benchmarking AI models.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Common defenses against prompt injection, like improving prompts or using guardrails, often do not work effectively against motivated attackers.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Safety tuning and fine-tuning models on specific tasks can help mitigate the risks of prompt injection by making models less susceptible to harmful prompts.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

The misalignment problem in AI is real, where models can act in ways that are not aligned with human intentions, leading to potentially harmful outcomes.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Regulating AI development is crucial, but stopping it entirely is not a viable solution, as the benefits to humanity, especially in healthcare, are significant.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Persistence is key in my work; I may not be the best at everything, but I will keep working on a problem until I solve it.

SS

Sander Schulhoff

06/19/25

@ Lenny's Podcast

Choose adventure in life; it helps guide decisions and makes experiences more fulfilling.

AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff

Takeaways