Test-Driven Prompt Engineering

blaise · July 31, 2023, 6:42pm

Tldr
The lack of determinism in llm makes test driven development more critical.

Code example:

preset-io/promptimize/blob/main/examples/python_examples.py

"""An example of how to test Python code generating prompts"""
import re

# Brining some "prompt generator" classes
from promptimize.prompt_cases import LangchainPromptCase

# Bringing some useful eval function that help evaluating and scoring responses
# eval functions have a handle on the prompt object and are expected
# to return a score between 0 and 1
from langchain import PromptTemplate
from langchain.output_parsers import StructuredOutputParser, ResponseSchema
import demjson

from RestrictedPython import compile_restricted, safe_globals, safe_builtins
from RestrictedPython.Guards import guarded_unpack_sequence
from RestrictedPython.Eval import default_guarded_getiter


response_schemas = [
    ResponseSchema(

This file has been truncated. show original