Wolfram Prompt Repository(Under Construction)
A curated collection of prompts, personas, functions, & more for LLMs (large language model AIs)
LLMPromptAssessment
Use an LLM to assess the quality of another LLM's response for a given prompt
LLMResourceFunction["LLMPromptAssessment"]["prompt"] generate a response from an LLM for the given prompt, then have another LLM assess the quality of that response. | |
LLMResourceFunction["LLMPromptAssessment"]["prompt",config] use the LLMConfiguration specified by config to generate the initial response. | |
LLMResourceFunction["LLMPromptAssessment"]["prompt",config,"extra"] includes the given extra instructions in the prompting for the assessment LLM. |
Details
Using LLMPromptAssessment involves making two LLM calls. One generates a response to the given prompt, and the other evaluates how well that response followed instructions.
The two LLM calls can be for different models.
In LLMResourceFunction["LLMPromptAssessment",LLMEvaluator→<|"Model"→"outer"|>]["prompt",<|"Model"→"inner"|>], the models "inner" and "outer" represent the following:
| "inner" | the model that generates the initial response for the given "prompt" |
| "outer" | the model that looks at the response and evaluates how well it followed instructions specified in "prompt" |
For best results, the "outer" model should be at least as capable as the "inner" model.
The extra instructions argument can be used to provide additional prompting for the assessment model.
The prompting for the assessment model instructs it to format its response as JSON, so extra instructions can be provided to add additional information to that JSON result which will be included in the parsed response.