Provide Socioeconomic Data with an LLM Tool to Avoid Hallucinations

Without access to relevant datasets, large language models are prone to hallucinate responses. Create an LLM tool with accurate rent and income data to mitigate this problem.

The dataset used below is computed in Compare Median Income and Fair Market Rent. For access to large language models, use Wolfram LLM Kit.

Without access to tools and data, large language models may "hallucinate" incorrect responses:

In[1]:=

userQuestion="According to 2019 data, which Illinois county had the lowest ratio of fair market rent (for a four bedroom unit) to median household income?";model="gpt-4.1-nano";ChatEvaluate[ChatObject[],userQuestion,LLMEvaluator"Model"model]["MessageGrid"]

Out[1]=

Notice that the hallucinated responses are likely to vary given the same user query:

In[2]:=

ChatEvaluate[ChatObject[],userQuestion,LLMEvaluator"Model"model]["MessageGrid"]

Out[2]=

Data to answer this specific question was computed in the example Compare Median Income and Fair Market Rent:

In[3]:=

actualData=

Tabular[

]

Out[3]=

Define a tool which can answer questions in this domain:

In[4]:=

lookuptool=LLMTool[{"IllinoisRentRatioLookup","This tool provides the 2019 ratio of fair-market rent to median income by county."},{"ComparisonType""Interpreter"Restricted["String",Alternatives["Highest","Lowest"]],"Help""Only accepts ONE of \"Highest\" or \"Lowest\" with no quotation marks.","ApartmentType""Interpreter"Restricted["String",Alternatives["Studio","1BR","2BR","3BR","4BR"]],"Help""Only accepts ONE of \"Studio\", \"1BR\", \"2BR\", \"3BR\", or \"4BR\" with no quotation marks."},Function[Switch[#ComparisonType,"Lowest",StringRiffle[MapApply[CommonName[#1]<>": "<>ToString[#2]&,FromTabular[TakeSmallestBy[actualData,#ApartmentType,10]〚All,{"County",#ApartmentType}〛,"Matrix"]],"\n"],"Highest",,StringRiffle[MapApply[CommonName[#1]<>": "<>ToString[#2]&,FromTabular[TakeLargestBy[actualData,#ApartmentType,10]〚All,{"County",#ApartmentType}〛,"Matrix"]],"\n"]_,"Lookup Failed."]]]

Out[4]=

LLMTool

Name: IllinoisRentRatioLookup

Description: This tool provides the 2019 ratio of fair-market rent to median income bycounty.



Check a sample output of the tool:

In[5]:=

lookuptool["ComparisonType""Lowest","ApartmentType""4BR"]

Out[5]=

Menard County, Illinois, United States: 0.183814Woodford County, Illinois, United States: 0.189704Washington County, Illinois, United States: 0.193902Scott County, Illinois, United States: 0.197906Monroe County, Illinois, United States: 0.200124Boone County, Illinois, United States: 0.20112De Witt County, Illinois, United States: 0.203357Moultrie County, Illinois, United States: 0.207563Bureau County, Illinois, United States: 0.2106Macoupin County, Illinois, United States: 0.213637

When provided with the correct computational tool, the LLM can accurately answer this user query:

In[6]:=

ChatEvaluate[ChatObject[],userQuestion,LLMEvaluator"Model"model,"Tools"lookuptool]["MessageGrid"]

Out[6]=

Publisher Information

Contributed by: John McNally

Wolfram Language Example Repository

Provide Socioeconomic Data with an LLM Tool to Avoid Hallucinations

See Also

Publisher Information

Provide Socioeconomic Data with an LLM Tool to Avoid Hallucinations

See Also

Related Symbols

Publisher Information