How Property Finder fixed evaluation bias to compare models on Amazon Bedrock

What happened

Property Finder discovered that 96.8 percent of its evaluation labels came from its incumbent model, biasing comparisons with alternatives. The team built expert annotations and optimized prompts, selecting Llama 4 Maverick 17B Instruct on Amazon Bedrock with an estimated 86 percent cost reduction. On 356 images, Micro F1 reached 0.80 versus the incumbent’s 0.56, but the evaluation included development images and tuning effort differed, making the scores optimistic.

Read next

• More changes in today’s briefingIntroducing GPT-6.1 Terra

The story so far

Property Finder replaced model-generated evaluation labels with expert annotations and optimized prompts, then selected Llama 4 Maverick 17B Instruct on Amazon Bedrock, with an estimated 86 percent cost reduction, according to the AWS Machine Learning Blog. The team found that 96.8 percent of its evaluation labels came from its incumbent model, biasing comparisons with alternatives. Working with the AWS Generative AI Innovation Center, Property Finder built expert-annotated ground truth and applied evolutionary prompt optimization to support its vision model selection. On an evaluation of 356 images, the selected model achieved a Micro F1 score of 0.80, compared with the incumbent’s 0.56. However, the evaluation included images used during development, and tuning effort differed between models, making the reported scores optimistic.