Property Finder Revises Biased Model Evaluation and Selects Llama 4 Maverick on Amazon Bedrock

Property Finder replaced model-generated evaluation labels with expert annotations and optimized prompts, then selected Llama 4 Maverick 17B Instruct on Amazon Bedrock, with an estimated 86 percent cost reduction, according to the AWS Machine Learning Blog. The team found that 96.8 percent of its evaluation labels came from its incumbent model, biasing comparisons with alternatives. Working with the AWS Generative AI Innovation Center, Property Finder built expert-annotated ground truth and applied evolutionary prompt optimization to support its vision model selection. On an evaluation of 356 images, the selected model achieved a Micro F1 score of 0.80, compared with the incumbent’s 0.56. However, the evaluation included images used during development, and tuning effort differed between models, making the reported scores optimistic.

Generated by AI from reports · Updated 2 hours ago

There is not enough continuous observation data to plot a trend yet.