Can a model judge product images like a client does?

Approval is the slowest step in producing a catalog. A person looks at every image against the real garment and decides. If a model could do that reliably, an agent could generate, check and retry on its own. We wanted a number before building on that idea.

The test

We took 24 generated images from five product folders of one womenswear client. The client had accepted 11 and sent 13 back. Labels came only from the client: accepted means the image ended up in a folder she approved, sent back means she left a note on that image. Anything we were unsure about was left out.

Each folder came with the references a producer would use: the fashion model and two to four photos of the garment. Among the 13 rejections, seven were about color or texture, four about construction, three about styling and one about identity. There were no pose or framing cases.

We tested three models from one frontier family: a large, a mid-size and a small one. Each saw the images only through two read-only tools, one to view an image and one to compare two side by side, and judged every image on identity, construction, color and texture, styling, pose and framing.

What we found

The large model matched the client on 20 of 24 images. It caught 11 of the 13 rejections, 9 of them for the client's own reason, and wrongly failed 2 of the 11 accepted images. Its lead over each smaller model is unlikely to be chance (p about 0.04), but this is one small set.

The mid-size and small models matched on 13 each, exactly what you get by rejecting every image. On a set where most images fail, that is the baseline any judge has to beat. Confidence did not help either: the models were about as sure of their wrong verdicts as of their right ones.

The olive top

One product was a brown top that came out olive in all four views, and the client sent back every one. No model reliably caught it. The large model failed two of the four, but for the face. The mid-size model passed all four. The small model flagged one as olive against tan, then passed another while describing the same shade.

Color is where a judge that looks and describes is weakest. A shift that ends a client review is small in numbers and easy to talk yourself out of in words. Our conclusion is to measure color rather than ask a model about it.

Planning and generating

We also asked each model to plan the shoot for four folders and then produce it. Graded blind out of 80, the plans scored 78 for the mid-size model, 76 for the small one and 74 for the large one. All three followed the procedure exactly. The large model lost points by nudging the brown top toward the color name on its tag, the same defect the client had already sent back.

In generation the large and mid-size models spent about the same per accepted image, and the small one about 30% more. The only brown set came from the mid-size model; the large model followed its own plan to olive in every view.

What we changed

• Judging goes to the large model, and a person stays on final approval.
• Color is measured, not judged by eye.
• Our playbook now says the garment photos override the color name on a tag.
• The next run adds pose, framing and identity cases, and repeats each model more than once.