There was something similar with a cancer-detecting model somewhat recently. It was trained on a bunch of scans both positive and negative. However, positive scans had to be signed off on by a doctor. So the model just ended up looking for signatures.