About 10/15 years ago, the EU did allow diesels to meet a lower pollution standard than gas engines (I can't remember which pollutant - NOx maybe?) in order to get the benefit of the diesels' greater fuel economy. I believe that part of this problem was addressed through reformulating the diesel sold, rather than changing the engines.
The US took a different course and the cynic in me can't help noticing that US automakers now don't have diesel engines to offer, so who (with a decent lobbying budget) is going to be pressing to either relax the one emissions standard slightly or change the diesel sold?
Catch 22, I think.