Wearing a dark grey shirt and khakis - fashion do or fashion don't?

Male here. I have a solid dark grey/charcoal dress shirt – for more detail, this color is pretty close: http://globaltotaloffice.com/textiles_finishes/large/CGR-Charcoal-Grey.jpg

Would wearing it with khaki pants be really ugly? I’m not trying to be the best dressed guy at the office or anything but if it’s noticeably breaking-the-fashion-rules ugly then I’ll find something else, but I do like how it fits and would love to add it to my work wardrobe if it doesn’t look horrendous.