I've found this discussion interesting for a couple of reasons.
First, I don't think it has much to do with Mahler specifically. If you tell most people that you are a fan of any composer, or even just generally, that you love classical music, you'll likely get blank stares. You'd likely get the same kind of reaction by saying "I'm play World of Warcraft" or "I love bass fishing" or "I write poetry."
Second, the fact is, the majority of people go to work, eat some dinner, watch mindless crap on TV, the go to bed. Computer nerds will do stuff on their computer instead of watching TV. Politicians will be involved in social functions, then go to bed.
The vast majority of people do not and have never read Shakespeare or Dostoevsky or Goethe, do not and have never listened to Taneyev or Raff or Chausson, do not and have never see an exhibit of Chihuly or Frankenthaler or Dali, do not and have never enjoyed fine wine, artisan cheeses, gourmet meals.
In fact, I'd bet that the number of people on this board who have ever done any othese things is small. So, why would you expect people at your workplace to share your interest in Mahler?
Mahler fans are no more losers than fans of Chateau Montelena wines or Akira Kurosawa movies.
All that said, I do think mention of Mahler does raise specific images for some people. Those who have little patience for anything too intellectual, who escape into South Park and CSI, but who have heard Mahler, may think of Mahler as long, very long, slow moving, tedious, turgid and above all, *depressing*. You must remember that, fo most people, music is not all that important. Mahler requires time and attention, two commodities that many people value much more highly than sitting through a symphony, and especially a long, turgid, depressing symphony.