how to deal with office politic?

I just joined a new company recently. I can't really adapt to the new environment as it is quite political working environment. The worst is.. my colleagues tend to work late. However, i always believe that if i can complete my tasks on time, i dun have to justify my early leaving to anybody, even my boss. however, my colleagues started questioning me whenever they see me leaving on time. It makes me feel very uncomfortable and stress whenever i want to go off on time. The worst is they started to load me with some of their scope which i think it is extremely unfair to me. However, my boss seems unhappy for my early leaving too... Thus, i can't really feedback to him as well. I am considering whether should i just quit this job. But it doesn't sound right to me as i shouldn't give up just because of this stupid reason. How should i deal with those politic collegues? or just ignore them and leave on time everyday as long as i can complete my work?