Can Christians hang out with non-believers?

My husband and I are new born again Christians and are having a hard time dealing with our Pastor's message that we should not hang out with people who are not saved. We have friends who are both saved and not saved. Most of my family are Christians and so are my friends, but we do have friends who are not born again but have no problem with us being saved. Our pastor says we should not be hanging around non-believers because they may make us fall into our old ways again. Personally I dont think its bad, because everyone sins and even professing Christians act like bad examples sometimes, I believe we are not perfect and in life we cannot always be surrounded by Chritians. I am always spreading the word where ever I go and have no problem practicing my beliefs around non-believers. I actually care more about what the non-believers think of me than the Christians do because they are the ones I want to encourage. So what I am asking should we stay away from non-believers unless we are spreading the gospel or have another Christian Witness with us?