A guy likes me but I don't like him?

This guy at my school likes me and I really don't care for him. I have nothing against him, I just know I won't like him. I would consider myself smart by average standards, I've always gotten good grades and I never study. I don't think that makes me better than anyone else. This guy with a learning disability likes me and I really don't like him in that way. I've never talked to him but his friends keep insisting that I should. He has written me notes but I've never responded back. I have tried a relationship with someone like this before and it didn't really work out because we had nothing to talk about or of similar interest. I am a book work and prefer to talk about computers, science or mythology rather than what you did last weekend. I don't have anything against people with learning disabilities, I just know for myself that something like this won't work out. I used to help people with learning disabilities through church oriented activities. My friends keep pestering me about him saying I should talk to him and for stating what I did here they are now irate with me and I don't know what to do...