Since Somewhere in Time, time travel stopped being just science fiction and could also be fantasy/drama. The story wasn’t in the science of how they got there, it was in what happened when they got there (and how to get back or stay there as the case may be). 
I just finished reading The Time Traveler’s Wife which is filled with time travel, paradoxes, meeting yourself, the irregularities in conversations with someone living a non-linear life and counting all of that, I would say that it is not science fiction. This book is by Audrey Niffenegger and unfortunately this is her first book, so I’ll have to wait for the next novel…
It is a story of a man and his wife. It takes place in the present past and future (but the story is generally in order to not confuse the reader) and tells the story of a man with “Chrono Displacement Disorder”. I know it still sounds like sci-fi, but it’s not. He generally travels to the same geographic areas: where his wife is/was, where he is/was and so on. Oh and it’s just himself that time travels, not his clothes. She first meets him when she is a child. When he first meets her she is an adult. At one point Henry meets her family, but makes a big social blunder and he asks Clare why she didn’t tell him, and she says ‘but I did’ (tell an older Henry when he visited earlier). At some points it gets confusing but is a wonderful story and the author tells it better than I. Each chapter is told from one person’s point of view and each chapter tells you the date and how old Henry and Clare are.
Clare waits and she waits. She’s waiting to grow up, to meet him, for him to come back and waits for things to happen. But she waits for him and this is what the story is about. The first chapter is available on-line.
