'Holding mobile phone while driving' I didn't do it!!?

I was driving with my friend tonight to pick another friend up in a place we'd never been before so before leaving home I routed the journey on google maps on my phone and gave the phone to my friend to direct me along the way. We came to a junction in the journey and my friend got flustered and said she didn't know which way to go and was waving the phone around by me, I randomly decided to turn left and pulled in at the next available turning (off the road) so we could look at the map, and realised I should have turned right back at the junction so ready to continue the journey when a police car pulled up and I was asked to sit in their car. I was completely confused as to why as I had done nothing wrong, this was when he told me that I was texting back at the junction. I was not, the phone was never even in my hand and my phone records will show that I have made no calls or texts etc. My friend will also speak for me in court as I am telling the truth. I explained to him what had happened and he refused to believe me. If I had looked at my phone back at the junction I would not have gone the wrong way. He was going to give me a caution to start with then changed it to a fixed penalty and points as I didn't admit what he was claiming. I am not going to admit something that I haven't done. He also threatened me saying 'I wouldn't bother appealing if I were you because I'll get you charged for wreckless driving' which I find discusting, he has no right to try and intimmidate me into not appealing. Really annoyed that I see people speeding and driving wrecklessly all the time and where are they then?? And I do nothing and get that treatment. Quite upset about it and feel helpless. Do I have any chance of successfully appealing this?