After watching Youtube, listening to advice given by friends I went to the pool for the 2nd time, HUGE success. I swam for roughly 40 minutes and had no issues with cramp, muscle fatigue or cardio. I went with a friend who I cycle with and he gave me some pointers which greatly improved technique, and after doing some more research online have confirmed that my main issue is breathing. When I actually did it right, my hole motion changed and I started to speed up.
I found a fantastic site http://www.swimsmooth.com/exhalation.html which detailed my exact problem