might be the brushes, read on here I think where a guy did a re-build on his TM because it was acting similar. He pulled the lower motor apart, cleaned it all, put some new parts in (I think) and it ran good afterwards.
You may have a switch going/gone bad. I had a 43# MG, and it would work half the time when I pressed the gray pad on mine, but would work fine when on constant. There is a pressure switch under the foot pad (the small rectangular shaped pad) you press to activate the motor when not on constant speed. This pressure switch is covered by a rubber boot (for lack of a better description), and mine had dry rotted. Got a new one for a couple bucks, and it was good as new. Mine was/is an 04 TM pro-guide off a Tracker PT 175. I have since upgraded so its in the shed collecting dust.
Post up a picture of the foot control and I'll see if its similar to mine.