In Central Administration I went to
System Settings->Manage Farm Solutions
In there I selected the WSP I want to retract, clicked the
Retract Solution link and clicked "OK" with all of the default values selected.
It then goes back to the "Solution Management" page and the status says "Retracting(Scheduled at ...)" with the current time in place of ...
For some reason when I go back to this page after a few minutes and check the Status it is back to deployed. What am I doing wrong?