Gardner impressed Brown in his loan spell and an extension was requested for a further stay but this was turned down by Bolton due to their hectic end of season programme.
The former Jamaican international is out of contract but would not relish playing at League One level. It appears he is set to join West Ham on a free transfer and is in talks with the London club.