After A2 Hosting managed to annihilate a virtual server due to some cloud server problems, I was in the position of rebuilding that server. I did not like the name showing in the bash prompt, though.
To do so, edit the name in /etc/hostname and in /etc/hosts to what you want, and then restart the hostname service.
sudo service hostname restart
The next time you log in, the hostname should be something more to your liking!