How to remove unwanted desktop packages from debian without deleting everything?

I've just installed debian wheezy on an old laptop (using the xfce4 desktop environment via debian-live-7.7.0-amd64-xfce-desktop.iso). I don't intend to use it as an office machine, so I decided to remove the main desktop applications, as, e.g., they keep cropping up in apt-get upgrades:

sudo apt-get remove libreoffice orage brasero exfalso quodlibet gimp imagemagick ristretto xsane

then I cleaned up any dependencies:

sudo apt-get autoremove

I did this via an ssh login from another, client machine as I wanted to navigate the application menus on the old laptop at the same time as typing in the packages to remove.

Only problem is that I didn't look to closely at all the xfce4 packages that I was warned would be removed, and it turns out that I have removed my whole desktop environment! When I saw a message on my client machine

Removing lightdm ...
[ ok ] Stopping Light Display Manager: lightdm.

my old laptop screen went blank! Clearly I have done something very very wrong, although I can't see what.

I have some other debian wheezy machines that I'm using as headless servers, and I'd intended to run the same command line on them once I'd got my old laptop sorted, but I don't want to go ahead until I'm sure I know what I'm doing.

So what did I do wrong, and how should I go about removing these desktop applications without suffering the catastrophic loss of my whole desktop?

Update: it seems if I simulate reinstalling xfce4 via

sudo apt-get install -s xfce4

the only package out of the ones I didn't want that gets reinstalled is orage, the calendar that integrates into the desktop environment.