Unlike Outlook 2007, Outlook 2010 will not hide itself by default when you minimize it.
Right-click on the Outlook icon in the taskbar next to the clock.
Tip: if you can't see it, click on the arrow button.
Select "Hide when minimized" from the context menu: the feature is enabled when a checkmark shows next to that option. If you still see Outlook in the taskbar, click on it to restore it, and once more to minimize. The taskbar button will be hidden.