Having to fire someone.
What Do You Believe Is The Most Difficult Part Of Being A Supervisor Of People?
From personal experience, three things come to mind. First, finding out how to motivate people to do well- it isn't difficult, exactly, but it does consume a lot of time. Second, if you have someone who has to be micro-managed, doing this successfully is (or can be) a large hassle. Third, binding the team together so that you only have to supervise a little bit and they all do what they've been assigned to do... It's not that it's difficult, I think my knowledge of how the US special forces have historically done this helped me, but it takes time, effort, and forethought and a willingness (and ability) to reward people both individually for big things that they accomplish and as a group for working together.
Just never forget where you have come from or what you have done to achieve your position. To be a good listener.