Location, volume and insurance have been my pitfalls. Location because I originally worked out of my driveway until zoning laws truly became an issue (your level of professionalism is somewhat affected when you work out of your house..) but that situation has now been fixed since I am renting out a shop. Volume is normally not *too* much of an issue for me - I have enough business to sustain myself as a college student with a lot of money saved up, but I couldn't support a family yet I don't think. That is to be expected when starting a business like this - you need to build up clientele, it isn't an overnight thing.
Insurance has also been an issue because as I've explained in other posts, no IC's would take me on when I was working out of my driveway and more recently, the excuse has been that I don't have enough experience or something. I still use a rotary on 9 out of 10 cars that I detail because I'm careful and I have the money to back-up any terrible mistake I might make.
I think the business management part of my business is the easiest - for me anyway. If I could find someone I trust to detail in my place and thus allow me to go out and run the business, I would much rather do that because my passion lies in business, not just detailing.