Worth leaving medical school?

It sounds like you've already made up your mind. You are clearly intelligent and thoughtful so go ahead and do what you think is right - I am pretty sure that what you think is the right decision.