Do you have the same swing speed on your second serve as your first? To fuzz's point, if your serve is technically sound and reliable as you say, racket head,speed + swing speed should be the close if not the same on both serves. The only difference is how much spin you put on the ball and where you aim.
Maybe the solution is to practice hard spinning first serves (kick, twist, slice, or however you hit your spin serve) so that you get get familiar feel with the control + motion of your spin serve. Once you get the hang of it, it will be a matter of how you decide to make contact.