Looks like the issue may be based on how the worker thread is stopped and restarted. We worked with support on this, and we think that the connection time is showing as much longer than it actually is - due to the thread needing to get restarted. We were able to troubleshoot this by changing the number of transactions per worker to 1, and the times were always 15+ second connections.
We are waiting to hear from support, and I will update the thread as we move forward.