Does your laptop have any other com ports listed in device manager, when the USB-serial cable is not plugged into the laptop? (most don't, these days) If it does, and they are not in use, sometimes disabling them may help. If you are not sure, I would leave them alone, but note which com number they are, to be sure not to set the USB serial to those numbers, and not to pic those numbers in the kelly program.
Now, to make sure what the specific problem is, is it that you literally cannot run (open) the kelly program at all? Meaning, when you double click on it, or click on it and press the enter key on the keyboard, or right click on it and choose Open, that it will not run at all?
Or does it run but give an error message? If so, what exactly is the error message, word for word with all info it provides.
Also, if the controller is still running or trying to run the motor, AFAIK it won't allow communication with the outside world because at least some kellys will be bricked (made permanently unusable) by connecting to the program while the motor is spinning for any reason. (no personal experience with that, but there have been posts around the web for that). So if it is still running, and the program simply won't talk to the ocntroller, then that would be why.
How long have you waited so far for the Identification Angle to complete? The manual says it can take several minutes.
https://kellycontroller.com/wp-content/uploads/kls-h/KellyKLS-HUserManualV1.11-100G.pdf
and this post
https://endless-sphere.com/forums/viewtopic.php?t=95368#p1398898
indicates that sometimes it can take multiple retries of the process to get it to identify the motor setup, before it will give the "error" indicating it's done, and go back to normal "85" mode.
Are there any error codes? (beeps)
I'm still poking around for info on this, but there are a LOT of posts on the internet about identify angle procedure problems with kellys, and almost none so far are relevant to your issue.