What protocol does SOTI use when connecting to a Windows Desktop computer?
Definitely not remote desktop. Control is provided with proprietary protocol via a proprietary device agent binary from Soti, though a well documented MDM API/protocol is defined by Microsoft for recent Windows versions such as Windows 10.
Both Microsoft and MDM/EMM vendors like Soti implemented multiple certificates and other security mechanisms (e.g. company domain/FQDN, Windows Notification Service WNS centrally administered/authenticated by Microsoft, etc.) to ensure a Windows device can be managed only by the intended MDM/EMM server the device is previously enrolled to.
Microsoft, like Google and Apple respectively for the Android and iOS platforms, defines in the Windows MDM API's what controls are possible for different use cases (company-owned, BYOD, etc.) to balance device controllability and privacy of device end-users.