SYMPTOMS

When trying to connect to a mail service over SSL it fails, with an error which indicates SSL negotiation failed. The service Debug logs indicate the error code 80090331, along with notes saying the SSL handshaking failed.

CAUSE

This can happen if the client connecting to the server does not support a protocol that the server is allowing. For instance, you may be trying to use a client application that supports TLS v1, but the server is only allowing TLSv1.2 or higher. If your client is an older application this may be the case.

If you have OpenSSL installed somewhere which has connectivity to the server you can check what protocols are supported with the following commands (this is against SMTP using STARTTLS). Each command will force a specific version of TLS to be used, or the connection fails:

openssl s_client -starttls smtp -tls1 -connect host:25

openssl s_client -starttls smtp -tls1_1 -connect host:25

openssl s_client -starttls smtp -tls1_2 -connect host:25

If you are trying to determine what protocols the client supports, then Wireshark could be used, but some knowledge of how to use this utility is needed:

https://www.wireshark.org/

SOLUTION

Upgrading the client application to support the same protocols of the server is the recommended solution. Downgrading the allowed server protocols is not recommended for security reasons. The MailEnable software will use the defaults of the server, so changes to the Windows settings, such as changing the allowed protocols, will be reflected in the mail services. You may find IIS Crypto an easy to use application to chekc what protocols are enabled on the server:

https://www.nartac.com/Products/IISCrypto/