I was troubleshooting some relatively minor SSL changes that had reportedly worked before but no longer worked. After switching back to old certificates and it still didn’t work I was mystified. It looked like it was working. I could telnet to the port so I knew that HTTP.SYS/IIS was answering. However, the browser refused to return anything on HTTPS.
I ended up breaking out Network Monitor 3.3 and getting a capture. What I saw in the packet capture was odd…
- ARP for the target address. (Good we’re starting from scratch)
- TCP connection negotiated (Good, we have a channel.)
- SSL Negotation and Key Exchange – with a TCP FIN flag set. The FIN flag is “I’m done let’s close this channel.” (This isn’t good. We shouldn’t negotiate and then turn around and close the channel.)
- Steps 1-3 repeated except instead of a FIN flag I see a TCP RST flag. That’s a “I’m not talking to you any more – go away” In other words, the connection is terminated abruptly. (This is really bad. This is the point at which the client knows there’s something horribly wrong.)
After some work we realized that the SSL certificates were self-signed and there was something wrong. We moved to a certificate from a CA and the servers started accepting connections on SSL without any more issues. I’m not clear exactly what the heck the problem was with the certificates, but replacing them definitely resolved the issue.