After debuggin the ssl_error_rx_record_too_long problem myself, it seems there is no one root of the error, not a single point to change. So here is how to track the error down on your system:
Go to Apache's /etc/apache2/ports.conf and comment out Listen on port 443, restart apache and see if you can connect to the port.
Very handy for this is the command:
openssl s_client -connect www.example.com:443 -state -debug
Before you deactivated port 443 it probably gave you:
SSL_connect:error in unknown state
07361128:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:749:
no peer certificate available
No client certificate CA names sent
SSL handshake has read 7 bytes and written 226 bytes
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
After deactivating port 443 it should give you
connect: Connection refused
Ok, so the port config seems okay.
do a grep over all your virtual hosts config files and see where you used *:443 or where port 443 is referenced, then comment out all other Vhosts except one. Restart Apache and see if you can connect with the above openssl debug command. By now, with only one site using SSL, it should work and the openssl debug output should look sane.
As you might have figured out by now, in most cases the ssl_error_rx_record_too_long error seems to indicate that you have configured a lot of contradicting stuff on port 443. So go back to a simple config to get it basically working again and then reactivate site by site from there. Keep in mind that at least on older versions of Apache and the SSL standard, there is only "one SSL host per IP".
Workarounds might be using GnuTls. But essentially with Named Virtual Hosts and multiple SSL certificates per IP, the problem is that Apache has to look into the Host name of the request ot know which VHost should answer the request. But at the time of the connection to Apache, this Header is encrypted with a certificate, and with multiple certificate can't be sure which certificate to use to decrypt, thus it logically can't know which VHost should answer.
Apart from that, if you still have trouble, try issuing a self signed certificate next, and run your single SSL VHost with that, to see if your certificate is working alright, but Apache checks the certificate on startup anyway.