Not able to ssh after reboot on aws ec2 instance "Connection timed out"

Question

After rebooting an aws ec2 instance I am not able anymore to ssh into it, putty on Windows is telling me "Network error: Connection timed out" and the same happens from the command line on a Linux machine.

The machine is running Ubuntu 16.04 The instance has an Elastic IP attached to it (so that shouldn't create any reboot issue due to IP changes).

The instance seems to be running as expected (the apache website is running fine) however I am not able to connect anymore into the instance shell.

Looking at the system logs I noticed lot of lines containg the "FAILED" word:

[[0;1;31mFAILED[0m] Failed to start OpenBSD Secure Shell server.
See 'systemctl status ssh.service' for details.

and an occurrence of these ones:

[[0;1;31mFAILED[0m] Failed to start Service for snap ap...azon-ssm-agent.amazon-ssm-agent.
See 'systemctl status snap.amazon-ssm-a...mazon-ssm-agent.service' for details.

I configured the ec2 instance User Data in order to get the output of the ssh:

service sshd status > /var/www/html/XXXX/sshlog.txt

The log output is this one:

â— ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: failed (Result: start-limit-hit) since Wed 2019-05-15 11:03:05 UTC; 1s ago
  Process: 1454 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=255)

May 15 11:03:05 ip-172-31-4-244 systemd[1]: Failed to start OpenBSD Secure Shell server.
May 15 11:03:05 ip-172-31-4-244 systemd[1]: ssh.service: Unit entered failed state.
May 15 11:03:05 ip-172-31-4-244 systemd[1]: ssh.service: Failed with result 'start-limit-hit'.
May 15 11:03:06 ip-172-31-4-244 systemd[1]: Stopped OpenBSD Secure Shell server.
May 15 11:03:06 ip-172-31-4-244 systemd[1]: ssh.service: Start request repeated too quickly.
May 15 11:03:06 ip-172-31-4-244 systemd[1]: Failed to start OpenBSD Secure Shell server.
May 15 11:03:06 ip-172-31-4-244 systemd[1]: ssh.service: Failed with result 'start-limit-hit'.
May 15 11:03:06 ip-172-31-4-244 systemd[1]: ssh.service: Start request repeated too quickly.
May 15 11:03:06 ip-172-31-4-244 systemd[1]: Failed to start OpenBSD Secure Shell server.
May 15 11:03:06 ip-172-31-4-244 systemd[1]: ssh.service: Failed with result 'start-limit-hit'.

Can anybody help me investigating the issue?

Thanks in advance!

Answer 1

It seems to me that there may be a config problem with the ssh server. The log entries you're seeing are just the ssh server complaining about being started too frequently. A more meaningful error message is probably somewhere earlier in the logs.

I'd suggest creating a new, temporary instance, then mounting your instance's volume (you first need to detach it after shutting down the instance) and taking a close look at the logs. Then I'd try to replace the /etc/ssh/sshd_config file with the one from the temporary instance (make a copy of the original one first). Then shut down the temporary instance, detach the volume, reattach it to your first instance, and start it. This may fix your problem. Or it may not -- but it's the first thing to try.