10

Linux systems sometimes remount the root file system as read-only, e.g. if there's an I/O error.

I have a machine that becomes useless when this happens, and I end up rebooting it manually.

Is there a way to make Linux just automatically reboot when this happens? A read-only mount is useless to me.

Improve this question
13
  • 15
    I'd also investigate the source of these I/O errors. The last time an ext2 filesystem went readonly for me was in 1994, and the cause could be traced to a broken CPU fan.
    – Simon Richter
    Nov 23, 2021 at 15:35
  • 12
    You have an XY problem here. The correct solution is not to make the system reboot on an IO error (the accepted answer explains how to do that, but that’s actually rather risky for multiple reasons), it’s to fix the root cause of the IO errors, because then the filesystem will not randomly get mounted read-only. If it’s only intermittent and the storage device is good, you probably have suspect RAM or a flaky PSU, both of which can cause much bigger issues than a simple filesystem error.
    – Austin Hemmelgarn
    Nov 23, 2021 at 21:27
  • 7
    Rebooting rather than sort out the reason for the R/O remount has a high likelihood of making the problem worse - especially if it fails to mount the system on reboot and you're now stuck with an entirely unresponsive system.
    – Shadur
    Nov 24, 2021 at 15:02
  • 7
    @user541686 You have random IO errors. That will cause other problems eventually (and trust me, they will be much more of a pain to fix than just rebooting the system), hence my assertion that this is an XY problem. The fact that you do not recognize the X as a problem does not make it any less of an XY problem.
    – Austin Hemmelgarn
    Nov 24, 2021 at 17:34
  • 4
    @user541686, if there's relevant information, provide it. Don't just say "trust me".
    – Mark
    Nov 24, 2021 at 21:13

2 Answers 2

Reset to default
23

I deduce you are using ext3 or ext4 as the file system. If so, you can mount it with the errors=panic option and configure watchdog to reboot your system in case a panic happen.

While more complex than roelvanmeer's answer (which I upvoted), it has the added bonus of working for all panic-level kernel crash.

As suggested by NikitaKipriyanov, setting the panic=5 kernel boot option can be a simpler alternative to watchdog (which has more configuration options but it is slightly more complex as result).

Improve this answer
8
  • 2
    Alternative to watchdog might be adding something like kernel.panic = 5 into the /etc/sysctl.d/panic-reboot.conf.
    – Nikita Kipriyanov
    Nov 23, 2021 at 9:51
  • Thank you! I'll give this a shot. Hopefully it won't fail to reboot!
    – user541686
    Nov 23, 2021 at 10:07
  • @NikitaKipriyanov good suggestion, I'll edit my answer. Thanks.
    – shodanshok
    Nov 23, 2021 at 10:45
  • 6
    warning: probable reboot loop
    – joshudson
    Nov 23, 2021 at 15:54
  • 3
    @AndrewHenle: I've brought a lot of systems up with a trashed root filesystem. Usually I can' take over the boot process and get fsck to run because the damage rarely hits /sbin or files that haven't changed in awhile.
    – joshudson
    Nov 25, 2021 at 16:13
14

Maybe not a very pretty solution, but my first thought would be to run a command from cron every minute:

test -w / || reboot
Improve this answer
8
  • +1 thanks, this'll be a great fallback if the other solution fails!
    – user541686
    Nov 23, 2021 at 10:07
  • I think it is not guaranteed that test -w checks if the filesystem is read-write. Though GNU test and test built into bash seems to do that. --- Here you can see what should POSIX-compliant test do: pubs.opengroup.org/onlinepubs/9699919799/utilities/… As I understand it test is only required to check the access rights of the file.
    – pabouk - Ukraine stay strong
    Nov 23, 2021 at 18:19
  • 1
    In which case tee -a /root/.bash_history < /dev/null || reboot will work.
    – joshudson
    Nov 23, 2021 at 19:15
  • 2
    @shodanshok that could lead to unexpected reboots - or reboot loops - from error conditions unrelated to filesystem errors, eg temporary upsets of the libc installation, OOM conditions, anything that could make touch fail....
    – rackandboneman
    Nov 24, 2021 at 19:23
  • 3
    @rackandboneman sure - but any script with || reboot is subject to these issues. Moreover, if touch fails on your system due to libc issues, you probably have worse problem then a reboot loop. Anyway, as stated in my answer, watchdog is the way to go for more advanced needings.
    – shodanshok
    Nov 24, 2021 at 20:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .