mkdir: cannot create directory ‘test’: Read-only file system

We had to reboot a server we manage which had frozen/crashed during the night. The server rebooted but none of the services would come online. We tried filtering the logs to a new file to drill down the errors at the time of the crash when the system threw the following error:

mkdir: cannot create directory ‘test’: Read-only file system

We couldn’t do anything with the filesystem mounted so we rebooted the server to recovery and tried to fix the filesystem with the following command:

e2fsck -p /dev/md2

/dev/md2 because this was our OS raid mount point.

The scan returned the following:

/dev/md2 contains a file system with errors, check forced.
/dev/md2: Inode 27791390 extent tree (at level 1) could be narrower.  IGNORED.
/dev/md2: Deleted inode 37879818 has zero dtime.  FIXED.
/dev/md2: 1518971/57778176 files (1.3% non-contiguous), 37334686/231112704 blocks

We proceeded to reboot the server, this time the partition was mounted with write support.

This isn’t the end, technically. The way the server crashed and then booting back with the filesystem mounted as reboot could signal a failing drive. We will cover how to scan and check the health of your drives to see if they are failing and need to be replaced.