TIL how to debug a broken nextcloud login

For the past month I was confronted with the downside of self-hosting. I use Nextcloud for me and my partner’s calendar, online file storage, and notes syncing. I host Nextcloud on a small 7EUR/month Hetzner VPC. Unfortunately, about a month ago it stopped working. Phone apps would not sync the calendar, Joplin notes were not synced across devices and I could not log into my Nextcloud account. I avoided trying to fix the issue quickly, because I know from experience that a short bug investigation in a self-hosting project can quickly spiral into a weekend long endeavour. Fortunately, I had some time this morning and I could fix the issue quickly. Leaving this here for future reference.

Problem

Phone and Mac apps stopped syncing data with the Nextcloud instance. In my browser I could access the login page, but after entering the details I was being sent back to the same login page that you see below without any error message.

Investigation

It is quite difficult to understand the issue without an error message. The Nextcloud docs suggest to look at the server logs to get a better idea.

To access the logs I use ssh to log into the VPC and opened the nextcloud.log log file.

$ ssh username@[VPC-IP] 
$ vim /var/www/nextcloud/data/nextcloud.log

Trying to open the file in vim showed me the likely issue. The log file was so large that it would take minutes to open. From the error message I could see that nextcloud ran out of space during previous requests.

Solution

Truncate the big log file using the following steps:

1. Put nextcloud in maintenance mode.

Set 'maintenance' => true in the config file /var/www/nextcloud/config/config.php

# config.php
<?php
[...]
  'dbtype' => 'mysql',
  'version' => '22.2.3.0',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'installed' => true,
  'maintenance' => true,
  'theme' => '',
  'loglevel' => 0,
[...]
);

2. Looking at the config file, I also noticed that I had loglevel set to 0 (the most verbose), which is the likely cause of the file size. Nextcloud has the following log levels:

• 0: DEBUG: All activity; the most detailed logging.
• 1: INFO: Activity such as user logins and file activities, plus warnings, errors, and fatal errors.
• 2: WARN: Operations succeed, but with warnings of potential problems, plus errors and fatal errors.
• 3: ERROR: An operation fails, but other services and operations continue, plus fatal errors.
• 4: FATAL: The server stops.

So I changed 'loglevel' => 2 to prevent the same issue from reappearing in a year.

3. Truncate the log file to 0 bytes to free space using the unix truncate   command:

$sudo -u www-data truncate /var/www/nextcloud/data/nextcloud.log --size 0

4. Disable maintenance mode 'maintenance' => false in the config file.

This allowed me to login using my browser. Moreover, all apps started syncing again.