How to unf*ck Mastodon

Hello! If you found this page for a reason, I feel sorry for you. There’s quite a lot of reasons and ways that Mastodon can break, so I figured I’d start a little place to collect all these issues, also to remind myself on how I fixed them

Clearing the cache with tootctl

Mastodon at least provides you with a useful little tool to manage and reset things in case of breakage; tootctl.

One thing that is very worth attempting in the case you ran your migrations, but there are still problems, is to clear the cache.

To clear the cache, simply run: tootcl cache clear

source

Database Trouble

This is probably going to be your main pain point, as both PostgreSQL can be a pain between upgrades, as well as Rails and Mastodon itself.

Failed/Pending migrations

When I started writing this post, the reason was that I just spent a good few hours trying to fix Mastodon after it had half-completed a bunch of migrations.

The main giveaway for this would be if you start your server, it appears to run, but occasionally there’s some failed executions of Puma with some PostgreSQL related errors that say something like “ ActionView::Template::Error “ … “Did you mean? {SOME TABLES OR USERS OR ENTITIES}”

To check all migrations have run successfully, from your mastodon installation, run: RAILS_ENV=production bundle exec rake db:migrate:status

If you run docker-compose, you probably want to run something like docker compose run -i web bash (Mastodon does not have to run for this)

If you see any status marked as ‘down’ here, now would be a good time to upgrade. NOTE: Mastodon MUST be stopped before doing this, or you will corrupt your database.

If you see any status marked as *** NO FILE *** then something went wrong, you might have to update your local git repo (that is updated separately from the container itself), and then run the migrations again. If it persists, see ‘Failed migrations’.

After you stopped Mastodon:

  • run docker compose run -i web bash again if you use docker-compose- this should also automatically bring up the postgres database and redis
  • if you run it in some other way, spin up the database before running the rails bundle command below
  • run RAILS_ENV=production bundle exec rake db:migrate and do NOT interrupt this process, it should automatically finish
  • If it has finished, you can exit out of the container if you use docker, and restart your mastodon server as normal. If all goes well it should come back up, if not.. see the next section

Failed migrations

If you still see *** NO FILE *** or something else is wack and you suspect that the migration didn’t go over succesfully, you might have to remove the migrations from the migrations table and run them again. I’m going to use pgAdmin to edit the database, also since showing this here may be useful for other reasons in the future.

  • Again make sure Mastodon is not running
  • If you use docker-compose, edit the docker-compose.yml file so it contains pgadmin: 
    pgadmin:
  image: dpage/pgadmin4
  environment:
    PGADMIN_DEFAULT_EMAIL: "admin@admin.com" # Set your desired email and password
    PGADMIN_DEFAULT_PASSWORD: "adminpassword"
  volumes:
    - ./pgadmin_data:/var/lib/pgadmin
  ports:
    - "127.0.0.1:8080:80" # This would make pgAdmin accessible at http://localhost:8080 on your machine
  networks:
    - internal_network
    - external_network
  depends_on:
    - db
  • Make sure to edit the networks such that it can contact your mastodon postgresql server
  • If you don’t use docker-compose, run pgadmin somehow and make sure it has access to the same network as the mastodon postgres server is on
  • Go to your schema_migrations table and check all the migrations listed in it.
  • Run this query: DELETE FROM schema_migrations WHERE version::bigint > [last_known_good_migration] where ‘last_known_good_migration’ is the last migration that you know worked out, or where you want to start (don’t go too far back because it’s an intensive process)
  • Now that you deleted that migration, run the rails migration mentioned above again.
  • if the migration fails, check the output, if it mentions that the key/constraint/whatever already exists, re-add the migration to the database using this query: INSERT INTO schema_migrations (version) VALUES ('MIGRATION_THAT_YOU_JUST_RAN');
  • repeat until you’re back at the latest migration
  • once you’re done, try and boot mastodon again.

Updated: