[Bug] - Database drops every time container restarts #108
Assignees
FrederikBaerentsen
Clear assignees
FrederikBaerentsen
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: FrederikBaerentsen/BrickTracker#108
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
I run my containers on a docker swarm cluster, using Ceph as shared storage between all nodes. I have had a problem for a while that anytime my BrickTracker container moves hosts (or updates) it acts like there is no database, and I have to restore a copy of the DB to get back going again.
I think this is a sqllite issue using shared storage (similar things are reported for other apps using NFS), and I have also had this issue using GlusterFS as the shared storage between swarm nodes. For other apps, have solved this by moving the DB over to a PostgreSQL instance. So I would propose adding support for an external DB like PostgreSQL to eliveate issues for users with NFS, Gluster, or Ceph storage
I have the same issue in running the container in Unraid, the database seems to be reset every time.
I thought it was an unraid issue may be, but from what you are saying, it's probably related to the SQL.
Or is possible that the start of the container rewrite a database file ? that would not be only in theses instances...
In the next version doing some breaking changes that moves all the user content (DB, images, instructions) to their own folder. This should hopefully fix these issues. The plan is a release around Christmas. Need some time off work to get the last fixes in.
I've been looking info your issue @cainmp. I'm not using Docker Swarm or Ceph myself, but know a bit about it.
BrickTracker uses a custom SQLite implementation with handwritten SQL queries rather than a standard ORM like SQLAlchemy. This means adding PostgreSQL support would require a complete rewrite of the database backend.
Your issue seems to be a known limitation of SQLite with network filesystems (NFS, Ceph, GlusterFS). SQLite relies on file-level locks that don't work reliably over network storage, leading to corruption and data loss. This is documented in SQLite's official FAQ.
I suggest for Docker Swarm deployments to use placement constraints to pin BrickTracker to a specific node with local storage instead of shared storage. This ensures the SQLite database uses local filesystem instead of network storage, avoiding all the corruption issues.
Unless something changes i don't see myself switching from SQLite to another database like PostgreSQL, as the rewrite is a major task that have very little upside for the overall project.
I'm closing the issue for now.
I think I fixed this with your latest update, as when I was migrating to the new file layout structure, I did not have a database file at all. I think I had a file permission issue that was preventing the DB from ever being created. As post update, I can no longer reproduce the issue.