BrickTracker/CHANGELOG.md

Changelog

1.3

Breaking Changes

Data Folder Consolidation

Warning

BREAKING CHANGE: Version 1.3 consolidates all user data into a single data/ folder for easier backup and volume mapping.

• Path handling: All relative paths are now resolved relative to the application root (/app in Docker)

  • Example: data/app.db → /app/data/app.db

• New default paths (automatically used for new installations):

  • Database: data/app.db (was: app.db in root)
  • Configuration: data/.env (was: .env in root) - optional, backward compatible
  • CSV files: data/*.csv (was: *.csv in root)
  • Images/PDFs: data/{sets,parts,minifigures,instructions}/ (was: static/*)

• Configuration file (.env) location:

  • New recommended location: data/.env (included in data volume, settings persist)
  • Backward compatible: .env in root still works (requires volume mount for admin panel persistence)
  • Priority: data/.env > .env (automatic detection, no migration required)

• Migration options:

  1. Migrate to new structure (recommended - single volume for all data including .env)
  2. Keep current setup (backward compatible - old paths continue to work)

See Migration Guide for detailed instructions

Default Minifigures Folder Change

Warning

BREAKING CHANGE: Default minifigures folder path changed from minifigs to minifigures

• Impact: Users who relied on the default BK_MINIFIGURES_FOLDER value (without explicitly setting it) will need to either:
  1. Set BK_MINIFIGURES_FOLDER=minifigs in their environment to maintain existing behavior, or
  2. Rename their existing minifigs folder to minifigures
• No impact: Users who already have BK_MINIFIGURES_FOLDER explicitly configured
• Improved consistency across documentation and Docker configurations

New Features

• Live Settings changes
  • Added live environment variable configuration management system
    • Configuration Management interface in admin panel with live preview and badge system
    • Live settings: Can be changed without application restart (menu visibility, table display, pagination, features)
    • Static settings: Require restart but can be edited and saved to .env file (authentication, server, database, API keys)
    • Advanced badge system showing value status: True/False for booleans, Set/Default/Unset for other values, Changed indicator
    • Live API endpoints: /admin/api/config/update for immediate changes, /admin/api/config/update-static for .env updates
    • Form pre-population with current values and automatic page reload after successful live updates
  • Fixed environment variable lock detection in admin configuration panel
    • Resolved bug where all variables appeared "locked" after saving live settings
    • Lock detection now correctly identifies only Docker environment variables set before .env loading
    • Variables set via Docker's environment: directive remain properly locked
    • Variables from data/.env or root .env are correctly shown as editable
  • Added configuration persistence warning in admin panel
    • Warning banner shows when using .env in root (non-persistent)
    • Success banner shows when using data/.env (persistent)
    • Provides migration instructions directly in the UI
• Spare Parts
  • Added spare parts control options
    • BK_SKIP_SPARE_PARTS: Skip importing spare parts when downloading sets from Rebrickable (parts not saved to database)
    • BK_HIDE_SPARE_PARTS: Hide spare parts from all parts lists (parts must still be in database)
    • Both options are live-changeable in admin configuration panel
    • Options can be used independently or together for flexible spare parts management
    • Affects all parts displays: /parts page, set details accordion, minifigure parts, and problem parts
• Pagination
  • Added individual pagination control system per entity type
    • BK_SETS_SERVER_SIDE_PAGINATION: Enable/disable pagination for sets
    • BK_PARTS_SERVER_SIDE_PAGINATION: Enable/disable pagination for parts
    • BK_MINIFIGURES_SERVER_SIDE_PAGINATION: Enable/disable pagination for minifigures
    • Device-specific pagination sizes (desktop/mobile) for each entity type
    • Supports search, filtering, and sorting in both server-side and client-side modes
• Peeron Instructions
  • Added Peeron instructions integration
    • Full image caching system with automatic thumbnail generation
    • Optimized HTTP calls by downloading full images once and generating thumbnails locally
    • Automatic cache cleanup after PDF generation to save disk space
• Parts checkmark
  • Added parts checking/inventory system
    • New "Checked" column in parts tables for tracking inventory progress
    • Checkboxes to mark parts as verified during set walkthrough
    • BK_HIDE_TABLE_CHECKED_PARTS: Environment variable to hide checked column
• Set Consolidation
  • Added set consolidation/grouping functionality
    • Automatic grouping of duplicate sets on main sets page
    • Shows instance count with stack icon badge (e.g., "3 copies")
    • Expandable drawer interface to view all set copies individually
    • Full set cards for each instance with all badges, statuses, and functionality
    • BK_SETS_CONSOLIDATION: Environment variable to enable/disable consolidation (default: false)
    • Backwards compatible - when disabled, behaves exactly like original individual view
    • Improved theme filtering: handles duplicate theme names correctly
    • Fixed set number sorting: proper numeric sorting in both ascending and descending order
    • Mixed status indicators for consolidated sets: three-state checkboxes (unchecked/partial/checked) with count badges
      • Template logic handles three states: none (0/2), all (2/2), partial (1/2) with visual indicators
      • Purple overlay styling for partial states, disabled checkboxes for read-only consolidated status display
      • Individual sets maintain full interactive checkbox functionality
• Statistics
  • Added comprehensive statistics system (#91)
    • New Statistics page with collection analytics
    • Financial overview: total cost, average price, price range, investment tracking
    • Collection metrics: total sets, unique sets, parts count, minifigures count
    • Theme distribution statistics with clickable drill-down to filtered sets
    • Storage location statistics showing sets per location with value calculations
    • Purchase location analytics with spending patterns and date ranges
    • Problem tracking: missing and damaged parts statistics
    • Clickable numbers throughout statistics that filter to relevant sets
    • BK_HIDE_STATISTICS: Environment variable to hide statistics menu item
    • Year-based analytics: Sets by release year and purchases by year
      • Sets by Release Year: Shows collection distribution across LEGO release years
      • Purchases by Year: Tracks spending patterns and acquisition timeline
      • Year summary with peak collection/spending years and timeline insights
    • Enhanced statistics interface and functionality
      • Collapsible sections: All statistics sections have clickable headers to expand/collapse
      • Collection growth charts: Line charts showing sets, parts, and minifigures over time
      • Configuration options: BK_STATISTICS_SHOW_CHARTS and BK_STATISTICS_DEFAULT_EXPANDED environment variables
• Admin Page Section Expansion
  • Added configurable admin page section expansion
    • BK_ADMIN_DEFAULT_EXPANDED_SECTIONS: Environment variable to specify which sections expand by default
    • Accepts comma-separated list of section names (e.g., "database,theme,instructions")
    • Valid sections: authentication, instructions, image, theme, retired, metadata, owner, purchase_location, status, storage, tag, database
    • URL parameters take priority over configuration (e.g., ?open_database=1)
    • Database section expanded by default to maintain original behavior
    • Smart metadata handling: sub-section expansion automatically expands parent metadata section
• Duplicate Sets filter
  • Added duplicate sets filter functionality
    • New filter button on Sets page to show only duplicate/consolidated sets
    • BK_SHOW_SETS_DUPLICATE_FILTER: Environment variable to show/hide the filter button (default: true)
    • Works with both server-side and client-side pagination modes
    • Consolidated mode: Shows sets that have multiple instances
    • Non-consolidated mode: Shows sets that appear multiple times in collection
• Bricklink Links
  • Added BrickLink links for sets
    • BrickLink badge links now appear on set cards and set details pages alongside Rebrickable links
    • BK_BRICKLINK_LINK_SET_PATTERN: New environment variable for BrickLink set URL pattern (default: https://www.bricklink.com/v2/catalog/catalogitem.page?S={set_num})
    • Controlled by existing BK_BRICKLINK_LINKS environment variable
• Dark Mode
  • Added dark mode support
    • BK_DARK_MODE: Environment variable to enable dark mode theme (default: false)
    • Uses Bootstrap 5.3's native dark mode with data-bs-theme attribute
    • Live-changeable via Admin > Live Settings
    • Setting persists across sessions via .env file
• Alphanumetic Set Number
  • Added alphanumeric set number support
    • Database schema change: Set number column changed from INTEGER to TEXT
    • Supports LEGO promotional and special edition sets with letters in their numbers
    • Examples: "McDR6US-1", "COMCON035-1", "EG00021-1"

Improvements

• Improved WebSocket/Socket.IO reliability for mobile devices
  • Changed connection strategy to polling-first with automatic WebSocket upgrade
  • Increased connection timeout to 30 seconds for slow mobile networks
  • Added ping/pong keepalive settings (30s timeout, 25s interval)
  • Improved server-side connection logging with user agent and transport details
• Fixed dynamic sort icons across all pages
  • Sort icons now properly toggle between ascending/descending states
• Improved DataTable integration
  • Disabled column header sorting when server-side pagination is enabled
  • Prevents conflicting sort mechanisms between DataTable and server-side sorting
• Enhanced color dropdown functionality
  • Automatic merging of duplicate color entries with same color_id
  • Keeps entries with valid RGB data, removes entries with None/empty RGB
  • Preserves selection state during dropdown consolidation
  • Consistent search behavior (instant for client-side, Enter key for server-side)
  • Mobile-friendly pagination navigation
• Added performance optimization
  • SQLite WAL Mode:
    • Increased cache size to 10,000 pages (~40MB) for faster query execution
    • Set temp_store to memory for accelerated temporary operations
    • Enabled foreign key constraints and optimized synchronous mode
    • Added ANALYZE for improved query planning and statistics
  • Database Indexes (Migration 0019):
    • High-impact composite index for problem parts aggregation (idx_bricktracker_parts_id_missing_damaged)
    • Parts lookup optimization (idx_bricktracker_parts_part_color_spare)
    • Set storage filtering (idx_bricktracker_sets_set_storage)
    • Search optimization with case-insensitive indexes (idx_rebrickable_sets_name_lower, idx_rebrickable_parts_name_lower)
    • Year and theme filtering optimization (idx_rebrickable_sets_year, idx_rebrickable_sets_theme_id)
    • Additional indexes for purchase dates, quantities, sorting, and minifigures aggregation
  • Statistics Query Optimization:
    • Replaced separate subqueries with efficient CTEs (Common Table Expressions)
    • Consolidated aggregations for set, part, minifigure, and financial statistics
  • Added default image handling for sets without images
    • Sets with null/missing images from Rebrickable API now display placeholder image
    • Automatic fallback to nil.png from parts folder for set previews
    • Copy of nil placeholder saved as set image for consistent display across all routes
    • Prevents errors when downloading sets that have no set_img_url in API response
  • Fixed instructions download from Rebrickable
    • Replaced cloudscraper with standard requests library
    • Resolves 403 Forbidden errors when downloading instruction PDFs
  • Fixed instructions display and URL generation
    • Fixed "Open PDF" button links to use correct data route
    • Corrected path resolution for data/instructions folder
    • Fixed instruction listing page to scan correct folder location
    • Fixed Peeron PDF creation to use correct data folder path
  • Fixed foreign key constraint error when adding sets
    • Rebrickable set is now inserted before BrickTracker set to satisfy FK constraints
    • Resolves "FOREIGN KEY constraint failed" error when adding sets
  • Fixed atomic transaction handling for set downloads
    • All database operations during set addition now use deferred execution
    • Ensures all-or-nothing behavior: if any part fails (set info, parts, minifigs), nothing is committed
    • Prevents partial set additions that would leave the database in an inconsistent state
    • Metadata updates (owners, tags) now defer until final commit

1.2.4

Warning

To use the new BrickLink color parameter in URLs, update your .env file: BK_BRICKLINK_LINK_PART_PATTERN=https://www.bricklink.com/v2/catalog/catalogitem.page?P={part}&C={color}

• Add BrickLink color and part number support for accurate BrickLink URLs
  • Database migrations to store BrickLink color ID, color name, and part number
  • Updated Rebrickable API integration to extract BrickLink data from external_ids
  • Enhanced BrickLink URL generation with proper part number fallback
  • Extended admin set refresh to detect and track missing BrickLink data

1.2.3

Added search/filter/sort options to parts and minifigures.

1.2.2

Fix legibility of "Damaged" and "Missing" fields for tiny screen by reducing horizontal padding Fixed instructions download from Rebrickable

1.2.2:

This release fixes a bug where orphaned parts in the inventory table are blocking the database upgrade.

1.2.1:

This release fixes a bug where you could not add a set if no metadata was configured.

1.2.0:

Warning

"Missing" part has been renamed to "Problems" to accomodate for missing and damaged parts. The associated environment variables have changed named (the old names are still valid)

Environment

• Renamed: BK_HIDE_MISSING_PARTS -> BK_HIDE_ALL_PROBLEMS_PARTS
• Added: BK_HIDE_TABLE_MISSING_PARTS, hide the Missing column in all tables
• Added: BK_HIDE_TABLE_DAMAGED_PARTS, hide the Damaged column in all tables
• Added: BK_SHOW_GRID_SORT, show the sort options on the grid by default
• Added: BK_SHOW_GRID_FILTERS, show the filter options on the grid by default
• Added: BK_HIDE_ALL_STORAGES, hide the "Storages" menu entry
• Added: BK_STORAGE_DEFAULT_ORDER, ordering of storages
• Added: BK_PURCHASE_LOCATION_DEFAULT_ORDER, ordering of purchase locations
• Added: BK_PURCHASE_CURRENCY, currency to display for purchase prices
• Added: BK_PURCHASE_DATE_FORMAT, date format for purchase dates
• Documented: BK_FILE_DATETIME_FORMAT, date format for files on disk (instructions, theme)

Code

• Changer

  • Revert the checked state of a checkbox if an error occured

• Form

  • Migrate missing input fields to BrickChanger

• General cleanup

• Metadata

  • Underlying class to implement more metadata-like features

• Minifigure

  • Deduplicate
  • Compute number of parts

• Parts

  • Damaged parts

• Sets

  • Refresh data from Rebrickable
  • Fix missing @login_required for set deletion
  • Ownership
  • Tags
  • Storage
  • Purchase location, date, price

• Storage

  • Storage content and list

• Socket

  • Add decorator for rebrickable, authenticated and threaded socket actions

• SQL

  • Allow for advanced migration scenarios through companion python files
  • Add a bunch of the requested fields into the database for future implementation

• Wish

  • Requester

UI

• Add

  • Allow adding or bulk adding by pressing Enter in the input field

• Admin

  • Grey out legacy tables in the database view
  • Checkboxes renamed to Set statuses
  • List of sets that may need to be refreshed

• Cards

  • Use macros for badge in the card header

• Form

  • Add a clear button for dynamic text inputs
  • Add error message in a tooltip for dynamic inputs

• Minifigure

  • Display number of parts

• Parts

  • Use Rebrickable URL if stored (+ color code)
  • Display color and transparency
  • Display if print of another part
  • Display prints using the same base
  • Damaged parts
  • Display same parts using a different color

• Sets

  • Add a flag to hide instructions in a set
  • Make checkbox clickable on the whole width of the card
  • Management
    • Ownership
    • Tags
    • Refresh
    • Storage
    • Purchase location, date, price

• Sets grid

  • Collapsible controls depending on screen size
  • Manually collapsible filters (with configuration variable for default state)
  • Manually collapsible sort (with configuration variable for default state)
  • Clear search bar

• Storage

  • Storage list
  • Storage content

• Wish

  • Requester

1.1.1: PDF Instructions Download

Instructions

• Added buttons for instructions download from Rebrickable

1.1.0: Deduped sets, custom checkboxes and database upgrade

Database

• Sets
  • Deduplicating rebrickable sets (unique) and bricktracker sets (can be n bricktracker sets for one rebrickable set)

Docs

• Removed extra <br> to accomodate Gitea Markdown
• Add an organized DOCS.md documentation page
• Database upgrade/migration
• Checkboxes

Code

• Admin

  • Split the views before admin because an unmanageable monster view

• Checkboxes

  • Customizable checkboxes for set (amount and names, displayed on the grid or not)
  • Replaced the 3 original routes to update the status with a generic route to accomodate any custom status

• Instructions

  • Base instructions on RebrickableSet (the generic one) rather than BrickSet (the specific one)
  • Refine set number detection in file name by making sure each first items is an integer

• Python

  • Make stricter function definition with no "arg_or_keyword" parameters

• Records

  • Consolidate the select() -> not None or Exception -> ingest() process duplicated in every child class

• SQL

  • Forward-only migration mechanism
  • Check for database too far in version
  • Inject the database version in the file when downloading it
  • Quote all indentifiers as best practice
  • Allow insert query to be overriden
  • Allow insert query to force not being deferred even if not committed
  • Allow select query to push context in BrickRecord and BrickRecordList
  • Make SQL record counters failsafe as they are used in the admin and it should always work
  • Remove BrickSQL.initialize() as it is replaced by upgrade()

• Sets

  • Now that it is deduplicated, adding the same set more than once will not pull it fully from the Rebrickable API (minifigures and parts)
  • Make RebrickableSet extend BrickRecord since it is now an item in database
  • Make BrickSet extend RebrickableSet now that RebrickableSet is a proper database item

UI

• Checkboxes

  • Possibility to hide the checkbox in the grid ("Sets") but sill have all them in the set details
  • Management

• Database

  • Migration tool

• Javascript

  • Generic BrickChanger class to handle quick modification through a JSON request with a visual feedback indicator
  • Simplify the way javascript scripts are loaded and instantiated

• Set grid

  • Filter by checkboxes and NOT checkboxes

• Tables

  • Fix table search looking inside links pills

• Wishlist

  • Add Rebrickable link badge for sets (@matthew)

1.0.0: New Year revamp

Code

• Authentication
  • Basic authentication mechanism with ONE password to protect admin and writes
• CSV
  • Remove dependencies to numpy and panda for simpler built-in csv
• Code
  • Refactored the Python code
  • Modularity (more functions, splitting files)
  • Type hinting whenever possible
  • Flake8 linter
  • Retained most of the original behaviour (with its quirks)
• Colors
  • Remove dependency on color.csv
• Configuration
  • Moved all the hard-coded parameters into configuration variables
  • Most of the variables are configuration through environment variables
  • Force instruction, sets, etc path to be relative to static
• Docker
  • Added an entrypoint to grab PORT / HOST from the environment if set
  • Remove the need to seed the container with files (*.csv, nil files)
• Flask
  • Fix improper socketio.run(app.run()) call which lead to hard crash on ^C
  • Make use of url_for to create URLs
  • Use blueprints to implement routes
  • Move views into their own files
  • Split GET and POST methods into two different routes for clarity
• Images
  • Add an option to use remote images from the Rebrickable CDN rather than downloading everything locally
  • Handle nil.png and nil_mf.jpg as true images in /static/sets/ so that they are downloaded whenever necessary when importing a se with missing images
• Instructions
  • Scan the files once for the whole app, and re-use the data
  • Refresh the instructions from the admin
  • More lenient set number detection
  • Update when uploading a new one
  • Basic file management
• Logs
  • Added log lines for change actions (add, check, missing, delete) so that the server is not silent when DEBUG=false
• Minifigures
  • Added a variable to control default ordering
• Part(s)
  • Added a variable to control default ordering of listing
• Retired sets
  • Open the themes once for the whole app, and re-use the data
  • Do not hard fail if themes.csv is missing, simply display the IDs
  • Light management: resync, download
• Set(s)
  • Reworked the set checkboxes with a dedicated route per status
  • Switch from homemade ID generator to proven UUID4 for sets ID
    • Does not interfere with previously created sets
  • Do not rely on sets.csv to check if the set exists
  • When adding, commit the set to database only once everything has been processed
  • Added a bulk add page
  • Keep spare parts when importing
  • Added a variable to control default ordering of listing
• Socket
  • Make use of socket.io rooms to avoid broadcasting messages to all clients
• SQLite
  • Do not hard fail if the database is not present or not initialized
  • Open the database once for the context, and re-use the connection
  • Move queries to .sql files and load them as Jinja templates
  • Use named arguments rather than sets for SQLite queries
  • Allow execute() to be deferred to the commit() call to avoid locking the database for long period while importing (locked while downloading images)
• Themes
  • Open the themes once for the whole app, and re-use the data
  • Do not hard fail if themes.csv is missing, simply display the IDs
  • Light management: resync, download

UI

• Admin
  • Initialize the database from the web interface
  • Reset the database
  • Delete the database
  • Download the database
  • Import the database
  • Display the configuration variables
  • Many things
• Accordions
  • Added a flag to make the accordion items independent
• Branding:
  • Add a brick as a logo (CC0 image from: https://iconduck.com/icons/71631/brick)
• Global
  • Redesign of the whole app
  • Sticky menu bar on top of the page
  • Execution time and SQL stats for fun
• Libraries
  • Switch from Bulma to Bootstrap, arbitrarily :D
  • Use of baguettebox for images (https://github.com/feimosi/baguetteBox.js)
  • Use of tinysort to sort and filter the grid (https://github.com/Sjeiti/TinySort)
  • Use of sortable for set card tables (https://github.com/tofsjonas/sortable)
  • Use of simple-datatables for big tables (https://github.com/fiduswriter/simple-datatables)
• Minifigures
  • Added a detail view for a minifigure
  • Display which sets are using a minifigure
  • Display which sets are missing a minifigure
• Parts
  • Added a detail view for a part
  • Display which sets are using a part
  • Display which sets are missing a part
• Templates
  • Use a common base template
  • Use HTML fragments/macros for repeted or parametrics items
  • a 404 page for wrong URLs
  • an error page for expected error messages
  • an exception page for unexpected error messages
• Set add
  • Two-tiered (with override) import where you see what you will import before importing it
  • Add a visual indicator that the socket is connected
• Set card
  • Badges to display info like theme, year, parts, etc
  • Set image on top of the card, filling the space
  • Trick to have a blurry background image fill the void in the card
  • Save missing parts on input change rather than by clicking
    • Visual feedback of success
  • Parts and minifigure in accordions
  • Instructions file list
• Set grid
  • 4-2-1 card distribution depending on screen size
  • Display the index with no set added, rather than redirecting
  • Keep last sort in a cookie, and trigger it on page load (can be cleared)