New Year revamp #40

gregoo · 2025-01-17T18:36:22+01:00

gregoo commented

2025-01-17 18:36:22 +01:00

Changes

2025-01-02: New Year revamp

Documentation

Brand new documentation

Code

Authentication
- Basic authentication mechanism with ONE password to protect admin and writes
CSV
- Remove dependencies to numpy and panda for simpler built-in csv
Code
- Refactored the Python code
- Modularity (more functions, splitting files)
- Type hinting whenever possible
- Flake8 linter
- Retained most of the original behaviour (with its quirks)
Colors
- Remove dependency on color.csv
Configuration
- Moved all the hard-coded parameters into configuration variables
- Most of the variables are configuration through environment variables
- Force instruction, sets, etc path to be relative to static
Docker
- Added an entrypoint to grab PORT / HOST from the environment if set
- Remove the need to seed the container with files (*.csv, nil files)
Flask
- Fix improper socketio.run(app.run()) call which lead to hard crash on ^C
- Make use of url_for to create URLs
- Use blueprints to implement routes
- Move views into their own files
- Split GET and POST methods into two different routes for clarity
Images
- Add an option to use remote images from the Rebrickable CDN rather than downloading everything locally
- Handle nil.png and nil_mf.jpg as true images in /static/sets/ so that they are downloaded whenever necessary when importing a se with missing images
Instructions
- Scan the files once for the whole app, and re-use the data
- Refresh the instructions from the admin
- More lenient set number detection
- Update when uploading a new one
- Basic file management
Logs
- Added log lines for change actions (add, check, missing, delete) so that the server is not silent when DEBUG=false
Minifigures
- Added a variable to control default ordering
Part(s)
- Added a variable to control default ordering of listing
Retired sets
- Open the themes once for the whole app, and re-use the data
- Do not hard fail if themes.csv is missing, simply display the IDs
- Light management: resync, download
Set(s)
- Reworked the set checkboxes with a dedicated route per status
- Switch from homemade ID generator to proven UUID4 for sets ID
  - Does not interfere with previously created sets
- Do not rely on sets.csv to check if the set exists
- When adding, commit the set to database only once everything has been processed
- Added a bulk add page
- Keep spare parts when importing
- Added a variable to control default ordering of listing
Socket
- Make use of socket.io rooms to avoid broadcasting messages to all clients
SQLite
- Do not hard fail if the database is not present or not initialized
- Open the database once for the context, and re-use the connection
- Move queries to .sql files and load them as Jinja templates
- Use named arguments rather than sets for SQLite queries
- Allow execute() to be deferred to the commit() call to avoid locking the database for long period while importing (locked while downloading images)
Themes
- Open the themes once for the whole app, and re-use the data
- Do not hard fail if themes.csv is missing, simply display the IDs
- Light management: resync, download

UI

Admin
- Initialize the database from the web interface
- Reset the database
- Delete the database
- Download the database
- Import the database
- Display the configuration variables
- Many things
Accordions
- Added a flag to make the accordion items independent
Branding:
- Add a brick as a logo (CC0 image from: https://iconduck.com/icons/71631/brick)
Global
- Redesign of the whole app
- Sticky menu bar on top of the page
- Execution time and SQL stats for fun
Libraries
- Switch from Bulma to Bootstrap, arbitrarily :D
- Use of baguettebox for images (https://github.com/feimosi/baguetteBox.js)
- Use of tinysort to sort and filter the grid (https://github.com/Sjeiti/TinySort)
- Use of sortable for set card tables (https://github.com/tofsjonas/sortable)
- Use of simple-datatables for big tables (https://github.com/fiduswriter/simple-datatables)
Minifigures
- Added a detail view for a minifigure
- Display which sets are using a minifigure
- Display which sets are missing a minifigure
Parts
- Added a detail view for a part
- Display which sets are using a part
- Display which sets are missing a part
Templates
- Use a common base template
- Use HTML fragments/macros for repeted or parametrics items
- a 404 page for wrong URLs
- an error page for expected error messages
- an exception page for unexpected error messages
Set add
- Two-tiered (with override) import where you see what you will import before importing it
- Add a visual indicator that the socket is connected
Set card
- Badges to display info like theme, year, parts, etc
- Set image on top of the card, filling the space
- Trick to have a blurry background image fill the void in the card
- Save missing parts on input change rather than by clicking
  - Visual feedback of success
- Parts and minifigure in accordions
- Instructions file list
Set grid
- 4-2-1 card distribution depending on screen size
- Display the index with no set added, rather than redirecting
- Keep last sort in a cookie, and trigger it on page load (can be cleared)

# Changes ## 2025-01-02: New Year revamp ### Documentation - Brand new documentation ### Code - Authentication - Basic authentication mechanism with ONE password to protect admin and writes - CSV - Remove dependencies to numpy and panda for simpler built-in csv - Code - Refactored the Python code - Modularity (more functions, splitting files) - Type hinting whenever possible - Flake8 linter - Retained most of the original behaviour (with its quirks) - Colors - Remove dependency on color.csv - Configuration - Moved all the hard-coded parameters into configuration variables - Most of the variables are configuration through environment variables - Force instruction, sets, etc path to be relative to static - Docker - Added an entrypoint to grab PORT / HOST from the environment if set - Remove the need to seed the container with files (*.csv, nil files) - Flask - Fix improper socketio.run(app.run()) call which lead to hard crash on ^C - Make use of url_for to create URLs - Use blueprints to implement routes - Move views into their own files - Split GET and POST methods into two different routes for clarity - Images - Add an option to use remote images from the Rebrickable CDN rather than downloading everything locally - Handle nil.png and nil_mf.jpg as true images in /static/sets/ so that they are downloaded whenever necessary when importing a se with missing images - Instructions - Scan the files once for the whole app, and re-use the data - Refresh the instructions from the admin - More lenient set number detection - Update when uploading a new one - Basic file management - Logs - Added log lines for change actions (add, check, missing, delete) so that the server is not silent when DEBUG=false - Minifigures - Added a variable to control default ordering - Part(s) - Added a variable to control default ordering of listing - Retired sets - Open the themes once for the whole app, and re-use the data - Do not hard fail if themes.csv is missing, simply display the IDs - Light management: resync, download - Set(s) - Reworked the set checkboxes with a dedicated route per status - Switch from homemade ID generator to proven UUID4 for sets ID - Does not interfere with previously created sets - Do not rely on sets.csv to check if the set exists - When adding, commit the set to database only once everything has been processed - Added a bulk add page - Keep spare parts when importing - Added a variable to control default ordering of listing - Socket - Make use of socket.io rooms to avoid broadcasting messages to all clients - SQLite - Do not hard fail if the database is not present or not initialized - Open the database once for the context, and re-use the connection - Move queries to .sql files and load them as Jinja templates - Use named arguments rather than sets for SQLite queries - Allow execute() to be deferred to the commit() call to avoid locking the database for long period while importing (locked while downloading images) - Themes - Open the themes once for the whole app, and re-use the data - Do not hard fail if themes.csv is missing, simply display the IDs - Light management: resync, download ### UI - Admin - Initialize the database from the web interface - Reset the database - Delete the database - Download the database - Import the database - Display the configuration variables - Many things - Accordions - Added a flag to make the accordion items independent - Branding: - Add a brick as a logo (CC0 image from: https://iconduck.com/icons/71631/brick) - Global - Redesign of the whole app - Sticky menu bar on top of the page - Execution time and SQL stats for fun - Libraries - Switch from Bulma to Bootstrap, arbitrarily :D - Use of baguettebox for images (https://github.com/feimosi/baguetteBox.js) - Use of tinysort to sort and filter the grid (https://github.com/Sjeiti/TinySort) - Use of sortable for set card tables (https://github.com/tofsjonas/sortable) - Use of simple-datatables for big tables (https://github.com/fiduswriter/simple-datatables) - Minifigures - Added a detail view for a minifigure - Display which sets are using a minifigure - Display which sets are missing a minifigure - Parts - Added a detail view for a part - Display which sets are using a part - Display which sets are missing a part - Templates - Use a common base template - Use HTML fragments/macros for repeted or parametrics items - a 404 page for wrong URLs - an error page for expected error messages - an exception page for unexpected error messages - Set add - Two-tiered (with override) import where you see what you will import before importing it - Add a visual indicator that the socket is connected - Set card - Badges to display info like theme, year, parts, etc - Set image on top of the card, filling the space - Trick to have a blurry background image fill the void in the card - Save missing parts on input change rather than by clicking - Visual feedback of success - Parts and minifigure in accordions - Instructions file list - Set grid - 4-2-1 card distribution depending on screen size - Display the index with no set added, rather than redirecting - Keep last sort in a cookie, and trigger it on page load (can be cleared)

gregoo added 14 commits 2025-01-17 18:36:22 +01:00

Massive rewrite 6477be00b6

Fix app init to work with gunicorn c3f3ed0593

Remove login.is_readonly() 32050acabc

Fix the open_database logic in the admin 50ebc3877c

Fix bricklink and rebrickable links trying to cast lower() on an int e2617daa40

De-verbosify some SQLite debug db9dbfae54

Fix part counts not taking into account the quantity of minifigures 463e00f3fe

Fix part list not using total_quantity 00310e0235

Fix missing 'missing' and 'quantity' badges for minifigure in a part card 7999572cf1

Add the path that 404'd in the log line 53b406f722

Add some spacing between the minifigure image and name in an accordion header e4d052f22c

Built-in fix for 'None' entries in the missing table 0d9276f639

Refactor nil image code, and add a built-in fix for missing nil images 05484f6148

Unbreak minifigure accordion 74094b2ea2

gregoo requested review from FrederikBaerentsen 2025-01-17 18:37:13 +01:00

gregoo added 16 commits 2025-01-19 00:09:49 +01:00

Shave 100MB from the Docker image by not caching and removing some dependencies 4bbf3a3e80

Fix garbage tag in default images ca06f568ef

Remove all quotes from the env documentation as they would be interpreted as litteral quotes by docker 8264ddb68d

Make sure tzdata is included in the dependencies 9f46034ff5

Display the number of instructions files if there are any 12835f16f0

Do not display upload instructions link if in read only mode 703d16f331

Give info about the environment variable name to use when a variable is missing 7b682e5859

Do nothing if the socket is not connected 5fa51c21a9

Precise the static path absolute path 68c18caaa1

Display the database path in the database section c403029a11

Do not show the login page if already logged in 6b62a1f89b

Do not use current_user to check if authenticated, go through the LoginManager class d4bce7491a

Unbreak the sort part of the missing part input 61a1b0ad94

Remove total and unique counters from accordion headers because they are wrong and misleading 57158f75c8

Add missing order options for most of the new queries 0420af5f95

Add new documentation fully contained within the repo f1ed402d21

gregoo changed title from ~~WIP: New Year revamp~~ to New Year revamp

2025-01-19 00:10:51 +01:00

gregoo referenced this pull request

2025-01-19 00:14:42 +01:00

Clean up code #8

gregoo added 1 commit 2025-01-19 00:26:37 +01:00

Don't add the documentation to the docker image 517be96957

FrederikBaerentsen commented

2025-01-19 08:18:45 +01:00

Looks great. Just tested it out and as far as i could see, this is the steps needed to upgrade from the previous version:

Update install (git pull)
Move database to folder (mkdir data && mv app.db data/)
Change .env file to new layout (either download the new sample or copy/paste)
Change compose.yaml file to new layout i used:

services:
  bricktracker:
    container_name: BrickTracker
    restart: unless-stopped
    build: .
    ports:
      - "3333:3333"
    volumes:
      - ./data:/data/
      - ./static/instructions:/app/static/instructions/
      - ./static/minifigs:/app/static/minifigures/
      - ./static/parts:/app/static/parts/
      - ./static/sets:/app/static/sets/
    environment:
      BK_DATABASE_PATH: /data/app.db
      BK_MINIFIGURES_FOLDER: minifigures
      BK_RETIRED_SETS_PATH: /data/retired_sets.csv
      BK_THEMES_PATH: /data/themes.csv
    env_file: ".env"

Pretty simple upgrade way and it worked nicely. I'll merge and before updating the docker image, i'll write a release post with upgrade instructions.

Looks great. Just tested it out and as far as i could see, this is the steps needed to upgrade from the previous version: 1. Update install (`git pull`) 2. Move database to folder (`mkdir data && mv app.db data/`) 3. Change .env file to new layout (either download the new sample or copy/paste) 4. Change compose.yaml file to new layout i used: ```` services: bricktracker: container_name: BrickTracker restart: unless-stopped build: . ports: - "3333:3333" volumes: - ./data:/data/ - ./static/instructions:/app/static/instructions/ - ./static/minifigs:/app/static/minifigures/ - ./static/parts:/app/static/parts/ - ./static/sets:/app/static/sets/ environment: BK_DATABASE_PATH: /data/app.db BK_MINIFIGURES_FOLDER: minifigures BK_RETIRED_SETS_PATH: /data/retired_sets.csv BK_THEMES_PATH: /data/themes.csv env_file: ".env" ```` Pretty simple upgrade way and it worked nicely. I'll merge and before updating the docker image, i'll write a release post with upgrade instructions.

FrederikBaerentsen merged commit e2425a106a into master

2025-01-19 08:19:01 +01:00

FrederikBaerentsen referenced this issue from a commit

2025-01-19 08:19:03 +01:00

Merge pull request 'New Year revamp' (#40) from gregoo/BrickTracker:master into master

Sign in to join this conversation.

No reviewers