Bulk additions stop midway #131

New Issue

2026-01-09T14:00:53+01:00

iAmRenzo commented

2026-01-09 14:00:53 +01:00

I've tried 3 times to add my Lego collection in bulk. First starting off with all sets, but that processes stopt halfway. But also smaller batches won't finish.
I keep deleting my database and try again with no luck. 28 seems the most reachable import number.

I've tried 3 times to add my Lego collection in bulk. First starting off with all sets, but that processes stopt halfway. But also smaller batches won't finish. I keep deleting my database and try again with no luck. 28 seems the most reachable import number.

FrederikBaerentsen commented

2026-01-09 15:14:36 +01:00

What version of BrickTracker are you using?
Which sets are you trying?
What error are you getting when it stops?

You shouldn't need to delete your database, as sets should not be added if an error occurs.

1. What version of BrickTracker are you using? 2. Which sets are you trying? 3. What error are you getting when it stops? You shouldn't need to delete your database, as sets should not be added if an error occurs.

FrederikBaerentsen added the Kind/Bug label 2026-01-09 15:14:39 +01:00

iAmRenzo commented

2026-01-10 13:24:38 +01:00

What version of BrickTracker are you using?

Which sets are you trying?

What error are you getting when it stops?

You shouldn't need to delete your database, as sets should not be added if an error occurs.

1.3.1
Can you be more specific? I have a lot of sets... not sure what you are asking.
I do not get an error, the progress bar is just stuck and it doesn't do anything.

> 1. What version of BrickTracker are you using? > 2. Which sets are you trying? > 3. What error are you getting when it stops? > > You shouldn't need to delete your database, as sets should not be added if an error occurs. 1. 1.3.1 2. Can you be more specific? I have a lot of sets... not sure what you are asking. 3. I do not get an error, the progress bar is just stuck and it doesn't do anything.

FrederikBaerentsen commented

2026-01-10 13:46:24 +01:00

Which sets do you try and add, where you get the error? I want to test if it is a specific set that causes it.
Could you enable debug mode and check the docker console for errors?

2. Which sets do you try and add, where you get the error? I want to test if it is a specific set that causes it. 3. Could you enable [debug](https://bricktracker.baerentsen.space/configuration/core.html#debug-mode) mode and check the docker console for errors?

iAmRenzo commented

2026-01-18 11:42:52 +01:00

Which sets do you try and add, where you get the error? I want to test if it is a specific set that causes it.

Could you enable debug mode and check the docker console for errors?

It took a while, sorry for that. I've deleted my database again and this is what I did.

897-1, 918-1, 3177-1, 4507-1, 4644-1, 4886-1, 5611-1, 5612-1, 6809-1, 6823-1, 6825-1, 6828-1, 6830-1, 6832-1, 6874-1, 6886-1, 6926-1, 7235-1, 7324-1, 7567-1, 7634-1, 7641-1, 7901-1, 7945-1, 8398-1, 10185-1, 10197-1, 10211-1, 10218-1, 10220-1, 10224-1, 10232-1, 10242-1, 10243-1, 10246-1, 10251-1, 10252-1, 10255-1, 10258-1, 10260-1, 10261-1, 10264-1, 10266-1, 10270-1, 10271-1, 10273-1, 10278-1, 10280-1, 10281-1, 10289-1, 10294-1, 10297-1, 10303-1, 10307-1, 10311-1, 10312-1, 10315-1, 10318-1, 10323-1, 10326-1, 10350-1, 10359-1, 21026-1, 21027-1, 21028-1, 21032-1, 21033-1, 21034-1, 21039-1, 21043-1, 21044-1, 21047-1, 21051-1, 21052-1, 21057-1, 21060-1, 21061-1, 21301-1, 21303-2, 21317-1, 21327-1, 21332-1, 21333-1, 21335-1, 21340-1, 21344-1, 21349-1, 30003-1, 30385-1, 31088-1, 31131-1, 31141-1, 31208-1, 40148-1, 40186-1, 40207-1, 40234-1, 40235-1, 40335-1, 40355-1, 40417-1, 40460-1, 40491-1, 40524-1, 40575-1, 40611-1, 40646-1, 40707-1, 40787-1, 41757-1, 43217-1, 60000-1, 60333-1, 71360-1, 71374-1, 71386-4, 71394-1, 71394-3, 71394-7, 71394-8, 71395-1, 71402-1, 71402-3, 71402-4, 71402-6, 71402-7, 71402-8, 71402-10, 71404-1, 71411-1, 71426-1, 71428-1, 71438-1, 72036-1, 72037-1, 76218-1, 76956-1, 80107-1, 80113-1, 910008-1, 910009-1, 910023-1, 910028-1, 910034-1
It's now stuck here:

Forgot debugging. But I see below the image of the set: Socket is disconnected. Apperently the socket disconnects after some time!? How to keep this open? I am not using nginx or something.

Also "- BK_DEBUG=false" is debugging on or off?

> 2. Which sets do you try and add, where you get the error? I want to test if it is a specific set that causes it. > 3. Could you enable [debug](https://bricktracker.baerentsen.space/configuration/core.html#debug-mode) mode and check the docker console for errors? It took a while, sorry for that. I've deleted my database again and this is what I did. 2. 897-1, 918-1, 3177-1, 4507-1, 4644-1, 4886-1, 5611-1, 5612-1, 6809-1, 6823-1, 6825-1, 6828-1, 6830-1, 6832-1, 6874-1, 6886-1, 6926-1, 7235-1, 7324-1, 7567-1, 7634-1, 7641-1, 7901-1, 7945-1, 8398-1, 10185-1, 10197-1, 10211-1, 10218-1, 10220-1, 10224-1, 10232-1, 10242-1, 10243-1, 10246-1, 10251-1, 10252-1, 10255-1, 10258-1, 10260-1, 10261-1, 10264-1, 10266-1, 10270-1, 10271-1, 10273-1, 10278-1, 10280-1, 10281-1, 10289-1, 10294-1, 10297-1, 10303-1, 10307-1, 10311-1, 10312-1, 10315-1, 10318-1, 10323-1, 10326-1, 10350-1, 10359-1, 21026-1, 21027-1, 21028-1, 21032-1, 21033-1, 21034-1, 21039-1, 21043-1, 21044-1, 21047-1, 21051-1, 21052-1, 21057-1, 21060-1, 21061-1, 21301-1, 21303-2, 21317-1, 21327-1, 21332-1, 21333-1, 21335-1, 21340-1, 21344-1, 21349-1, 30003-1, 30385-1, 31088-1, 31131-1, 31141-1, 31208-1, 40148-1, 40186-1, 40207-1, 40234-1, 40235-1, 40335-1, 40355-1, 40417-1, 40460-1, 40491-1, 40524-1, 40575-1, 40611-1, 40646-1, 40707-1, 40787-1, 41757-1, 43217-1, 60000-1, 60333-1, 71360-1, 71374-1, 71386-4, 71394-1, 71394-3, 71394-7, 71394-8, 71395-1, 71402-1, 71402-3, 71402-4, 71402-6, 71402-7, 71402-8, 71402-10, 71404-1, 71411-1, 71426-1, 71428-1, 71438-1, 72036-1, 72037-1, 76218-1, 76956-1, 80107-1, 80113-1, 910008-1, 910009-1, 910023-1, 910028-1, 910034-1 1. It's now stuck here: <img width="1276" alt="Schermafbeelding 2026-01-18 om 11.43.16.png" src="attachments/00c61f2b-95c9-4d8d-8090-b6932dceefe7"> Forgot debugging. But I see below the image of the set: Socket is disconnected. Apperently the socket disconnects after some time!? How to keep this open? I am not using nginx or something. Also "- BK_DEBUG=false" is debugging on or off?

Schermafbeelding 2026-01-18 om 11.43.16.png

463 KiB

FrederikBaerentsen commented

2026-01-18 16:45:39 +01:00

The cause is the WebSocket connection times out during long bulk imports. When processing 140+ sets, each set takes time to download from Rebrickable and insert into the database. If this takes too long, the socket disconnects and all remaining sets in your list are lost.

The Solution: Use Smaller Batches

You need to break up your list into smaller chunks:

Add 10-20 sets at a time instead of 140+
Wait for each batch to complete before starting the next
This keeps the import duration under the timeout threshold

For Debugging (Optional)

About BK_DEBUG:

BK_DEBUG=false = debugging is OFF
BK_DEBUG=true = debugging is ON

To see detailed logs, change to BK_DEBUG=true in your docker-compose.yml, restart with docker compose restart, then view logs with docker compose logs -f.

You mentioned deleting your database between attempts. You shouldn't need to do this! If you're having to delete the database to recover, that's a separate issue.

I'm closing this issue as the solution is to use smaller batches (10-20 sets at a time) instead of bulk-importing 140+ sets. If you continue experiencing problems even with smaller batches, please reopen the issue and let me know what you're seeing.

The cause is the WebSocket connection times out during long bulk imports. When processing 140+ sets, each set takes time to download from Rebrickable and insert into the database. If this takes too long, the socket disconnects and all remaining sets in your list are lost. ## The Solution: Use Smaller Batches You need to break up your list into smaller chunks: - Add 10-20 sets at a time instead of 140+ - Wait for each batch to complete before starting the next - This keeps the import duration under the timeout threshold ## For Debugging (Optional) About `BK_DEBUG`: - `BK_DEBUG=false` = debugging is OFF - `BK_DEBUG=true` = debugging is ON To see detailed logs, change to `BK_DEBUG=true` in your `docker-compose.yml`, restart with `docker compose restart`, then view logs with `docker compose logs -f`. You mentioned deleting your database between attempts. You shouldn't need to do this! If you're having to delete the database to recover, that's a separate issue. I'm closing this issue as the solution is to use smaller batches (10-20 sets at a time) instead of bulk-importing 140+ sets. If you continue experiencing problems even with smaller batches, please reopen the issue and let me know what you're seeing.

FrederikBaerentsen closed this issue

2026-01-18 16:45:39 +01:00

iAmRenzo commented

2026-01-18 16:48:37 +01:00

Thanks. Brick tracker is not really for big collections then I guess. I’m sticking with Rebrickable.
Not really an option to do smaller batches. There is also no duplicate detection.
Would be easier for the server to keep the connection awake imo

Thanks. Brick tracker is not really for big collections then I guess. I’m sticking with Rebrickable. Not really an option to do smaller batches. There is also no duplicate detection. Would be easier for the server to keep the connection awake imo

FrederikBaerentsen commented

2026-01-18 16:52:43 +01:00

You can of course use whatever you prefer, but BrickTracker is made for big collections. I have over 500 sets in it myself and it is much faster than Rebrickable.

As you can have duplicate sets, there isn't duplicate detection, that wouldn't make sense.

The best way is to copy your whole list of sets into a notepad, then copy 20 at a time to import to BrickTracker.

You will most likely run into API restrictions if you import 100+ sets at once. Rebrickable does not allow unlimited API usage.

You can of course use whatever you prefer, but BrickTracker is made for big collections. I have over 500 sets in it myself and it is much faster than Rebrickable. As you can have duplicate sets, there isn't duplicate detection, that wouldn't make sense. The best way is to copy your whole list of sets into a notepad, then copy 20 at a time to import to BrickTracker. You will most likely run into API restrictions if you import 100+ sets at once. Rebrickable does not allow unlimited API usage.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: FrederikBaerentsen/BrickTracker#131