CSV/XML export #52

New Issue

FrederikBaerentsen · 2025-01-25T21:26:30+01:00

2025-01-25 21:26:30 +01:00

I started working on the CSV export functionality from #25.

Rebrickable allows importing a CSV with just the fields element_id,quantity as a minimal example. This can be achieved with the following implementation (just showing the export file, not all the added html and routes):

`/views/admin/export.py` (New File)

# Export missing parts as CSV
@admin_export_page.route('/missing_parts', methods=['GET'])
@login_required
@exception_handler(__file__)
def missing_parts() -> Response:
    BrickPartList().export_parts_csv('total_missing')
    return redirect(url_for('admin.admin', open_export=True))


# Export all parts as CSV
@admin_export_page.route('/all_parts', methods=['GET'])
@login_required
@exception_handler(__file__)
def all_parts() -> Response:
    BrickPartList().export_parts_csv('quantity')
    return redirect(url_for('admin.admin', open_export=True))

`part_list.py`

def export_parts_csv(self, qty: str, /) -> str:
    # Export CSV of the selected parts
    csv = 'Element,Quantity\n'
    for part in getattr(BrickPartList(), 'missing' if qty == 'total_missing' else 'all')():
        csv += '{element_id},{value}\n'.format(
            element_id=part.fields.element_id,
            value=getattr(part.fields, qty)
        )

While the CSV export is useful, my initial goal was to export missing parts in Bricklink XML format. This would make it easier to import the data into a Bricklink wanted list and proceed with purchasing the parts.

`part_list.py`

def export_bricklink_xml(self, /) -> str:
    xml = '<?xml version="1.0" encoding="UTF-8"?>\n'
    xml += '<INVENTORY>\n'
    for part in self.missing():
        xml += '  <ITEM>\n'
        xml += '    <ITEMTYPE>P</ITEMTYPE>\n'
        xml += '    <ITEMID>{}</ITEMID>\n'.format(part.fields.part_num)
        xml += '    <COLOR>{}</COLOR>\n'.format(part.fields.color_id)
        xml += '    <QTY>{}</QTY>\n'.format(part.fields.total_missing)
        xml += '  </ITEM>\n'
    xml += '</INVENTORY>\n'
    return xml

Challenges:

One issue encountered during the Bricklink XML export is with parts like 3622pr0056. While this is the part number on Rebrickable, the corresponding part number on Bricklink is 3622pb122. This discrepancy arises from differences in part numbering between the platforms (see Part Numbering). Rebrickable’s API does provide Bricklink part numbers via the external_ids field:

"external_ids": {
    "BrickLink": [
        "3622pb122"
    ]
}

However, utilizing this data would require adding an extra field in the database to store Bricklink part numbers.

Next Steps:

The CSV export feature can still be useful, however I’m putting this task on the back burner for now and not committing any of the code. The examples here should be enough to implement it if needed.
It would be ideal to provide flexibility by allowing users to define which fields to export. This could be implemented via:
1. Environment variables: Let users define the fields to be included in the CSV.
2. Admin panel configuration: Provide a list of available fields that can be selected during the export process.

I started working on the CSV export functionality from #25. Rebrickable allows importing a CSV with just the fields `element_id,quantity` as a minimal example. This can be achieved with the following implementation (just showing the export file, not all the added html and routes): ![](https://xbackbone.baerentsen.space/LaMU8/fabAHEwA04.png/raw) ### `/views/admin/export.py` (New File) ```python # Export missing parts as CSV @admin_export_page.route('/missing_parts', methods=['GET']) @login_required @exception_handler(__file__) def missing_parts() -> Response: BrickPartList().export_parts_csv('total_missing') return redirect(url_for('admin.admin', open_export=True)) # Export all parts as CSV @admin_export_page.route('/all_parts', methods=['GET']) @login_required @exception_handler(__file__) def all_parts() -> Response: BrickPartList().export_parts_csv('quantity') return redirect(url_for('admin.admin', open_export=True)) ``` ### `part_list.py` ```python def export_parts_csv(self, qty: str, /) -> str: # Export CSV of the selected parts csv = 'Element,Quantity\n' for part in getattr(BrickPartList(), 'missing' if qty == 'total_missing' else 'all')(): csv += '{element_id},{value}\n'.format( element_id=part.fields.element_id, value=getattr(part.fields, qty) ) ``` While the CSV export is useful, my initial goal was to export missing parts in Bricklink XML format. This would make it easier to import the data into a Bricklink wanted list and proceed with purchasing the parts. ### `part_list.py` ```python def export_bricklink_xml(self, /) -> str: xml = '<?xml version="1.0" encoding="UTF-8"?>\n' xml += '<INVENTORY>\n' for part in self.missing(): xml += ' <ITEM>\n' xml += ' <ITEMTYPE>P</ITEMTYPE>\n' xml += ' <ITEMID>{}</ITEMID>\n'.format(part.fields.part_num) xml += ' <COLOR>{}</COLOR>\n'.format(part.fields.color_id) xml += ' <QTY>{}</QTY>\n'.format(part.fields.total_missing) xml += ' </ITEM>\n' xml += '</INVENTORY>\n' return xml ``` ### Challenges: One issue encountered during the Bricklink XML export is with parts like `3622pr0056`. While this is the part number on Rebrickable, the corresponding part number on Bricklink is `3622pb122`. This discrepancy arises from differences in part numbering between the platforms (see [Part Numbering](https://rebrickable.com/help/part-numbering/)). Rebrickable’s API does provide Bricklink part numbers via the `external_ids` field: ```json "external_ids": { "BrickLink": [ "3622pb122" ] } ``` However, utilizing this data would require adding an extra field in the database to store Bricklink part numbers. ### Next Steps: - The CSV export feature can still be useful, however I’m putting this task on the back burner for now and not committing any of the code. The examples here should be enough to implement it if needed. - It would be ideal to provide flexibility by allowing users to define which fields to export. This could be implemented via: 1. **Environment variables**: Let users define the fields to be included in the CSV. 2. **Admin panel configuration**: Provide a list of available fields that can be selected during the export process.

FrederikBaerentsen added the

Kind/Feature

label 2025-01-25 21:26:30 +01:00

FrederikBaerentsen added this to the General Improvements project 2025-01-25 21:26:30 +01:00

FrederikBaerentsen referenced this issue

2025-01-25 21:26:50 +01:00

Improvements #25

FrederikBaerentsen referenced this issue

2025-01-26 16:32:41 +01:00

BrickLink Links Fail for Certain Parts #55

Sign in to join this conversation.