Publishing to Firefly-iii using Pandas

This article is part of the Self-hosted Finances series.

    Previously, in my Self-hosted finances series, I cleaned and identified transfers in my Mint transactions for the purposes of of importing into Firefly-iii. In this post, I’m going to import the transactions into Firefly-iii.

    This part is comparatively easy vs the previous steps, however it’s only a one time import. A continue updating workflow is tricky and I’m working on some logic to do that.

    Authenticating

    First, we need a token that allows us to call the API. To get this, login to Firefly, and open Options, then Profile.

    Then go to OAuth and click Create a new token

    A screenshot of FIrefly showing the options page and the OAuth client page. Firefly will then show you a token. Copy and store this in the script as a variable and also specify the endpoint where Firefly is being served:

    1
    2
    
    fireflyToken = 'eYJ...'
    host = 'https://firefly.example.com'
    

    Constructing the request

    This next method will translate the DataFrame created in the previous post request into the format that the Firefly API expects and creates the transaction.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    
    import json
    import requests
    
    session = requests.Session()
    
    def send_to_firefly(record):
        item = {
            'type': record['type'],
            'amount': record['amount']
        }
        for key in ['description',  'destination_name', 'source_name', 'category_name', 'notes']:
            if key in record and not pd.isnull(record[key]):
                item[key] = record[key]
        for key in ['source_id', 'destination_id']:
            if key in record and not pd.isnull(record[key]) and record[key] != '':
                item[key] = str(int(record[key]))
        item['foreign_amount'] = '0'
        item['reconciled'] = False
        item['order'] = '0'
        # If you use something other than USD, change this value
        item['currency_id'] = '8'
        item['currency_code'] = 'USD'
        for key in ['date', 'process_date']:
            if key in record and not pd.isnull(record[key]) and record[key] != '':
                item[key] = record[key].isoformat()
        if isinstance(record['tags'], str):
            item['tags'] = [record['tags']]
        else:
            item['tags'] = []
        payload = {
                "transactions": [
                    item
                ],
                "apply_rules": True,
                "fire_webhooks": False,
                "error_if_duplicate_hash": False
            }
        try:
            json.dumps(payload)
        except Exception as e:
            raise Exception(f"Can't serialize '{item}: {e}")
        resp = session.post(f"{host}/api/v1/transactions", headers={"Authorization": f"Bearer {fireflyToken}", "Content-Type": "application/json", 'Accept': 'application/json'}, json=payload, allow_redirects=False)
        if resp.status_code == 200:
            return pd.Series([resp.status_code, resp.json(), resp.json()['data']['id']], index=['status', 'message', 'firefly_id'])
        else:
            return pd.Series([resp.status_code, resp.json()], index=['status', 'message'])
    

    Sending the request

    Next, it’s easy to send the requests to Firefly:

    1
    2
    3
    4
    5
    
    # First find the transfers and translate into common format
    output = transactions.apply(func=process_record, axis=1, result_type='expand')
    
    # Then send it to Firefly
    fireflyResult = output[output['amount'] != 0].apply(func=send_to_firefly, axis=1, result_type='expand')
    

    Reviewing Results

    Firefly can return non-200 status code responses. Running this will identify all the non-successful writes. In theory, this should empty. It’s a good idea to check the error messages and see what’s wrong.

    1
    
    print(fireflyResult[(fireflyResult['status'] != 200)])
    

    Conclusion

    Once we’ve identified the transfers, it’s comparatively easy to write the one-time extract to Firefly. The code can be found in my GitHub repository. I’ve got a few projects in progress right now, including tools to automatically scrape data from accounts and how to merge new transactions into Firefly.

    Copyright - All Rights Reserved

    Comments

    Comments are currently unavailable while I move to this new blog platform. To give feedback, send an email to adam [at] this website url.