Previously, in my Self-hosted finances series, I cleaned and identified transfers in my Mint transactions for the purposes of of importing into Firefly-iii. In this post, I’m going to import the transactions into Firefly-iii.

This part is comparatively easy vs the previous steps, however it’s only a one time import. A continue updating workflow is tricky and I’m working on some logic to do that.

Authenticating

First, we need a token that allows us to call the API. To get this, login to Firefly, and open Options, then Profile.

Then go to OAuth and click Create a new token

A screenshot of FIrefly showing the options page and the OAuth client page. Firefly will then show you a token. Copy and store this in the script as a variable and also specify the endpoint where Firefly is being served:

1
2
fireflyToken = 'eYJ...'
host = 'https://firefly.example.com'

Constructing the request

This next method will translate the DataFrame created in the previous post request into the format that the Firefly API expects and creates the transaction.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
import json
import requests

session = requests.Session()

def send_to_firefly(record):
    item = {
        'type': record['type'],
        'amount': record['amount']
    }
    for key in ['description',  'destination_name', 'source_name', 'category_name', 'notes']:
        if key in record and not pd.isnull(record[key]):
            item[key] = record[key]
    for key in ['source_id', 'destination_id']:
        if key in record and not pd.isnull(record[key]) and record[key] != '':
            item[key] = str(int(record[key]))
    item['foreign_amount'] = '0'
    item['reconciled'] = False
    item['order'] = '0'
    # If you use something other than USD, change this value
    item['currency_id'] = '8'
    item['currency_code'] = 'USD'
    for key in ['date', 'process_date']:
        if key in record and not pd.isnull(record[key]) and record[key] != '':
            item[key] = record[key].isoformat()
    if isinstance(record['tags'], str):
        item['tags'] = [record['tags']]
    else:
        item['tags'] = []
    payload = {
            "transactions": [
                item
            ],
            "apply_rules": True,
            "fire_webhooks": False,
            "error_if_duplicate_hash": False
        }
    try:
        json.dumps(payload)
    except Exception as e:
        raise Exception(f"Can't serialize '{item}: {e}")
    resp = session.post(f"{host}/api/v1/transactions", headers={"Authorization": f"Bearer {fireflyToken}", "Content-Type": "application/json", 'Accept': 'application/json'}, json=payload, allow_redirects=False)
    if resp.status_code == 200:
        return pd.Series([resp.status_code, resp.json(), resp.json()['data']['id']], index=['status', 'message', 'firefly_id'])
    else:
        return pd.Series([resp.status_code, resp.json()], index=['status', 'message'])

Sending the request

Next, it’s easy to send the requests to Firefly:

1
2
3
4
5
# First find the transfers and translate into common format
output = transactions.apply(func=process_record, axis=1, result_type='expand')

# Then send it to Firefly
fireflyResult = output[output['amount'] != 0].apply(func=send_to_firefly, axis=1, result_type='expand')

Reviewing Results

Firefly can return non-200 status code responses. Running this will identify all the non-successful writes. In theory, this should empty. It’s a good idea to check the error messages and see what’s wrong.

1
print(fireflyResult[(fireflyResult['status'] != 200)])

Conclusion

Once we’ve identified the transfers, it’s comparatively easy to write the one-time extract to Firefly. The code can be found in my GitHub repository. I’ve got a few projects in progress right now, including tools to automatically scrape data from accounts and how to merge new transactions into Firefly.

Comments

Comments are currently unavailable while I move to this new blog platform. To give feedback, send an email to adam [at] this website url.

Publishing to Firefly-iii using Pandas

This article is part of the Self-hosted Finances series.