Migrating from Google Location History to OwnTracks

I’ve been slowly reducing the amount of data shared with Google. I’ve been using Google Location History since 2013. I found it really useful just because I could figure out what restaurant I went to when I was traveling or any number of things.

I found OwnTracks which was an open-source location history storage solution. It’s not nearly as polished as Google Maps where it natively integrates your location history, but step one is owning my data, step 2 can be better UIs.

In this post, I’m going to walk through exactly how to get data from Google Location History and import it into OwnTracks

Setup OwnTracks

Feel free to follow the guides here to setup the OwnTracks recorder and UI.

Exporting from Google

  1. https://takeout.google.com/
  2. Deselect all, select Google location History
  3. Export once, file size: 4GB
  4. Wait for it to export, then download and extract the .zip file

Importing into OwnTracks

I first tried to follow this guide which uses this importer in the OwnTracks/recorder repository. My exported JSON file was 2GB, and it took hours to import and the MQTT importer seemed to have dropped a few records here and there.

Instead, let’s try to import it directly and bypass MQTT. I opened up OwnTrack’s /store folder and checked out the file format:

1
2
3
4
ls /[...]/rec/user/phone# head 2023-01.rec
2023-01-01T00:00:52Z    *                       {"_type":"location","tid":"fl","tst":1672531252,"lat":47.12345,"lon":-122.12345,"acc":13,"alt":123,"vac":4}
2023-01-01T00:01:52Z    *                       {"_type":"location","tid":"fl","tst":1672531312,"lat":47.12345,"lon":-122.12345,"acc":13,"alt":123,"vac":4}
[...]

The file was plain-text and appeared easy to generate. I created a Python function that loaded data from /data/Records.json then generated the text files to /data/location/yyyy-mm-.rec.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
import pandas as pd
import json
import time

df_gps = pd.read_json('/data/Records.json', typ='frame', orient='records')
print('There are {:,} rows in the location history dataset'.format(len(df_gps)))

df_gps = df_gps.apply(lambda x: x['locations'], axis=1, result_type='expand')
df_gps['latitudeE7'] = df_gps['latitudeE7'] / 10.**7
df_gps['longitudeE7'] = df_gps['longitudeE7'] / 10.**7
df_gps['timestamp'] = pd.to_datetime(df_gps['timestamp'])

owntracks = gps.rename(columns={'latitudeE7': 'lat', 'longitudeE7': 'lon', 'accuracy': 'acc', 'altitude': 'alt', 'verticalAccuracy': 'vac'})
owntracks['tst'] = (owntracks['timestamp'].astype(int) / 10**9)

files = {}

for year in range(years['min'], years['max'] + 1):
    for month in range(1, 13):
        files[f"{year}-{month}"] = open(f"/data/location/{year}-{str(month).rjust(2, '0')}.rec", 'w')

try:
    for index, row in owntracks.iterrows():
        d = row.to_dict()
        record = {
            '_type': 'location',
            'tid': 'aj'
        }
        record['tst'] = int(time.mktime(d['timestamp'].timetuple()))

        for key in ['lat', 'lon']:
            if key in row and not pd.isnull(row[key]):
                record[key] = row[key]
        for key in ['acc', 'alt', 'vac']:
            if key in row and not pd.isnull(row[key]):
                record[key] = int(row[key])
        
        timestamp = row['timestamp'].strftime("%Y-%m-%dT%H:%M:%SZ")
        line = f"{timestamp}\t*                 \t{json.dumps(record, separators=(',', ':'))}\n"
        files[f"{d['timestamp'].year}-{d['timestamp'].month}"].write(line)
finally:
    for key, file in files.items():
        file.flush()
        file.close()

After it generates the data, find your OwnTracks data directory. The copy the output into /{owntracks_store_dir}/store/{user}/{device} folder. If you replace any existing data you will lose data recorded from Opentracks itself, so I would recommend downloading the Google Location History data first, converting, importing, then start collecting data afterwards.

Removing extra devices

When I loaded this, the history was crazy showing me moving in and out of a city instantly: A screenshot showing my location appearing to be in one city, then out, then back in at a physically impossible speed.

Turns out, multiple devices on my accounts were reporting location so it appeared that I kept moving between two cities. Easy enough to exclude the device.

The following script produces a chart that shows when devices are reporting their location:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
gps.sort_values(by='timestamp', inplace=True)
devices = gps['deviceTag'].unique()

plt.figure(figsize=(10, 6))
colors = plt.cm.viridis_r([i / len(devices) for i in range(len(devices))])


for i, device in enumerate(devices):
    device_data = gps[gps['deviceTag'] == device]
    first_year = device_data['timestamp'].dt.year.min()
    last_year = device_data['timestamp'].dt.year.max()
    middle_year = (first_year + last_year) / 2
    
    plt.hlines(y=i, xmin=first_year, xmax=last_year, color=colors[i], linewidth=4)
    plt.text(middle_year, i + 0.25, i, verticalalignment='center', fontsize=8)

plt.xlabel('Year')
plt.ylabel('Device')
plt.title('Device Reporting Years')
plt.grid(True)

plt.show()

With this I can identify which device is reporting in parallel. In my case, it’s the 10th device that’s reporting in parallel with my phone. Note that I’ve replaced the device ids, yours will be a long random number:

Chart showing the years that a device is reporting data. Most devices are only reporting for a brief amount of time, but one device is reporting 2018-2024 along side other devices. This is my old tablet.

From there, I filtered out this device from the location history

1
2
3
4
5
+excluded_devices = [1234510]

 owntracks = gps.rename(columns={'latitudeE7': 'lat', 'longitudeE7': 'lon', 'accuracy': 'acc', 'altitude': 'alt', 'verticalAccuracy': 'vac'})
 owntracks['tst'] = (owntracks['timestamp'].astype(int) / 10**9)
+owntracks = owntracks[~owntracks['deviceTag'].isin(excluded_devices)]

And the history looks a lot better:

Screenshot of my location history showing me staying within a single city as expected.

Adding new data

Now, once you have your historical data, what do you do for new data moving forward? There’s a lot of options depending on personal preference including:

  • Using the OwnTracks mobile apps
  • Copying location from Home Assistant like here. This is the method that I use.

Conclusion

OwnTracks is an open-source place to store your location history. You can import existing history from Google Maps, then report new data into OwnTracks and have a localized

Copyright - All Rights Reserved
Last updated on Jan 27, 2024 00:00 UTC

Comments

Comments are currently unavailable while I move to this new blog platform. To give feedback, send an email to adam [at] this website url.