I’ve been slowly reducing the amount of data shared with Google. I’ve been using Google Location History since 2013. I found it really useful just because I could figure out what restaurant I went to when I was traveling or any number of things.
I found OwnTracks which was an open-source location history storage solution. It’s not nearly as polished as Google Maps where it natively integrates your location history, but step one is owning my data, step 2 can be better UIs.
In this post, I’m going to walk through exactly how to get data from Google Location History and import it into OwnTracks
Setup OwnTracks
Feel free to follow the guides here to setup the OwnTracks recorder and UI.
Exporting
Google is moving their location history from server side to device side news to be ensure that they don’t have access to your location data, but as of now (August 2024) I’ve not been able to export my data on my phone.
Android
It looks like Google is trying to add support for exporting
- Settings
- Location
- Location Services
- Export timeline date
- Authenticate
- Pick a folder and then save it
Then if you’re lucky, the save will succeed and you can upload it to a computer. Mine, unfortunately, which failed with an error and it created a partially generated file. There appears to be an application bug where Google Maps isn’t able to handle certain timeline entries.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| Shushing crash.
FATAL EXCEPTION: lowpool[2]
Process: com.google.android.gms, PID: 28769
java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.Class java.lang.Object.getClass()' on a null object reference
at erfs.A(:com.google.android.gms@243333039@24.33.33 (190408-666381490):1)
at dgix.f(:com.google.android.gms@243333039@24.33.33 (190408-666381490):672)
at bopo.fx(:com.google.android.gms@243333039@24.33.33 (190408-666381490):1)
at boqa.run(:com.google.android.gms@243333039@24.33.33 (190408-666381490):66)
at epiu.run(:com.google.android.gms@243333039@24.33.33 (190408-666381490):21)
at amrj.c(:com.google.android.gms@243333039@24.33.33 (190408-666381490):50)
at amrj.run(:com.google.android.gms@243333039@24.33.33 (190408-666381490):76)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
at amwv.run(:com.google.android.gms@243333039@24.33.33 (190408-666381490):8)
at java.lang.Thread.run(Thread.java:1012)
Suppressed: epjf:
at tk_trace.314-ExportOperation(Unknown Source:0)
at tk_trace.semanticlocationhistory-SemanticLocationHistoryZeroPartyClientChimeraService-ISemanticLocationHistoryZeroPartyService_7(Unknown Source:0)
|
iOS
- Google Maps
- Your Timeline (three dots, top right corner)
- Location and privacy settings
- Export Timeline Data
Google Takeout
If you haven’t migrated to on-device location tracking, then you can use the Google Takeout method:
- https://takeout.google.com/
- Deselect all, select Google location History
- Export once, file size: 4GB
- Wait for it to export, then download and extract the .zip file
Importing into OwnTracks
I first tried to follow this guide which uses this importer in the OwnTracks/recorder repository. My exported JSON file was 2GB, and it took hours to import and the MQTT importer seemed to have dropped a few records here and there.
Instead, let’s try to import it directly and bypass MQTT. I opened up OwnTrack’s /store folder and checked out the file format:
1
2
3
4
| ls /[...]/rec/user/phone# head 2023-01.rec
2023-01-01T00:00:52Z * {"_type":"location","tid":"fl","tst":1672531252,"lat":47.12345,"lon":-122.12345,"acc":13,"alt":123,"vac":4}
2023-01-01T00:01:52Z * {"_type":"location","tid":"fl","tst":1672531312,"lat":47.12345,"lon":-122.12345,"acc":13,"alt":123,"vac":4}
[...]
|
The file was plain-text and appeared easy to generate. I created a Python function that loaded data from /data/Records.json then generated the text files to /data/location/yyyy-mm-.rec.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| #!/usr/bin/python
import json
import datetime
output = []
for r in json.load(open('/data/Timeline.json'))['semanticSegments']:
if 'timelinePath' in r:
for point in r['timelinePath']:
latlng = point['point'].replace('°', '').split(',')
output.append({
'latitudeE7': float(latlng[0]),
'longitudeE7': float(latlng[1].strip()),
'timestamp': datetime.datetime.fromisoformat(point['time'])
})
df_gps = pd.DataFrame(output)
del output
|
1
2
3
4
5
6
7
8
9
| #!/usr/bin/python
import json
import os
import pandas as pd
import time
# https://owntracks.org/booklet/features/tid/
tracker_id = 'ex' # A two-character identifier
df_gps = pd.read_json('Records.json', typ='frame', orient='records')
|
Processing
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
| print('There are {:,} rows in the location history dataset'.format(len(df_gps)))
output_folder = "output"
df_gps = df_gps.apply(lambda x: x['locations'], axis=1, result_type='expand')
df_gps['latitudeE7'] = df_gps['latitudeE7'] / 10.**7
df_gps['longitudeE7'] = df_gps['longitudeE7'] / 10.**7
df_gps['timestamp'] = pd.to_datetime(df_gps['timestamp'], format='ISO8601', utc=True)
owntracks = df_gps.rename(columns={'latitudeE7': 'lat', 'longitudeE7': 'lon', 'accuracy': 'acc', 'altitude': 'alt', 'verticalAccuracy': 'vac'})
owntracks['tst'] = (owntracks['timestamp'].astype(int) / 10**9)
files = {}
years = df_gps['timestamp'].dt.year.agg(['min', 'max'])
if not os.path.exists(output_folder):
os.makedirs(output_folder)
for year in range(years['min'], years['max'] + 1):
for month in range(1, 13):
files[f"{year}-{month}"] = open(f"{output_folder}/{year}-{str(month).rjust(2, '0')}.rec", 'w')
try:
for index, row in owntracks.iterrows():
d = row.to_dict()
record = {
'_type': 'location',
'tid': tracker_id
}
record['tst'] = int(time.mktime(d['timestamp'].timetuple()))
for key in ['lat', 'lon']:
if key in row and not pd.isnull(row[key]):
record[key] = row[key]
for key in ['acc', 'alt', 'vac']:
if key in row and not pd.isnull(row[key]):
record[key] = int(row[key])
timestamp = row['timestamp'].strftime("%Y-%m-%dT%H:%M:%SZ")
line = f"{timestamp}\t* \t{json.dumps(record, separators=(',', ':'))}\n"
files[f"{d['timestamp'].year}-{d['timestamp'].month}"].write(line)
finally:
for key, file in files.items():
file.flush()
file.close()
|
After it generates the data, find your OwnTracks data directory. The copy the contents of the output/ folder into /{owntracks_store_dir}/store/{user}/{device} folder. If you replace any existing data you will lose data recorded from Opentracks itself, so I would recommend downloading the Google Location History data first, converting, importing, then start collecting data afterwards.
When I loaded this, the history was crazy showing me moving in and out of a city instantly:
Turns out, multiple devices on my accounts were reporting locations so it appeared that I kept moving between two cities. Easy enough to exclude the device.
The following script produces a chart that shows when devices are reporting their location:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| df_gps.sort_values(by='timestamp', inplace=True)
devices = df_gps['deviceTag'].unique()
plt.figure(figsize=(10, 6))
colors = plt.cm.viridis_r([i / len(devices) for i in range(len(devices))])
for i, device in enumerate(devices):
device_data = df_gps[df_gps['deviceTag'] == device]
first_year = device_data['timestamp'].dt.year.min()
last_year = device_data['timestamp'].dt.year.max()
middle_year = (first_year + last_year) / 2
plt.hlines(y=i, xmin=first_year, xmax=last_year, color=colors[i], linewidth=4)
plt.text(middle_year, i + 0.25, i, verticalalignment='center', fontsize=8)
plt.xlabel('Year')
plt.ylabel('Device')
plt.title('Device Reporting Years')
plt.grid(True)
plt.show()
|
With this I can identify which device is reporting in parallel. In my case, it’s the 10th device that’s reporting in parallel with my phone. Note that I’ve replaced the device ids, yours will be a long random number:
From there, I filtered out this device from the location history
1
2
3
4
5
| +excluded_devices = [1234510]
owntracks = df_gps.rename(columns={'latitudeE7': 'lat', 'longitudeE7': 'lon', 'accuracy': 'acc', 'altitude': 'alt', 'verticalAccuracy': 'vac'})
owntracks['tst'] = (owntracks['timestamp'].astype(int) / 10**9)
+owntracks = owntracks[~owntracks['deviceTag'].isin(excluded_devices)]
|
And the history looks a lot better:
Adding new data
Now, once you have your historical data, what do you do for new data moving forward? There’s a lot of options depending on personal preference including:
- Using the OwnTracks mobile apps
- Copying location from Home Assistant like here. This is the method that I use.
Conclusion
OwnTracks is an open-source place to store your location history. You can import existing history from Google Maps, then report new data into OwnTracks and have a localized copy of your location history.
Updates
- 2024-05-04 - Corrected a few bugs in the Python code
- 2024-07-15 - Automatically create output folder in Python
- 2024-08-02 - Updated the timestamp parsing logic again
- 2024-08-28 - Updated downloading section to include Android and iOS examples (thanks to reader contribution)
Comments
Comments are currently unavailable while I move to this new blog platform. To give feedback, send an email to adam [at] this website url.