Looking for the data export tool without the tear-jerking narrative? Go straight to the instructions.
Back in 2010, EveryTrail was the site to use if you wanted to find an awesome set of GPS tracks to follow for a hike. My first (and fondest) memory of it is an absolutely epic village-hopping hike tracked by EveryTrail user ebrunach in Anhui, China. Before setting off, we downloaded the trail to our phones, which included not only the route, but also geotagged photos, a play-by-play text summary of the sights, and annotated waypoints like “start of jungle.”
Over two days and three charged phone batteries, my friends and I faithfully monitored our screens and followed the route they spelled out. A few times, we lost the trail, and had to backtrack to realign our blue location dot with ebrunach’s sinuous, improbable path, leading us through jungly hills, quiet farming valleys, and weathered trade roads. At the summit of one of those hills, we stopped to gaze across the vast green panorama stretching from the village where we’d stayed to our next destination village. It was then, marveling at how effortlessly this seemingly undocumentable adventure had been made available to us, when we first truly felt the power of this tool.
My favorite way to use EveryTrail has always been their World Map, which can be zoomed in anywhere to find tracked activities. Virtually every corner of the earth abounds with trails contributed by the community. For my part, I’ve documented a modest collection of my own hiking and backpacking trips.
TripAdvisor acquired EveryTrail in 2011, and thereafter apparently abandoned it: EveryTrail’s Twitter account hasn’t posted an update in a year and a half, links on EveryTrail’s homepage to their blog and forum go nowhere, and entire TripAdvisor support threads are dedicated to the inability to get support for EveryTrail.
While EveryTrail’s rich database has endured, its website and app have predictably declined in utility. These days, when tracking hikes, I record using RunKeeper then upload the resulting GPX file to EveryTrail (EveryTrail’s app is a comparative battery hog and has crashed while tracking). When following someone else’s trail, I download the raw data from EveryTrail and import it to an app like AlpineQuest (which provides more useful tools mid-hike such as distance traveled, elevation, and multiple map tiles).
Nonetheless, EveryTrail remains the most comprehensive site on the web to find and view trails. As the creator of Gaia GPS, another GPS app, concluded in 2014, “using EveryTrail feels a little dated, but in terms of data there is still no better compendium than EveryTrail.” I’ve continued contributing my hikes, many with trip notes and photos, because it’s the most effective platform for others to find and enjoy the same routes.
I could deal with the aging interface and clunky transfer methods so long as EveryTrail continued to serve that role as a platform. But in recent months, the site’s availability has been spotty, sometimes going down for days at a time. Even the individual trip page, which previously included graphs and photos on the map, has regressed (or broken) so that it displays far less information.1 The final straw was when, earlier this year, some friends and I wanted to repeat our favorite river-tracing trail, which we’d discovered and tracked so long ago that it happened to be saved only on EveryTrail. The site was down for days, leaving us hanging until it came back up the morning when we were leaving.
I realized it’s only a matter of time before I lose access to everything I’ve stored there: not merely my GPS data, but also many trip notes and selections of photos which I haven’t stored elsewhere.
Getting my data out
I wrote a Python script which scrapes an EveryTrail trip page and saves its data. Using this script just automates a series of copy-and-paste and save-link-as operations that I would otherwise do by hand, which is consistent with their Terms of Service.2
Since this script relies on a specific layout of the trip page, an update on EveryTrail’s end could easily break it. But let’s be real, what are the odds of that happening?
The code is on GitHub, along with instructions to install and run the script.
Passing the login cookie from web browser to scraper
The only real gotcha is that you must be logged into EveryTrail in order to download a trip’s GPX file. This is where the
--trailauth option comes in. I first log into EveryTrail as usual on my web browser, which causes EveryTrail to set a cookie called
TRAILAUTH to keep me logged in. I then find the value of that cookie and pass it to the script. It can thereby download the GPX files as if logged in as me.
Downloading a trip or three
Here’s what it looks like to download a single trip:
$ everytrail_export 2671553 --trailauth d9b61ab30a10... Trip 1/1: Downloading http://www.everytrail.com/view_trip.php?trip_id=2671553 Ventana Double Cone loop - Big Sur - California, United States Saved trails/2671553-ventana-double-cone-loop-big-s/title.txt Saved trails/2671553-ventana-double-cone-loop-big-s/info.html Saved trails/2671553-ventana-double-cone-loop-big-s/stats.html Saving GPX and KML files... Downloading http://www.everytrail.com/downloadGPX.php?trip_id=2671553 Saved trails/2671553-ventana-double-cone-loop-big-s/2671553.gpx Downloading http://www.everytrail.com/downloadKML.php?trip_id=2671553 Saved trails/2671553-ventana-double-cone-loop-big-s/2671553.kmz Downloading photos page: http://www.everytrail.com/trip_pictures.php?trip_id=2671553&code= Photo 1/21: Downloading photo info page: http://www.everytrail.com/view_picture.php?trip_id=2671553&picture_id=7004037 ----- Response came back with HTTP status 502; trying again... ----- Finding full photo: http://www.everytrail.com/picture/vieworiginal?picture_id=7004037 Downloading full photo: http://images.everytrail.com/pics/original/7004037-IMG_7611.jpg Saved "Setting off" to trails/2671553-ventana-double-cone-loop-big-s/images/7004037-IMG_7611.jpg ... Photo 21/21: Downloading photo info page: http://www.everytrail.com/view_picture.php?trip_id=2671553&picture_id=7004057 Finding full photo: http://www.everytrail.com/picture/vieworiginal?picture_id=7004057 ----- Response came back with HTTP status 502; trying again... ----- Downloading full photo: http://images.everytrail.com/pics/original/7004057-IMG_7805.jpg Saved "Stream crossing" to trails/2671553-ventana-double-cone-loop-big-s/images/7004057-IMG_7805.jpg Saved trails/2671553-ventana-double-cone-loop-big-s/photo_info.json
Yes, those 502s are fairly typical. We’re not a moment too soon…
Downloading trips is also possible via trip page URL rather than trip ID:
$ everytrail_export \ http://www.everytrail.com/view_trip.php?trip_id=2671553 \ http://www.everytrail.com/view_trip.php?trip_id=2991898 \ http://www.everytrail.com/view_trip.php?trip_id=2348794 \ --trailauth d9b61ab30a10...
Downloading all my trips at once
Rather than specify the URL for each trip individually, the
--trips-page option scrapes my trips page for them:
$ everytrail_export \ --trips-page http://www.everytrail.com/my_trips.php?user_id=154142 \ --trailauth d9b61ab30a10... Scraping http://www.everytrail.com/my_trips.php?user_id=154142 for trip URLs... Found links to 15 trips: 1550019 1673357 1693258 1733157 1741278 1820769 1924844 1924847 2022884 2053816 2108623 2301920 2348794 2671553 2991898 Trip 1/15: Downloading http://www.everytrail.com/view_trip.php?trip_id=1550019 El Corte de Madera Creek Trail - California, United States ...
There are many people who have put much more energy than I have into tracking routes, writing trip stories, captioning photos, and marking waypoints. EveryTrail is a genuinely revolutionary platform in that it has encouraged tens of thousands to compile their outdoor experiences for the world.3 It would be a great shame if, along with EveryTrail, all those trips were to rot and die, unavailable to share with friends, never to be discovered by new travelers.
There are plenty of alternatives solving the same problems as EveryTrail, and I will migrate to one eventually. Some of them show serious promise, and some can even import from EveryTrail. Still, none has reached the same critical mass, and all have the potential to remain siloed in the same way.
I designed this export tool with the goal of capturing as much raw data as someone would need to republish their routes with the same degree of detail elsewhere. EveryTrail did a great job presenting my trips, but they are ultimately my property. Anyone who has put energy into creating trip content should have a straightforward way of sharing it with others, while retaining faith in its continued existence.
I’m still exploring what that might look like.
A simple first step, which solves the problems of ownership and hosting, is to build a tool which converts the exported EveryTrail data into a self-contained static site which can be hosted somewhere like Neocities. The problems of indexing and discovery would remain unsolved.
In a utopian future world, a single format, probably endorsed by the OGC, will emerge as the standard for representing anything from bike races to photo-annotated travel logs, and all sites will support free import and export of that format. When we live in that world, extending this script to synthesize all its output in that format will be a trivial task.
Stay tuned for part 2. If you have ideas about this, I’d love to hear from you.
The map for a trip like this one would previously show a graph like this, or alternatively play a slideshow of photos demonstrating their locations along the trail, but now neither feature is present. The “fullscreen” view appears to still have controls for those features, but it hangs on the loading screen and displays nothing. ↩
EveryTrail’s Terms of Service state that the company (1) “does not claim ownership of User Content you submit or make available for inclusion on the Service, except as specifically provided otherwise” and grants a (2) “license to access and use the Web Sites and electronically copy (except where prohibited without a license) and print to hard copy portions of the Site Materials for your informational, non-commercial and personal use only.” This license excludes (3) “use of any data mining, robots or similar data gathering or extraction methods.” I am downloading only my data, which I own, strictly for personal use. Doing so means accessing a specific, bounded subset of the site, rather than crawling links deeply to discover and gather content as the terms “data mining” and “robots” imply. ↩
I have no idea how many EveryTrail users have actually contributed trips, at least not without writing a scraper which much more clearly violates their Terms of Service. But a few manual iterations of changing the URL to find the maximum valid user ID suggest that they have just about three million users. A very rough estimate using the 90-9-1 principle is that 30,000 users have contributed trips. ↩