More CLI Tips & Tricks
This document shows off a range of more advanced command-line workflows, making use of a wider range of tools in the command-line & geospatial ecosystem. Some of them can be a pain to install, like GDAL/OGR, and several pop in and out of web tools, so these are kept out of the main tutorial section.
WORK IN PROGRESS: This document is still under construction, with a number of TODO’s remaining, but we are publishing as there’s a lot of good information here.
- GDAL/OGR - We’ll mostly use OGR, the vector tooling. Great for things like format conversion and basic simplification.
- Keplergl_cli - Nice tool to call the awesome kepler.gl library from the commandline. Useful for visualization of large amounts of geojson.
- GeoJSON.io - Simple tool to do editing of geojson, useful for creating AOI’s. It integrates with github, but the ability to save a GeoJSON to github doesn't seem to work so well.
- Placemark.io - More advanced tool from the creator of GeoJSON.io, very nice for creating AOI’s and piping them in, with lots of rich geometry editing features.
- MapShaper - Tool to do interactive simplification of GeoJSON, has a nice CLI.
- STACTools - CLI for working with STAC data. There is also a planet plugin that may be useful in the future.
- Github commandline - Nice CLI for quickly posting things to gists.
While the command-line can often be quicker than using a UI, one place that can be slower is getting the geometry input for searching or clipping. Hand-editing GeoJSON is a huge pain, so most people will open up a desktop tool like QGIS or ArcGIS Pro and save the file. But there are a few tools that can get you back into the CLI workflow more quickly.
Draw with GeoJSON.io¶
One great tool for quickly drawing on a map and getting GeoJSON output is GeoJSON.io. You can draw and save the file, but an even faster workflow is to use your operating system’s clipboard to command-line tools.
Just draw with the tools, then copy (with right click and 'copy' from the menu or 'ctrl c') the JSON from the text box:
On a mac you can use
pbpaste to grab whatever is currently in your clipboard:
pbpaste | planet data filter --geom - | planet data search SkySatCollect --filter -
(TODO: Find alternatives for windows and linux)
Draw with Placemark¶
A really fantastic tool for working with GeoJSON is Placemark. It is a
commercial tool that you’ll have to pay for, but it’s got a really nice feature that makes it very
compatible with command-line workflows. You can easily grab the URL of any individual GeoJSON
feature and stream it in as your geometry using
curl -s https://api.placemark.io/api/v1/map/a0BWUEErqU9A1EDHZWHez/feature/278cd610-05ee-11ed-8fdd-15633e4f8f01 | \ planet data filter --geom - | jq
Looking at a big list of GeoJSON is a lot less useful then actually being able to see the footprints. This is often
done by saving the output files and opening with a desktop GIS program, but there are some nice alternatives that
let you pipe (
|) the output more directly.
Copy GeoJSON to clipboard¶
One of the quicker routes to visualizing search output is to copy the output to your clipboard and paste into a tool that will take GeoJSON and visualize it.
You can do this on GeoJSON.io:
(TODO: record example)
Or also on Placemark, which tends to perform a bit better (especially when you get above 1000 features).
(TODO: record example)
For both it’s recommended to pass the output through
planet collect to get properly formatted GeoJSON:
planet data filter --string-in strip_id 5743669 | planet data search PSScene --filter - | planet collect - | pbcopy
(TODO: Get pbcopy equivalents for windows and linux)
Post to Github as gist¶
Another easy option that is a bit more persistent is to post to Github using the
gh cli tool. Specifically using the
gist create command.
The following command will get the latest SkySat image captured, upload to github, and open your browser to see it:
planet data search SkySatCollect --sort 'acquired desc' --limit 1 \ | planet collect - | jq | gh gist create -f latest-skysat.geojson -w
Or you can show all ps-scenes in a strip on github gist. (You may need to reload the page, for some reason it doesn't always showing up immediately after open)
planet data filter --string-in strip_id 5743640 | planet data search PSScene --filter - \ | planet collect - | gh gist create -f ps-search.geojson -w
TODO: get a command that gets the latest strip id and uses that as input in one line. May need to update filter commands to take stdin? This current command doesn't quite work.
strip-id=`planet data search PSScene --limit 1 \ | jq -r '.properties.strip_id' | sed 's/\\[tn]//g'` planet data filter --string-in strip_id $stripid | planet data search PSScene --filter -
One of the best tools to visualize large numbers of imagery footprints is a tool called kepler.gl,
which has a really awesome command-line version which is perfect for working with Planet’s CLI. To get the CLI go to
keplergl_cli and follow the
installation instructions. Be sure to get a Mapbox API key (from
the access tokens page) - just sign up for a free account if you don't have
one already. The kepler CLI won't work at all without getting one and setting it as the
Once it’s set up you can just pipe any search command directly to
kepler (it usually does fine even without
planet collect to go from ndgeojson to geojson). For example:
curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json \ | planet data filter --geom - \ | planet data search PSScene --filter - \ | kepler
(TODO: Add animated gif, showing some options)
Kepler really excels at larger amounts of data, so try it out with larger limits:
curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json \ | planet data filter --geom - \ | planet data search PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect,Sentinel1 \ --sort 'acquired desc' --limit 1500 --filter - \ | kepler
(show animated gif with 600 - lower amount so it takes less time to load).
And you can bring it all together using Placemark for input and Kepler for output:
(TODO: Figure out why this link isn't working... Maybe needs to be smaller?)
curl -s https://api.placemark.io/api/v1/map/a0BWUEErqU9A1EDHZWHez/feature/91a07390-0652-11ed-8fdd-15633e4f8f01 \ | planet data filter --geom - \ | planet data search PSScene,Landsat8L1G,SkySatCollect,Sentinel1 --filter - | kepler
Large Dataset Visualization¶
Oftentimes it can be useful to visualize a large amount of data, to really get a sense of the coverage and then do some filtering of the output. For this we recommend downloading the output to disk. Getting 20,000 skysat collects will take at least a couple of minutes, and will be over 100 megabytes of GeoJSON on disk.
planet data search SkySatCollect --limit 20000 > skysat-large.geojsons
Kepler can fairly easily handle 20,000 skysat footprints, try:
Many GIS programs will open that geojson file by default. But if you have any trouble it can be useful to run it
planet data collect skysat-large.geojsons > skysat-large-clean.geojson
If you want to visualize even larger sets of footprints there’s a few things we recommend:
Convert to GeoPackage¶
GeoJSON is an amazing format for communicating online, but is less good with really large amounts of data, so we recommend converting it to a format that has a spatial index and isn't so large on disk. We recommend geopackage, but shapefile can work as well.
ogr2ogr of GDAL/OGR is a great tool for this. We
recommend using the binaries, or if
you’re on a Mac then use homebrew (run
brew install gdal
after you get it set up). If you’re having trouble getting GDAL working well a
good backup can be to use docker and the
To convert to a geopackage run:
ogr2ogr skysat-large.gpkg skysat-large.geojsons
The Python program
Fiona and plugin commands
from the fio-planet package are
another option for converting and manipulating GeoJSON. Fiona and fio-planet
are easy to install alongside the Planet SDK and CLI. The Fiona way to convert
a sequence of GeoJSON features from a search to Geopackage is this:
cat skysat-large.geojsons | fio load -f GPKG skysat-large.gpkg
Simplification with Fiona and fio-planet¶
There are a number of ways to simplify areas of interest. The one that best balances effectiveness with ease of use is the union, or merger, of concave hulls.
cat skysat-large.geojsons \ | fio map 'concave_hull g :ratio 0.4' --dump-parts \ | fio reduce 'unary_union c' \ | fio load -f GPKG skysat-large-simplified.gpkg
ratio parameter describes the desired distance from the
algorithm. A ratio of 1.0 produces hulls that are the same as convex hulls.
Smaller ratios preserve the character of concave features better.
Simplification with OGR¶
The other thing you’ll likely want to do to visualize large amounts of data is to simplify it some. Many simplification tools call for a 'tolerance', often set in degrees. For SkySat some useful values are:
|0.001||Mostly removes unnecessary points, visually looks pretty much the same, but much easier for programs to render.|
|0.01||Messes with the shape a bit, but the footprint generally looks the same, with a couple vertices off.|
|0.1||Mashes the shape, often into a triangle, but still useful for understanding broad coverage.|
It’s worth experimenting with options between these as well. The more simplification the easier it is for programs to
render the results.
ogr2ogr includes the ability to simplify any output:
ogr2ogr skysat-large.gpkg skysat-large.json -simplify .008
Alternative - use convex hull. TODO: test this, write it up
ogr2ogr skysat-convex.gpkg skysat.geojson ogr2ogr -sql "select st_convexhull(geometry) from skysat" -dialect sqlite
Other alternative for really big ones, centroid. GDAL should be able to do this, need to figure out the similar sql.
Simplification with Mapshaper¶
Another great tool is Mapshaper, which excels at simplification. It offers a web-based user interface to see the results of simplification, and also a command-line tool you can use if you find a simplification percentage you’re happy with. After you get it installed you can fire up the UI with:
(TODO: Show animated gif of recording)
It’s easy to get a sense of how much simplification affects the shape. You can download the output from the web interface, or you can also run the command-line program:
mapshaper -i footprints.geojson -simplify 15% -o simplified.geojson
Once you find a simplification amount you’re happy with you can use it as a piped output.
planet data search --limit 20 SkySatCollect - | planet collect - | mapshaper -i - -simplify 15% -o skysat-ms2.geojson
Mapshaper also has more simplification algorithms to try out, so we recommend diving into the CLI options.
Simplification with QGIS¶
Another good tool for simplification is QGIS.
TODO: Flesh out this section, add in command-line qgis_processing option.
Other simplification options for large datasets:
- Use QGIS, run 'convex hull' (Vector -> Geoprocessing -> Convex Hull). Good idea to convert to gpkg or shapefile before you open in qgis if large.
do a limit 0 (unlimited) on a constrained search (geom and time range) with jq count to see how many scenes are in the area (note you can also do
get id by array number
planet orders list | jq -rs '. | "\(.id) \(.created_on) \(.name) \(.state)"'
- Use jq to show just orders that have a given item type, like just skysat.
planet orders list | jq -rs '. | "\(.id) \(.created_on) \(.state) \(.products.item_type)"'
will show the item type https://gist.github.com/ipbastola/2c955d8bf2e96f9b1077b15f995bdae3 has ideas for contains, but haven't got it right yet
use jq to get the id of the an order by it’s name
get total number of items, add up each year of
Simplify Geometries to 500 vertices¶
One of the limits of Planet’s API’s is that they demand geometries have less than 500 vertices. This section shows some tools that can help you to do that simplification.
TODO: flesh these out.
- simplify CLI
- simplify UI, download and use
- count number of resulting vertices
ogrinfo -al simplify_test.geojson | grep POLYGON | sed 's/$/,/' | tr -d -c "," | wcBut there should be a JQ way to do the same...
- see count of vertices
- union together
- buffer and union