AI Segmentation¶

Processing → GeoLibre Toolbox → AI Segmentation turns imagery into vector features using segment-geospatial (SamGeo) and Meta's SAM 3 model. Describe what you want — "trees", "buildings", "water" — or run automatic segmentation, and GeoLibre adds the resulting polygons as a new vector layer.

How it works¶

The heavy model stack (PyTorch + SAM 3) does not run inside the GeoLibre sidecar. Instead the sidecar exposes a thin /ml reverse-proxy in front of a separate samgeo-api server (the REST server shipped with segment-geospatial). GeoLibre uploads the image to the sidecar, which forwards it to samgeo-api, runs SAM 3, and returns GeoJSON polygons.

SegmentationDialog ──image+prompt──▶ sidecar /ml/segment/* ──▶ samgeo-api (SAM 3) ──▶ GeoJSON

This keeps the GeoLibre sidecar small and lets the model server run wherever you have a GPU.

Setup¶

Install segment-geospatial with the API and SAM 3 extras in a Python environment that has a working PyTorch build (ideally CUDA):

pip install "segment-geospatial[api,samgeo3]"

Then either let GeoLibre launch the model server for you, or run it yourself:

• Auto-launch (default). If samgeo-api is on the PATH of the environment the sidecar runs in, GeoLibre starts it automatically on first use.

• Run it yourself / on another machine. Start the server and point GeoLibre at it:

  samgeo-api --port 8000
  # then run the sidecar with:
  GEOLIBRE_ML_SAMGEO_URL=http://127.0.0.1:8000
  

# in your PyTorch env
samgeo-api --port 8000

# launch the desktop app with the proxy target set
GEOLIBRE_ML_SAMGEO_URL=http://127.0.0.1:8000 npm run tauri:dev

Install the sidecar's optional ml extra (just an HTTP client — the models live in samgeo-api):

pip install -e "backend/geolibre_server[ml]"

Configuration¶

Using it¶

1. Open Processing → GeoLibre Toolbox → AI Segmentation.
2. If the backend isn't reachable, the dialog shows a message and a Start server button (desktop) — click it to launch the sidecar and model server.
3. Choose a GeoTIFF. Pick a georeferenced raster (.tif/.tiff; PNG/JPG also work but produce pixel-space results). The polygons come back in the raster's coordinate system, so they land in the right place on the map.
4. Pick a mode:
   • Text prompt — type what to segment (e.g. trees, buildings, cars) and set a Confidence threshold (0–1; lower finds more, higher is stricter). SAM 3 finds every matching object in the image.
   • Automatic (everything) — segments all distinct objects without a prompt.
5. Click Segment. The result is added as a new GeoJSON polygon layer named after the prompt, and the map zooms to it.

Notes & limitations¶

• SAM 3 only. GeoLibre uses the SAM 3 backend, which covers text, box/point, and automatic prompts well. (SAM 2 is intentionally not wired up.)
• Desktop only. Like the raster tools, segmentation needs the desktop app and the Python sidecar; it is not available in the browser-only build.
• No objects found is a normal outcome — try a different prompt or lower the confidence threshold.
• Box / point prompts drawn on the map are available in the interactive SamGeo plugin below; this dialog covers text and automatic modes only.

Interactive SamGeo plugin¶

Plugins → SamGeo opens a dockable panel that talks to samgeo-api directly (default http://127.0.0.1:8000, editable in the panel) rather than through the sidecar, so it works in the browser build as well as on desktop. It mirrors segment-geospatial's own interactive map: one uploaded image is shared by every prompt type, and results land as GeoJSON polygon layers.

1. Start the model server (samgeo-api --port 8000; the panel links to the setup guide) and click Check connection.
2. Choose the image. Image source lists every COG or GeoTIFF layer already on the map — pick one to segment it in place — or leave it on Upload a file and choose a local .tif/.png/.jpg. The Aerial imagery (UC Berkeley) sample in Add Data → Raster Layer is a good first image.
3. Pick a mode:
   • Text prompt — a concept such as building or tree, with a confidence threshold and mask-size limits.
   • Point prompts — click + Foreground point / − Background point, then click the map; green and red markers show the prompts. At least one foreground point is required.
   • Bounding box (find similar) — click Draw box on map and drag a rectangle; SAM 3 returns every object that resembles the boxed one.
   • Automatic (everything) — segments all distinct objects. SAM 3 is a prompt-driven model, so this mode runs SAM 2's automatic mask generator instead (choose the checkpoint under SAM2 model; points_per_side, IoU and stability thresholds apply here).
4. Click Segment. Polygons are reprojected from the raster's CRS to WGS84 and added as a layer named after the prompt or mode; the map zooms to it. With segment-geospatial 1.4.2 or newer each feature carries a score attribute with the model's confidence, visible in the attribute table.

Use Clear prompts to remove the points or box. Switching modes clears the other mode's prompts, and closing the panel removes the on-map prompt overlay. The panel's settings (API URL, mode, model, thresholds) persist with the project.

API (advanced)¶

The sidecar exposes these endpoints (proxied to samgeo-api):

Each segment/* endpoint takes a multipart file (the image) plus model_version (default sam3) and output_format (default geojson) and returns a GeoJSON FeatureCollection.

See also Processing Tools and Reference → Architecture.