GeoAI Building Intelligence

LiDAR + Sentinel-2 + OpenStreetMap, Washington DC
Geospatial AI Demo

Multimodal building extraction, validated the way a GIS reviewer actually checks it

A real USGS 3DEP LiDAR tile and a cloud-free Sentinel-2 scene, fused into per-pixel features (canopy/building height, NDVI, terrain slope, texture), trained against real OpenStreetMap building footprints, and evaluated at both the pixel level and the polygon level, over a dense rowhouse block on Capitol Hill, Washington DC.

AOI: 38.882-38.888N, -77.005 to -76.998W · 2m grid, 327×311 px · spatial holdout test region (right 25% of grid)
79.7%
Pixel F1, multimodal RF
66.3%
Pixel IoU, multimodal RF
44.6%
Building-level detection rate
74
OSM buildings in test region

The raster stack

Every layer below is resampled onto the same 2m grid in EPSG:32618 (UTM 18N). nDSM comes from the LiDAR point cloud (DSM minus DTM); NDVI comes from Sentinel-2's native 10m red/NIR bands upsampled onto the 2m grid, which is why NDVI looks visibly blockier than the LiDAR-derived layers, an honest illustration of the spatial-resolution tradeoff between the two sensors.

nDSM (LiDAR building/canopy height, 0-25m)
NDVI (Sentinel-2, resampled to 2m)
Ground truth: OSM buildings (gold) over nDSM
RandomForest prediction (green) over nDSM

Why multimodal fusion beats either sensor alone

Two naive single-feature baselines, evaluated on the exact same held-out test region as the full model, make the case with real numbers rather than a marketing claim:

ModelPrecisionRecallF1IoU
NDVI-only (low NDVI → guess building)77.3%43.7%55.8%38.7%
nDSM-only (height > 2.5m → guess building)35.7%99.7%52.5%35.6%
RandomForest, nDSM+NDVI+slope+texture79.3%80.2%79.7%66.3%

NDVI-only is precise but misses most buildings (roads, parking lots, and bare soil also read as "not vegetation"). nDSM-only catches almost everything raised off the ground (99.7% recall) but drags in every tree along the way, cratering precision. Only the combined model gets both right.

Feature importance

ndvi
45.2%
ndsm
39.8%
texture
11.8%
slope
3.2%

The part pixel accuracy hides: polygon-level evaluation

Pixel metrics look strong (F1 79.7%), but a GIS reviewer cares about individual buildings, not pixels. Vectorizing the prediction and matching it against individual OSM building polygons in the test region tells a more honest story:

MetricValue
Ground-truth buildings in test region74
Predicted building polygons72
Matched at IoU ≥ 0.333
Building-level detection rate44.6%
Mean matched-polygon IoU0.61
Mean building area error36.1%
Mean centroid displacement6.5 m
The detection rate (44.6%) is much lower than the pixel F1 suggests, and the reason is structural, not a model bug: Capitol Hill rowhouses share party walls, so neighboring units present one continuous rooftop in the nDSM with no height discontinuity between them. The classifier correctly flags the roofline as "building" at the pixel level, but the vectorizer then merges 3-4 attached rowhouses into a single polygon, which counts as one match against many ground-truth parcels. This is exactly the kind of instance-segmentation problem pixel accuracy hides and polygon-level GIS evaluation exposes.

Grounded error explanations (Bedrock Amazon Nova Lite)

Three real misclassifications from the held-out test region, with Nova Lite explaining each one from its actual measured nDSM/NDVI/slope values, not a generic answer:

False positive (predicted building, none in OSM)
nDSM 9.0 m · NDVI 0.21 · slope 1.0°

The false positive likely occurred due to the relatively low normalized surface height (nDSM) of 9.00 m, which might be near the classifier's threshold for distinguishing buildings from non-buildings. Additionally, the low NDVI value of 0.21 suggests sparse vegetation, which could cause the classifier to misinterpret the area as built-up rather than natural terrain. The gentle terrain slope of 1.0 degree further supports this, as it is unlikely to cause significant misclassification on its own.

False negative (missed a real OSM building)
nDSM 9.7 m · NDVI 0.65 · slope 7.7°

The false negative is likely due to the moderate NDVI value of 0.65, indicating a significant presence of vegetation which may have confused the classifier, making it difficult to distinguish the building from the surrounding vegetation. Additionally, the terrain slope of 7.7 degrees might have contributed to the error by causing distortions in the LiDAR data, further complicating the classification.

False negative (missed a real OSM building)
nDSM 12.8 m · NDVI 0.43 · slope 1.6°

The false negative classification is likely due to the moderate normalized surface height (nDSM) of 12.85 m, which may be near the threshold for building detection, combined with a relatively low NDVI of 0.43 indicating moderate vegetation cover that could obscure building features. Additionally, the gentle terrain slope of 1.6 degrees might reduce the distinctness of building structures in the LiDAR data.

GIS / Remote Sensing knowledge benchmark

GIS and remote sensing is a domain where an AI can sound confident and still be subtly wrong, so the pipeline turns that lens on itself: 30 domain questions put to Amazon Nova Lite, with the ones this project has direct ground truth for (DSM/DTM/nDSM, LiDAR returns) graded against that ground truth rather than taken at face value.

CRS/ProjectionsWhat is the difference between a geographic coordinate system and a projected coordinate system?

A geographic coordinate system (GCS) uses latitude and longitude to define locations on the Earth's surface in a three-dimensional space, referenced to the ellipsoid model of the Earth. In contrast, a projected coordinate system (PCS) transforms the curved surface of the Earth into a flat, two-dimensional surface using specific projections, which minimizes distortion for specific areas or purposes.

CRS/ProjectionsWhy can EPSG:4326 (WGS84) coordinates not be used directly for area or distance calculations in meters?

EPSG:4326 (WGS84) coordinates represent geographic latitude and longitude, which are angular measurements and not linear. Therefore, distances and areas calculated directly from these coordinates would be inaccurate because they do not account for the Earth's curvature and the distortion that occurs when projecting a spherical or ellipsoidal surface onto a flat plane.

Raster vs VectorWhat is the fundamental difference between raster and vector spatial data models, and when would you prefer one over the other?

The fundamental difference between raster and vector spatial data models lies in their representation of geographic data: raster data uses a grid of cells to store values (e.g., elevation, temperature), while vector data employs points, lines, and polygons to represent discrete objects. Vector data is preferred for precise boundary representation and attribute-rich information, whereas raster data is better suited for continuous data and seamless integration with imagery.

GeoTIFF/COGWhat makes a Cloud-Optimized GeoTIFF (COG) different from a regular GeoTIFF, and why does that matter for cloud-based analysis?

A Cloud-Optimized GeoTIFF (COG) differs from a regular GeoTIFF by incorporating tiled structure and efficient compression, such as JPEG2000 or DEFLATE, which allows for faster data access and streaming, significantly enhancing performance in cloud-based analysis by enabling rapid subsetting and parallel processing. This optimization is crucial for handling large datasets efficiently in cloud environments.

NDVIHow is NDVI calculated, and what does a value close to zero versus close to one indicate?

NDVI (Normalized Difference Vegetation Index) is calculated using the formula (NIR - Red) / (NIR + Red), where NIR is the near-infrared and Red is the visible red light reflected from vegetation. A value close to zero typically indicates non-vegetative surfaces like water or bare soil, while a value close to one indicates dense, healthy vegetation.

SAR vs OpticalWhat are the main advantages of SAR imagery over optical imagery for remote sensing, and what is one major drawback?

SAR imagery offers the advantages of all-weather and day-night imaging capabilities, providing high-resolution data that is effective through cloud cover and darkness. It is also sensitive to surface roughness and moisture content, which is beneficial for applications like land cover classification and monitoring. However, one major drawback is its relatively complex data interpretation and processing requirements compared to optical imagery.

SARWhat causes speckle noise in SAR imagery, and how is it typically mitigated?

Speckle noise in SAR imagery arises from coherent interference of backscattered signals from various scatterers within a resolution cell. It is typically mitigated using filters such as Lee, Frost, or wavelet-based methods, which smooth the image while preserving edges.

DSM/DTM/DEMExplain the difference between a DTM, a DSM, and a DEM.

A Digital Terrain Model (DTM) represents the bare earth surface, excluding any objects like buildings or vegetation. A Digital Surface Model (DSM) includes all surfaces, such as buildings and vegetation, providing a comprehensive view of the Earth's surface. A Digital Elevation Model (DEM) is a type of DTM that specifically quantifies the elevation of the terrain.

Verified correct Standard, accurate definitions - though in practice 'DEM' is often used loosely as a synonym for DTM rather than a distinct third category, which the answer doesn't flag.
nDSMWhat is a normalized Digital Surface Model (nDSM) and how is it computed from a DSM and DTM?

A normalized Digital Surface Model (nDSM) is a raster dataset that separates ground and non-ground surface elements by subtracting a Digital Terrain Model (DTM) from a Digital Surface Model (DSM). It is computed by performing a pixel-wise subtraction of the DTM from the DSM, resulting in a model that highlights the height of above-ground features such as buildings and vegetation.

Verified correct Matches the actual nDSM = DSM - DTM computation used in this pipeline's own preprocessing step.
PhotogrammetryWhat is orthorectification and why is it a necessary step before mosaicking aerial or satellite imagery?

Orthorectification is the process of correcting aerial or satellite imagery to account for sensor movement and terrain-induced distortions, ensuring that the imagery accurately represents the Earth's surface in a uniform projection. It is a necessary step before mosaicking because it aligns and standardizes the geometric integrity of individual image tiles, which is crucial for creating a seamless and accurate composite image.

PhotogrammetryHow does photogrammetric structure-from-motion differ from LiDAR for generating a digital surface model?

Photogrammetric structure-from-motion (SfM) uses overlapping images to triangulate point positions, creating a dense point cloud and subsequently a digital surface model (DSM), while LiDAR employs laser pulses to directly measure distances to the Earth's surface, providing a more accurate and direct representation of the terrain. SfM is generally less expensive but can be less precise compared to LiDAR.

ResolutionExplain the difference between spatial, temporal, and spectral resolution in the context of a satellite sensor.

Spatial resolution refers to the smallest object that can be detected on the Earth's surface, measured in meters per pixel. Temporal resolution indicates how frequently a satellite revisits the same location to capture data, typically in days. Spectral resolution denotes the sensor's ability to distinguish between different wavelengths of light, influencing the detail in the spectral information obtained.

ResolutionWhy might a sensor with coarser spatial resolution but finer temporal resolution be preferred for agricultural monitoring?

A sensor with coarser spatial resolution but finer temporal resolution might be preferred for agricultural monitoring to capture frequent changes in crop conditions and growth stages over time, which is crucial for timely decision-making in crop management. This allows for more frequent updates on crop health, stress detection, and growth patterns, despite the lower detail in individual observations.

Segmentation/DetectionWhat is the difference between semantic segmentation and instance segmentation in the context of extracting building footprints from imagery?

Semantic segmentation classifies each pixel in an image into predefined categories, such as "building" or "non-building," without distinguishing between individual objects, while instance segmentation not only classifies pixels but also differentiates between multiple instances of the same category, enabling the delineation of individual building footprints.

Segmentation/DetectionWhy is pixel-level accuracy an insufficient metric for evaluating a building extraction model, and what polygon-level metrics would you use instead?

Pixel-level accuracy is insufficient for building extraction models because it does not account for the spatial arrangement and connectivity of pixels that form buildings. Instead, polygon-level metrics such as overall accuracy, precision, recall, and the F1 score should be used to evaluate how well the extracted building polygons match the reference data.

Spatial JoinsWhat is a spatial join, and how does an intersects join differ from a within join?

A spatial join is a GIS operation that combines attributes from one feature layer to another based on their spatial relationship. An "intersects" join identifies features that overlap or touch each other, while a "within" join identifies features that are completely contained within another feature.

TopologyIn GIS, what does topology mean, and what is a common topological error in a building footprint dataset?

In GIS, topology refers to the arrangement of geographic features in relation to one another, defining how they connect, adjacency, and proximity. A common topological error in a building footprint dataset is "overlaps," where two or more polygons intersect, indicating that the same area is incorrectly assigned to multiple buildings.

Spatial IndexingWhy is a spatial index like an R-tree necessary for efficient geometric queries on large vector datasets?

A spatial index like an R-tree is necessary for efficient geometric queries on large vector datasets because it significantly reduces the search space by organizing spatial data in a hierarchical structure, allowing for faster query response times through spatial partitioning and minimizing the number of data comparisons required.

Coordinate TransformationsWhat is a datum shift, and why can converting between NAD83 and WGS84 introduce a small positional error even though both look like similar coordinate systems?

A datum shift refers to the difference between two geodetic datums, which can cause positional discrepancies when converting coordinates. Even though NAD83 and WGS84 are both widely used datums, converting between them can introduce a small positional error due to their distinct reference points and ellipsoids, leading to potential misalignments of up to a few meters.

LiDARWhat does the classification field of a LAS/LAZ point cloud typically encode, and why is ground classification important before deriving a DTM?

The classification field in a LAS/LAZ point cloud typically encodes the type of surface each point represents, such as ground, vegetation, or building. Ground classification is crucial before deriving a Digital Terrain Model (DTM) because it ensures that only ground points are used, thereby eliminating noise from non-ground features and improving the accuracy and reliability of the DTM.

LiDARWhat is the difference between first-return and last-return LiDAR points, and why does that matter for vegetation versus building surface modeling?

First-return LiDAR points represent the first instance a laser pulse hits a surface, often capturing the canopy top for vegetation, while last-return points denote the final interaction, typically reaching the ground or building surface. This distinction is crucial as first-returns are more suitable for modeling vegetation canopies, whereas last-returns are essential for accurately modeling building surfaces, ensuring precise elevation data for different surface types.

Flagged: imprecise The first part is right, but 'last-returns are essential for accurately modeling building surfaces' overstates it: buildings are opaque single-return surfaces, so first return equals last return there. The first/last distinction only really matters for vegetation, not buildings - this pipeline used first-return + all-non-noise points for the DSM specifically because building rooftops don't need last-return filtering.
LiDAR DensityWhat does 'points per square meter' mean for a LiDAR dataset, and how does point density affect the minimum building size that can be reliably extracted?

'Points per square meter' in a LiDAR dataset refers to the number of laser pulses that hit a specific area, typically measured in square meters. Higher point density increases the likelihood of capturing detailed surface information, allowing for the reliable extraction of smaller buildings by ensuring sufficient data points to accurately delineate their dimensions.

Multimodal FusionWhy might combining SAR or LiDAR height data with optical spectral data outperform either data source alone for building extraction?

Combining SAR or LiDAR height data with optical spectral data enhances building extraction by leveraging the strengths of each: SAR and LiDAR provide accurate elevation and structural information, while optical data offers detailed surface texture and material properties. This synergy allows for more precise differentiation between buildings and other land cover types.

Accuracy AssessmentWhat is Intersection over Union (IoU), and what IoU threshold is commonly used to consider an object detection a true positive?

Intersection over Union (IoU) is a metric used to evaluate the accuracy of object detection models by measuring the overlap between the predicted bounding box and the ground truth bounding box. A commonly used IoU threshold to consider an object detection a true positive is 0.5, meaning at least 50% overlap is required.

Accuracy AssessmentWhat is the difference between a spatial (geographic block) train/test split and a random pixel split for evaluating a raster classifier, and why does it matter?

A spatial train/test split groups contiguous geographic blocks for training and testing, preserving spatial autocorrelation, which is crucial for capturing local patterns and dependencies in the data. In contrast, a random pixel split may include spatially correlated pixels in both training and test sets, potentially leading to overoptimistic accuracy estimates. This matters because it affects the generalizability of the classifier to unseen areas.

Map ProjectionsWhy is UTM (Universal Transverse Mercator) commonly used as the working coordinate system for local-area raster analysis instead of a global geographic CRS?

UTM is commonly used for local-area raster analysis because it minimizes distortion within a limited region, providing high accuracy for measurements and spatial analysis. Unlike global geographic CRS, UTM offers consistent scale and shape across smaller areas, which is crucial for precise local-scale applications.

Data FormatsWhat is a GeoJSON, and what is one limitation of GeoJSON for very large vector datasets?

GeoJSON is an open standard format for encoding a variety of geographic data structures in JSON, facilitating the use of vector data in web applications. One limitation of GeoJSON for very large vector datasets is its lack of support for efficient spatial indexing, which can result in slow query performance and high memory usage when handling extensive spatial data.

Change DetectionHow would you detect new construction between two time periods using a combination of optical imagery and LiDAR-derived nDSM?

To detect new construction between two time periods, I would use multi-temporal optical imagery to identify changes in land cover and land use, then overlay the LiDAR-derived normalized Digital Surface Model (nDSM) to detect height differences indicative of new structures. This combination allows for accurate identification and verification of new constructions.

Slope/TerrainHow is terrain slope typically derived from a DTM, and what is one common algorithm used for this?

Terrain slope is typically derived from a Digital Terrain Model (DTM) by calculating the rate of change in elevation over a specified area. One common algorithm used for this is the finite difference method, which estimates slope by analyzing the elevation differences between a central cell and its neighboring cells.

SAR Building ExtractionWhy does building extraction from SAR imagery (like SpaceNet 6) pose additional challenges compared to optical imagery, such as layover and shadow effects?

Building extraction from SAR imagery poses additional challenges due to layover and shadow effects, where layover occurs when topography causes distant objects to appear closer, and shadows result from obstruction by buildings, both of which can distort the radar signal and complicate accurate building delineation. These effects are less prevalent in optical imagery, which relies on visible light and is less influenced by such geometric distortions.

Data sources and stack

USGS 3DEP LiDARPublic LAZ tile, no-sign-request S3/HTTPS, read with laspy. Ground-classified points to a DTM, all non-noise returns to a DSM.
Sentinel-2 L2ASearched via the free Element84 STAC API, red/NIR/blue bands windowed-read straight off the public COGs with rasterio, no download.
OpenStreetMapBuilding footprints via a single Overpass API query, used as ground truth for both training labels and evaluation.
scikit-learnRandomForestClassifier on 4 engineered features, spatially held out (not a random pixel split) for evaluation.
rasterio / geopandas / shapelyGrid alignment, polygon vectorization, and the polygon-level GIS metrics.
AWS Bedrock (Amazon Nova Lite)eu-west-1 cross-region inference profile. Benchmark Q&A and grounded error explanations, ~30 short prompts, well under a cent total.

Full pipeline, methodology, and limitations write-up: github.com/fborbon/gis-lidar. Longer narrative: Field Notes write-up.