How to register blurry IR to sharp RGB in repeating scenes?

Current Code Attempt (Python + OpenCV)

import cv2 import numpy as np # Load images rgb = cv2.imread("rgb.png") ir = cv2.imread("ir.png", cv2.IMREAD_GRAYSCALE) # Convert RGB to grayscale for matching rgb_gray = cv2.cvtColor(rgb, cv2.COLOR_BGR2GRAY) # Feature detector sift = cv2.SIFT_create() # Detect keypoints and descriptors kp1, des1 = sift.detectAndCompute(ir, None) kp2, des2 = sift.detectAndCompute(rgb_gray, None) # FLANN matcher FLANN_INDEX_KDTREE = 1 index_params = dict(algorithm=FLANN_INDEX_KDTREE, trees=5) search_params = dict(checks=50) flann = cv2.FlannBasedMatcher(index_params, search_params) matches = flann.knnMatch(des1, des2, k=2) # Lowe's ratio test good_matches = [] for m, n in matches: if m.distance < 0.75 * n.distance: good_matches.append(m) print(f"Good matches found: {len(good_matches)}") # Homography (fails often due to wrong matches) if len(good_matches) > 10: src_pts = np.float32([kp1[m.queryIdx].pt for m in good_matches]).reshape(-1, 1, 2) dst_pts = np.float32([kp2[m.trainIdx].pt for m in good_matches]).reshape(-1, 1, 2) H, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 5.0) aligned_ir = cv2.warpPerspective(ir, H, (rgb.shape[1], rgb.shape[0])) else: print("Not enough reliable matches")Problem

Repeating structures cause ambiguous matches

IR image blur reduces feature quality

RANSAC still fails to find a stable homography in many cases

Registering a blurry IR image to a sharp RGB image in repetitive scenes is fundamentally a cross-modal + low-texture + ambiguity problem, so classical feature matching alone (SIFT/ORB + RANSAC) often fails.

A more robust approach is to combine multi-stage alignment + modality-invariant features:

1. Preprocessing (Critical for IR stability)

Apply CLAHE (Contrast Limited Adaptive Histogram Equalization) on IR image
Use edge enhancement (Sobel / Laplacian) to reduce blur impact
Optionally denoise using bilateral filter instead of Gaussian blur

2. Use Modality-Invariant Features

Instead of raw SIFT:

Dense feature matching (SuperPoint / SuperGlue)
D2-Net or R2D2 (more stable in repetitive patterns)
Or edge-based descriptors (HOG / gradient maps)

These reduce dependence on texture uniqueness, which is the main failure point in repetitive structures.

3. Add Structural Constraints (Very Important)

For repeating scenes:

Use keypoint clustering + spatial consistency filtering
Enforce global geometric constraints (e.g., homography or affine consistency checks)
Use RANSAC with stricter inlier thresholds + MAGSAC++ if available

4. Cross-Modal Registration Strategy

Best-performing pipeline in practice:

Convert both IR and RGB → edge maps
Match on edge representations instead of intensity
Then refine alignment using:
- ECC (Enhanced Correlation Coefficient maximization in OpenCV)
- Or optical flow refinement (Farneback / RAFT)