Publications

2023

1. ISRF

   Interactive Segmentation of Radiance Fields

   Rahul Goel\^*, Dhawal Sirikonda\^*, Saurabh Saini , and 1 more author

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023

   Abs Website

   Radiance Fields (RF) are popular to represent casually-captured scenes for new view generation and have been used for applications beyond it. Understanding and manipulating scenes represented as RFs have to naturally follow to facilitate mixed reality on personal spaces. Semantic segmentation of objects in the 3D scene is an important step for that. Prior segmentation efforts using feature distillation show promise but don’t scale to complex objects with diverse appearance. We present a framework to interactively segment objects with fine structure. Nearest neighbor feature matching identifies high-confidence regions of the objects using distilled features. Bilateral filtering in a joint spatio-semantic space grows the region to recover accurate segmentation. We show state-of-the-art results of segmenting objects from RFs and compositing them to another scene, changing appearance, etc., moving closer to rich scene manipulation and understanding.

2022

1. PRTonAnySurf

   Real-Time Rendering of Arbitrary Surface Geometries using Learnt Transfer

   Sirikonda Dhawal, Aakash KT, and P.J. Narayanan

   In Proceedings of the Thirteenth Indian Conference on Computer Vision, Graphics and Image Processing , 2022

   Abs Website

   Precomputed Radiance Transfer (PRT) is widely used for real-time photorealistic effects. PRT disentangles the rendering equation into transfer and lighting, enabling their precomputation. Transfer accounts for the cosine-weighted visibility of points in the scene while lighting for emitted radiance from the environment. Prior art stored precomputed transfer in a tabulated manner, either in vertex or texture space. These values are fetched with interpolation at each point for shading. Vertex space methods require densely tessellated mesh vertices for high quality images. Texture space methods require non-overlapping and area-preserving UV mapping to be available. They also require a high-resolution texture to avoid rendering artifacts. In this paper, we propose a compact transfer representation that is learnt directly on scene geometry points. Specifically, we train a small multi-layer perceptron (MLP) to predict the transfer at sampled surface points. Our approach is most beneficial where inherent mesh storage structure and natural UV mapping are not available, such as Implicit Surfaces as it learns the transfer values directly on the surface. We demonstrate real-time, photorealistic renderings of diffuse and glossy materials on SDF geometries with PRT using our approach.

2. PRTonSDF

   Learnt Transfer for Surface Geometries

   Sirikonda Dhawal, Aakash KT, and P.J. Narayanan

   2022

   Website
3. StyleTRF

   StyleTRF: Stylizing Tensorial Radiance Fields

   Rahul Goel^*, Sirikonda* Dhawal, Saurabh Saini , and 1 more author

   In Proceedings of the Thirteenth Indian Conference on Computer Vision, Graphics and Image Processing , 2022

   Abs Website

   Stylized view generation of scenes captured casually using a camera has received much attention recently. The geometry and appearance of the scene are typically captured as neural point sets or neural radiance fields in the previous work. An image stylization method is used to stylize the captured appearance by training its network jointly or iteratively with the structure capture network. The state-of-the-art SNeRF method trains the NeRF and stylization network in an alternating manner. These methods have high training time and require joint optimization. In this work, we present StyleTRF, a compact, quick-to-optimize strategy for stylized view generation using TensoRF. The appearance part is fine-tuned using sparse stylized priors of a few views rendered using the TensoRF representation for a few iterations. Our method thus effectively decouples style-adaption from view capture and is much faster than the previous methods. We show state-of-the-art results on several scenes used for this purpose.

4. PRTT

   PRTT: Precomputed Radiance Transfer Textures

   Sirikonda Dhawal, Aakash KT, and P.J. Narayanan

   2022

   Abs Website

   Precomputed Radiance Transfer (PRT) can achieve high quality renders of glossy materials at real-time framerates. PRT involves precomputing a k-dimensional transfer vector of Spherical Harmonic (SH) coefficients at specific points for a scene. Most prior art precomputes transfer at vertices of the mesh and interpolates color for interior points. They require finer mesh tessellations for high quality renderings. In this paper, we explore and present the use of textures for storing transfer. Using transfer textures decouples mesh resolution from transfer storage and sampling which is useful especially for glossy renders. We further demonstrate glossy inter-reflections by precomputing additional textures. We thoroughly discuss practical aspects of transfer textures and analyze their performance in real-time rendering applications. We show equivalent or higher render quality and FPS and demonstrate results on several challenging scenes.

5. TransferTextures

   Transfer Textures for Fast Precomputed Radiance Transfer

   Sirikonda Dhawal, Aakash Kt, and P. J. Narayanan

   In Eurographics 2022 - Posters , 2022

2021

1. NeuralViewSyn

   Neural View Synthesis with Appearance Editing from Unstructured Images

   Pulkit Gera, Aakash K T, Dhawal Sirikonda , and 1 more author

   In Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image Processing , Jodhpur, India, 2021

   Abs

   We present a neural rendering framework for simultaneous view synthesis and appearance editing of a scene with known environmental illumination captured using a mobile camera. Existing approaches either achieve view synthesis alone or view synthesis along with relighting, without control over the scene’s appearance. Our approach explicitly disentangles the appearance and learns a lighting representation that is independent of it. Specifically, we jointly learn the scene appearance and a lighting-only representation of the scene. Such disentanglement allows our approach to generalize to arbitrary changes in appearance while performing view synthesis. We show results of editing the appearance of real scenes in interesting and non-trivial ways. The performance of our view synthesis approach is on par with state-of-the-art approaches on both real and synthetic data.

2. AppearaceEdit

   Appearance Editing with Free-viewpoint Neural Rendering

   Pulkit Gera, Aakash KT, Sirikonda Dhawal , and 2 more authors

   2021

   Abs Website

   We present a neural rendering framework for simultaneous view synthesis and appearance editing of a scene from multi-view images captured under known environment illumination. Existing approaches either achieve view synthesis alone or view synthesis along with relighting, without direct control over the scene’s appearance. Our approach explicitly disentangles the appearance and learns a lighting representation that is independent of it. Specifically, we independently estimate the BRDF and use it to learn a lighting-only representation of the scene. Such disentanglement allows our approach to generalize to arbitrary changes in appearance while performing view synthesis. We show results of editing the appearance of a real scene, demonstrating that our approach produces plausible appearance editing. The performance of our view synthesis approach is demonstrated to be at par with state-of-the-art approaches on both real and synthetic data.