From Passive Perception to Active Interaction: A Survey of Affordance Learning for Embodied AI

1. Introduction

Embodied intelligence studies agents that perceive, reason, and act in physical environments. Unlike image classifiers or passive scene understanding systems, embodied agents must use perception to support interaction: they need to identify task-relevant objects and actionable parts, anticipate the effects of actions, and assess whether those effects are achievable with their bodies and skills. This makes action-oriented perception a central requirement for embodied AI.

Affordance provides a way to ground perception in possible interactions. While semantic categories describe what an object is, affordances characterize what an agent can do with it. Recognizing a mug, drawer, or chair is only a first step: an agent must further infer how the mug can be grasped or used for pouring, where the drawer should be manipulated, and whether the chair should serve as a seat, provide support, or be treated as an obstacle. Such action possibilities are not fixed object properties but arise from relations among objects, tasks, environments, and embodiment. For example, a cup may be grasped by its handle or body depending on the intended action and end-effector; an occupied tabletop may no longer afford placement; and a tool may become useful only when paired with an appropriate target.

In embodied AI, affordance learning infers how visual, geometric, semantic, and physical cues enable interaction with the environment. Beyond recognizing objects or localizing regions, it identifies which elements are actionable, what interactions they support, why they are relevant to a task, and how they constrain feasible behavior. Affordance therefore provides a functional intermediate representation connecting perception, task-level reasoning, and executable control. Early learning-based work largely treated affordance learning as visual prediction, recognizing functional categories or localizing action regions in RGB observations using labels or masks [1,2,3,4]. Although effective for identifying visible functional regions, these formulations were often tied to predefined labels and offered limited guidance for physical execution. With the rise of embodied AI, the field has expanded toward task-oriented and action-aware understanding. Recent methods ground affordances in 3D space [5,6], derive contact and motion cues from human interactions and egocentric videos [7,8,9,10], use language and multimodal models to identify task-relevant objects and parts [11,12,13], and incorporate affordances into robot planning and policy learning [14,15,16,17,18]. Collectively, these developments shift affordance learning from appearance-driven recognition toward interactions that are contextualized by task goals and physically executable by embodied agents.

Several surveys have examined different aspects of the affordance literature. Visual affordance surveys examine function understanding, recognition, segmentation, and reproducibility [19,20,21], while robotic reviews cover cognitive foundations, developmental learning, affordance relations, and reinforcement-learning formulations [22,23,24,25]. These works provide valuable foundations, yet important gaps remain in the context of modern embodied AI. Perception-centered surveys provide limited coverage of how affordances are selected and interpreted under language instructions, implicit intents, tool-use reasoning, and feasibility-aware planning [11,13,26,27]. Conversely, robotics-centered reviews do not fully capture recent advances driven by foundation models, large-scale robot learning, vision-language-action models, and video-action models [28,29,30,31,32,33,34,35]. What remains missing is a unified view of affordance learning that explains how these threads jointly connect perception, reasoning, and action in support of embodied intelligence.

To address this gap, this survey examines affordance learning from a comprehensive embodied intelligence perspective. As illustrated in Figure 1, we organize the literature around three core roles: affordance perception, affordance reasoning, and affordance-guided action. Together, they describe how an embodied agent identifies actionable opportunities, reasons about their task relevance, and turns them into executable behavior. This organization also defines the survey scope: we include studies in which affordance plays an explicit role in problem formulation, supervision, model output, planning process, or evaluation. General embodied AI methods are considered only when they explicitly ground, utilize, or evaluate action possibilities. This criterion accommodates recent developments in large language model (LLM)- and multimodal large language model (MLLM)-based reasoning, agentic systems, and vision-language-action models while maintaining the focus on affordance learning.

We first introduce the background in Section 2, covering definition, unified problem formulation, and evolution of affordance learning. Our technical review then begins with affordance perception, which asks what is actionable and where it is located. Section 3 covers visual, spatial, interaction-driven, and generalizable affordance grounding at both object and scene levels. Second, affordance reasoning asks why and when an affordance is relevant. Section 4 examines how task-relevant objects, parts, and interactions are selected through relation-centric modeling, the interpretation of language instructions or implicit intentions, and agentic workflows. Third, affordance-guided action asks how affordances are translated into executable behavior. Section 5 reviews hierarchical affordance-to-action pipelines and affordance-integrated policy learning, in which affordance provides functional structure for action generation. The three roles form a non-exclusive framework: each method is discussed according to its primary contribution, while cross-role methods may be revisited from complementary perspectives. Finally, Section 6 summarizes datasets, benchmarks, and evaluation protocols, while Section 7 discusses open challenges and future directions.

In summary, this survey makes the following contributions:

• We present a unified framework for affordance learning in embodied AI, organized around perception, reasoning, and affordance-guided action. It clarifies how actionable opportunities are grounded, contextualized, and translated into executable behavior.
• Within this framework, we develop fine-grained taxonomies and compare representative methods along consistent dimensions of inputs, representations, supervision, generalization, and execution.
• We organize representative datasets and evaluation protocols, and outline open challenges and future directions, providing the community with a structured resource for future research in embodied affordance learning.

2. Background

2.1. Definition and Problem Formulation

The term affordance, introduced by Gibson [36], refers to action possibilities that the environment offers to an agent. Early research on active perception and interaction laid foundations for understanding object functionality [37,38]. Subsequent approaches learned tool affordances [39] and visual cues for predicting possible interactions [40] and recognizing functional object classes [41]. Building on these efforts, affordance learning has expanded across computer vision, robotics, and embodied AI, encompassing tasks such as affordance segmentation, contact prediction, action-region grounding, and affordance-conditioned robot control [2,4,10,42,43,44].

Different problems vary in supervision, input modalities, output representations, and downstream uses. To provide a common notation, we formulate affordance learning as the problem of estimating structured affordance information from observations, task information, and auxiliary evidence. Let o denote an observation, such as an RGB or RGB-D image, video, or 3D representation; l, language or task information, such as an affordance word, action query, or natural-language instruction; and c, contextual evidence, such as human–object interactions, demonstrations, support examples, or retrieved memories. An affordance model $f_{\theta }$ can then be formulated as

$$\mathcal{F}=f_{\theta }(o,l,c)=(a,r,\eta ),$$

(1)

where the structured affordance representation $\mathcal{F}$ contains three complementary components. The action or affordance label a describes what action is possible, such as grasping, opening, or pulling; the grounded region r specifies where the action can be performed, such as an object part, mask, heatmap, keypoint, contact point, or 3D spatial location; and the execution-related specification $\eta$ captures how the affordance can be used for embodied action, such as a 6-DoF pose, action score, motion direction, or post-contact trajectory. In practice, inputs l and c are optional: purely visual methods may use only o, language-conditioned methods use $(o,l)$, and transfer-based methods further condition on c. Depending on the problem setting, some output components may also be omitted or left implicit. For example, visual affordance segmentation mainly predicts affordance labels a along with their grounded regions r, while robot manipulation methods often further model $\eta$ to enable action execution.

This formulation connects the main settings reviewed in this survey. Visual and spatial perception primarily focus on grounding relevant regions r, while interaction-driven methods leverage c to transfer knowledge from human interactions or demonstrations [7,42,45]. Affordance reasoning incorporates task information l to identify task-relevant objects, parts, or relationships [11,13,46], and affordance-guided action supplies $\mathcal{F}$ to a planner, policy, or controller [15,16,47].

2.2. Evolution of Affordance Learning Paradigms

Affordance learning has evolved from supervised visual prediction of action regions toward action-grounded embodied learning. Figure 2 traces representative advances over the past decade, while the following discussion summarizes five interrelated paradigm shifts in supervision, grounding space, output representation, generalization, and integration with action.

From dense labels to diverse evidence. Early methods commonly relied on dense manually annotated supervision, such as pixel-level affordance masks over objects and their parts [2,4]. This setting provides clear training targets and evaluation, but it is costly to scale because annotators must reason about potential object uses and functional interactions rather than merely label visible boundaries. Alternative approaches draw on image-level labels, sparse points, human–object interactions, and model-generated pseudo-labels or priors [7,10,42,48,49]. Physical simulation provides further evidence by generating interaction labels for learning [50] or directly evaluating candidate interactions to infer affordances and functional poses [51,52]. These approaches reduce reliance on dense manual annotation, although their reliability depends on data quality, model priors, and simulation fidelity.

From 2D image plane to 3D metric space. Classical affordance prediction often represented action possibilities on the image plane, such as 2D masks, heatmaps, keypoints, or part-level regions. Although these predictions identify visible functional areas, they remain view-dependent and do not directly encode metric geometry. Embodied agents, however, act in 3D space and require estimates of surface position, orientation, scale, reachability, navigability, and surrounding constraints in the physical workspace. This need has driven affordance grounding from 2D image regions toward 3D physical space, as illustrated in Figure ??(a). By grounding affordances directly in metric space, 3D-based approaches associate functional regions with physical surfaces, integrate evidence across viewpoints, and support feasibility assessment and action planning.

From static regions to actionable dynamics. Early visual methods often represent affordances as binary or multi-class masks, indicating what action a region supports and where it applies. Later work additionally models how interaction should unfold: contact points identify candidate interaction locations [9,53], motion parameters describe part movement [6,54], poses specify how an agent’s hand or end-effector should be configured for interaction [43,55], and post-contact trajectories describe behavior after contact [10,56]. The appropriate representation depends on its intended downstream use: masks and heatmaps are suitable for perceptual grounding, contact points and actionability scores support action selection, and motion directions, poses, or trajectories connect affordance with planning and control.

From closed-set recognition to open-set generalization. Classical supervised settings typically assume predefined objects, affordance labels, and action categories, with substantial overlap between training and test distributions. This closed-set assumption is restrictive in real environments containing long-tail objects, unseen object–action combinations, and instructions that do not match a fixed affordance vocabulary. As shown in Figure 3(b), the number of open-set affordance methods has increased substantially in recent years. Open-set methods address this limitation by using language, limited demonstrations, and foundation models to generalize beyond predefined labels. Open-vocabulary 3D approaches align language with point features to recognize unseen affordances and object categories [57,58,59], while vision-language grounding exploits instructions and multimodal priors to address novel objects, part queries, and interaction goals [13,46,60]. This direction reduces the need for exhaustive affordance annotation when embodied agents encounter new tasks and objects.

From modular pipelines to integrated policy learning. In robotics, affordance has often been used in a hierarchical pipeline that predicts affordance regions, plans a grasp or motion, and executes the resulting action. Such modularity is interpretable and can explicitly incorporate physical feasibility and robot kinematics, but hand-designed representations, planner interfaces, or primitives may limit transfer across tasks, objects, and embodiments. Recent systems increasingly integrate affordance into policy learning and action prediction: it may condition diffusion or vision-language-action policies explicitly [18,61,62], appear as an implicit intermediate representation [47,63], and provide auxiliary supervision or an optimization signal [64,65]. This integration aligns affordance learning more closely with the robot’s embodiment, action space, and execution objective, shifting it from a standalone perceptual output toward a functional component of embodied decision making.

Together, these shifts show how affordance learning is becoming more scalable, task-conditioned, spatially grounded, and action-relevant. For embodied agents, these developments support identifying interaction opportunities, contextualizing them under task goals, and grounding them in feasible actions. Figure 3(c) further shows rapid expansion of affordance research, with an increasing number of studies exploring embodied settings and real-world robot integration. Nevertheless, affordance grounding remains dominant: perception- and reasoning-oriented studies account for 68% of the 193 surveyed works, indicating that connecting affordance understanding with physical execution remains an open challenge.

3. Affordance Perception

Affordance perception addresses the grounding problem in affordance learning: given an observation, it identifies potential action opportunities and where their corresponding functional entities are located. We categorize a method as perception when its main contribution is to detect, segment, score, or spatially localize affordances through masks, heatmaps, keypoints, point-level scores, or 3D regions. Such grounding can be conditioned on action verbs, affordance concepts, or task queries, and can draw supervision or priors from observed human behavior or foundation models.

We organize affordance perception through four complementary and non-exclusive perspectives, as shown in Figure 4. The first two concern the grounding space. Visual affordance perception predicts functional regions on the 2D image plane, while spatial affordance perception grounds affordances in metric 3D representations of objects or scenes. Interaction-driven affordance perception derives supervision or contextual cues from human–object interaction images, videos, or task demonstrations, whereas generalizable affordance perception studies transfer to novel objects, unseen affordance labels, open-vocabulary queries, or unfamiliar interaction contexts. Table 1 summarizes representative methods across these perspectives, organized by setting, input modality, output form, supervision regime, and open-set generalization ability.

3.1. Visual Affordance Perception

Visual affordance perception studies affordance grounding on the 2D image plane. Given an RGB or RGB-D observation, it localizes image regions associated with an interaction, where the interaction may be predicted from a predefined label set or specified by a task or query. Existing methods differ primarily in prediction scope: object-centric grounding identifies functional regions of individual objects, while scene-level grounding situates affordances within broader environmental layouts. Together, they provide fine-grained object-part cues and context-aware interaction opportunities for downstream action.

Early efforts mainly investigated object-centric affordance grounding from 2D visual observations. Nguyen et al. [2,3] develop convolutional approaches that first combine appearance with depth-derived cues and later decouple object localization from pixel-level affordance refinement. AffordanceNet [4] extends instance segmentation to jointly detect objects and predict fine-grained affordance masks in an end-to-end framework. Complementary work examines object-part structure and develops manipulation-oriented representations. Part-based analysis highlights the role of geometric object parts in tool functionality [1], while part–affordance coupling [90] and relationship-aware detection [91] model associations between parts and affordances and dependencies among affordances. Efficient attention-based segmentation [92] and boundary-preserving detection [93] further refine functional regions using contextual and boundary cues. Beyond region masks, AffKp [67] encodes manipulation positions and directions as structured 2D keypoints, while AffContext [94] uses predicted object affordances as intermediate cues for manipulation-oriented scene reasoning. To improve scalability, AffordanceLearn [95] reduces reliance on dense manual annotation through synthetic training data. Further efforts expand dataset scale and viewpoint coverage through a large-scale multi-view RGB-D affordance benchmark [96] and develop a hierarchical vision transformer to capture affordance cues across viewpoints [97].

In parallel, scene-level perception broadens the prediction target from isolated objects to action-relevant regions in full visual contexts. Early approaches combine appearance, semantic structure, and spatial layout as complementary cues. Roy and Todorovic [98] infer depth, surface normals, and semantic scene structure from a single RGB image and combine these multi-scale cues for affordance segmentation. Luddecke and Worgotter [99] derive dense affordance supervision by mapping object- and part-level semantic annotations to affordance labels. A subsequent extension [100] expands this label-transfer formulation to a larger set of robot-relevant affordances and evaluates it in robotic settings. Qian and Fouhey [44] jointly predict object properties, action types, and affordance heatmaps from a single RGB image and query points. Panoramic methods further extend scene-level prediction to whole-environment observations. PanoAffordanceNet [101] studies holistic affordance prediction in 360-degree indoor scenes and introduces 360-AGD to capture affordance cues under equirectangular distortion, semantic dispersion, and cross-scale layout variation. PAP [102] adopts a coarse-to-fine pipeline to recover affordance masks from ultra-high-resolution panoramic imagery.

Orthogonal to the object–scene distinction, existing methods also differ in supervision strength. Most object-centric and scene-level approaches rely on dense pixel-level affordance annotations, which are costly because annotators must infer potential interactions rather than simply delineate visible object boundaries. Weakly supervised affordance perception reduces this burden by learning dense functional predictions from sparse spatial annotations or coarse interaction labels. One line replaces dense masks with sparse point supervision. Sawatzky et al. [48], Sawatzky and Gall [103] perform affordance segmentation from a limited number of annotated points, while STRAP [104] further propagates sparse supervision through label- and spatial-affinity constraints. A complementary line exploits action labels from human–object interaction data. Interaction Hotspots [42] learns from action-labeled human–object interaction videos, whereas Cross-view-AG [45] transfers affordance cues from exocentric interaction images to target objects. Both localize affordances from coarse action labels using CAM-style activation maps [105].

Overall, visual affordance perception grounds interaction opportunities in 2D images. Object-centric methods identify functional parts and structured interaction cues on individual objects, scene-level methods situate affordances within broader environmental contexts, and weakly supervised methods reduce reliance on dense pixel-level annotations. These settings establish image-plane affordance grounding as a fundamental component of embodied perception by identifying where interactions may be supported.

3.2. Spatial Affordance Perception

Spatial affordance perception grounds affordances in metric 3D representations, such as point clouds, meshes, and 3D Gaussian representations. Its key distinction from RGB-D image prediction lies in the output space. While depth provides geometric cues, spatial affordance perception directly associates affordance predictions with object surfaces or scene locations in 3D coordinates. This grounding makes contact locations, surface geometry, orientations, and spatial relations explicit, providing a geometric interface for downstream action. Existing methods span two main settings: 3D object grounding, which localizes affordances on object geometries, and 3D scene grounding, which anchors affordances to locations or functional elements in full 3D environments.

Early work on spatial affordance perception primarily addressed object-level grounding on point-cloud representations. Kokic et al. [106] predict part-level affordance scores from complete or partial object point clouds, jointly estimate object class and orientation, and convert these predictions into contact-location and approach-direction constraints for task-specific grasp planning. 3D AffordanceNet [5] later introduces the first large-scale 3D affordance benchmark with dense probabilistic point-wise annotations propagated from sparse affordance keypoints, supporting full-shape, partial-view, and rotation-invariant affordance estimation. Subsequent methods broaden 3D object grounding by reducing annotation requirements, improving geometric robustness, and producing more manipulation-oriented outputs. O2O-Afford [50] uses simulation to generate annotation-free supervision for object–object interaction regions, while PartAfford [69] discovers part-level affordances from object-level labels through reconstruction and primitive-based regularization. LGAfford-Net [107] emphasizes local surface geometry, prototype-based detection [108] explains predictions through learned prototypes, and Affostruction [85] reconstructs complete geometry from partial RGB-D observations to ground affordances on both observed and unobserved surfaces. To make these predictions more directly actionable, ACF [109] estimates 3D keypoints and directed axes of functional parts; FAKP-Net [77] encodes execution position, direction, and extent as 3D keypoint quadruplets; and ComA [110] models human–object proximity and orientation distributions.

At the scene level, spatial affordance perception anchors interaction opportunities to persistent 3D environments rather than isolated object shapes. EPIC-Aff [111] aggregates egocentric interactions into multi-label affordance maps registered in scene coordinates, so that affordances become properties of metric locations and a single location may support several actions. SceneFun3D [6] provides fine-grained functional-element annotations, task descriptions, and motion parameters in real indoor scans, supporting functionality segmentation, task-driven affordance grounding, and 3D motion estimation. IAAO [112] constructs hierarchical feature and label fields on 3D Gaussian primitives and jointly estimates affordances, global transformations, and local articulation parameters. AffordMatcher [86] establishes semantic correspondences between image-based and point-cloud instances, using visual signifiers to identify affordance regions in 3D scenes.

Overall, spatial affordance perception moves affordance grounding from image coordinates to metric object and scene representations. Object-level methods localize functional surfaces and capture manipulation-relevant geometry, whereas scene-level methods associate interaction opportunities with locations and functional elements through 3D environments. Together, these 3D affordance estimates provide a spatial interface for downstream planning and embodied interaction.

3.3. Interaction-Driven Affordance Perception

Humans do not acquire affordance knowledge solely by memorizing object categories or explicitly labeled functional parts. Instead, much of this knowledge emerges from observing how people interact with objects. Through such observations, humans learn to associate specific object regions with particular actions and can readily transfer these expectations to novel instances. This learning mechanism motivates interaction-driven affordance perception, which derives supervision or contextual cues from observed human–object interactions (HOI) rather than relying solely on directly annotated affordance regions. Existing methods mainly pursue this idea using two forms of data: demonstration or human–object interaction videos, which reveal temporal dynamics such as hand motion, object state change, and action trajectories, and human–object interaction images, which provide scalable spatial cues about contact and functional regions.

Video-based methods draw on both exocentric and egocentric observations. Exocentric videos, collected from web demonstrations or third-person recordings, capture human poses, manipulated objects, and broader task context at scale [7,113]. This holistic view supports learning action-conditioned usage patterns, but can obscure fine-grained contact and introduces a viewpoint gap from embodied agents. Egocentric videos [114,115] better align with embodied perception and expose hand dynamics, object-state transitions, and upcoming contact, making them informative about where and how manipulation will occur. However, rapid camera motion, restricted fields of view, and embodiment-specific biases remain challenging. One line of work transfers temporal interaction evidence to static target images. Demo2Vec [7] encodes a product-demonstration video to predict the action and interaction region in a target image of the same object, thereby transferring action-conditioned visual priors from video to a static view. Afformer [116] improves this setting with a fine-grained decoder and MaskAHand pre-training to bridge the video–image discrepancy. These methods transfer action-conditioned cues to static images but still require annotated heatmaps and action labels. Interaction Hotspots [42] instead uses action-labeled HOI videos as weak supervision, transferring features from active interactions to infer action-specific hotspots on inactive object images. Other work mines structured cues from egocentric video: Liu et al. [8] automatically extract hand trajectories and future hotspots and train an object-centric transformer to anticipate interaction; VRB [10] learns contact points and post-contact trajectories as a reusable representation for robot learning; Yoshida et al. [117] add textual annotations and rotational motion; and Li et al. [118] derive graspable and functional points as segmentation-mask supervision for broader data collection. More recently, Chen et al. [119] learn action, dynamics, and value models from human videos, capturing task-conditioned predictions of how actions transform the scene and contribute to task completion. For a broader review of human-video supervision for robotics, we refer to Feng et al. [120].

In parallel, a larger body of work learns from human–object interaction images, which provide less temporal information but are easier to collect at scale. Cross-view-AG [45] first formulates exocentric-to-egocentric affordance grounding, using interaction images as low-cost supervision without pixel-level affordance masks. This makes HOI images practical affordance supervision, although human occlusion and action ambiguity complicate the transfer. Subsequent methods extract more reliable object-centric and part-level signals: Luo et al. [121] model relations between human body parts and object regions, whereas LOCATE [71] clusters interaction-region features into prototypes and selects the object-part prototype for target grounding. Later methods enrich this evidence with textual action descriptions [122], depth and semantic prompts [123], or part-level semantic priors [124]. Other approaches improve representation learning and transfer. INTRA [78] learns interaction-aware representations without paired ego–exo images; Reasoning Mamba [80] models local and global visual relations through hypergraph-guided state-space learning; LoopTrans [83] uses a closed loop to refine exocentric knowledge; and Moon et al. [125] introduce prototypical and pixel-level contrastive learning to capture affordance-relevant cues at both the object and part levels.

Interaction evidence can also condition affordance grounding in 3D, transferring human-centered visual cues to metric object representations. IAGNet [72] aligns a 2D interaction image with point-cloud object regions for point-level prediction, while LEMON [74] jointly predicts object affordances, human contact, and their 3D spatial relations from an interaction image and paired geometries. MIFAG [126] fuses invariant patterns from multiple interaction images with point-cloud features, and VideoAfford [127] extracts temporal priors from HOI videos using a multimodal large language model. Beyond individual objects, observed interactions reveal how scene regions are used. Ego-Topo [66] organizes egocentric video into activity-centric zones, while EPIC-Aff [111] projects first-person interactions into metric maps where multiple affordances may coexist. Ego-SAG [128] aligns egocentric interaction video with a paired 3D scene point cloud to predict point-level affordance masks. FunREC [84] goes beyond grounding by reconstructing simulation-ready functional scenes and recovering articulated geometry, motion, and kinematics from RGB-D video, while AffordMatcher [86] uses HOI images as visual signifiers for locating functional object parts in 3D scenes.

Overall, interaction-driven affordance perception converts observed behavior into supervision and contextual priors for affordance grounding. Videos provide temporal contact and motion cues, static HOI images offer scalable cross-view evidence, and cross-modal methods transfer these cues to 3D objects and environments. Together, these approaches make human behavior a practical source of affordance supervision across representations.

3.4. Generalizable Affordance Perception

Most affordance-perception methods assume predefined affordance categories, object distributions, and task settings. Embodied agents, however, must handle objects absent from training, unseen action expressions, and unfamiliar interaction contexts. Generalizable affordance perception addresses these cases without requiring dense annotations for every object–affordance combination. Existing work mainly follows two complementary paradigms: example-based transfer, which uses sparse support observations, annotated samples, or reference interactions as task-specific evidence; and open-set grounding, which uses language and foundation-model knowledge to move beyond fixed label spaces.

Example-based transfer covers support-conditioned grounding and learning or adaptation from one or a few examples. In support-conditioned grounding, a reference observation specifies the affordance to be transferred to a target. One-Shot Affordance Detection [68,129] uses an HOI support image to indicate an action purpose and segments complete target objects sharing that affordance. AffCorrs [70] instead performs training-free part-level transfer by matching an annotated reference region with corresponding parts in a target scene through cyclic correspondence. Other methods use limited labeled examples as sparse training supervision. OOAL [76] is a representative approach that learns from one labeled image per base object category and exploits foundation-model priors for zero-shot grounding of novel object and affordance classes. OS-AGDO [130] extends one-shot grounding to deformable objects by combining semantic and geometric cues to accommodate changes in shape, appearance, and configuration. Example-based transfer also applies to 3D and interactive settings. Where2Explore [73] adapts to unseen articulated-object categories through a few exploratory interactions, transferring known affordances via local geometric similarity while selecting informative regions to explore. O3Afford [131] learns bidirectional object-to-object affordance maps from one annotated object pair per affordance type by combining pretrained visual semantics with point-cloud geometry. AffoRo-GS [132] grounds verb-conditioned affordances in 3D Gaussian scene representations from multi-view observations using only one affordance-labeled image per scene.

Open-set grounding uses semantic queries and pretrained knowledge to generalize beyond fixed vocabularies and training distributions. In 2D, MaskPrompt [79] refines object-shape masks to enhance visual features and construct mask-focused textual prompts for open-vocabulary affordance segmentation. AffordanceSAM [89] adapts a text-promptable SAM backbone through an affordance module and coarse-to-fine supervised training, supporting transfer to novel objects and action queries. UAD [49] uses large vision and vision-language models to generate instruction–affordance pairs automatically from rendered objects, then distills them into a lightweight task-conditioned predictor without manual affordance annotations. In 3D, several methods directly align language with point-level geometry. OpenAD [57] jointly embeds affordance text and point features for zero-shot detection of unseen labels, while Van Vo et al. [59] improve transfer to unseen labels and objects through knowledge distillation and explicit text–point correlation. 3DAPNet [58] further predicts compatible 6-DoF poses together with affordance regions. PointRefer [75] grounds object regions specified by natural-language functional queries, and GLANCE [82] adds intermediate cross-modal connectors and geometric priors to improve generalization to unseen object categories. OVA-Fields [133] extends this paradigm to sparse and noisy RGB-D scene reconstructions, combining multilevel geometry with semantic affordance embeddings to map complex commands to operational parts under weak supervision. Foundation models further improve the scale and robustness of 3D grounding. GEAL [81] transfers pretrained 2D semantic priors to point clouds through Gaussian-splatting-based cross-modal consistency, improving generalization to novel categories and corrupted observations. Affogato [88] orchestrates vision-language and segmentation models to generate open-vocabulary queries and corresponding 3D affordance heatmaps at scale. DAG [87] extracts multi-level affordance priors from a frozen text-to-image diffusion model, combines them with affordance text, and fuses the result with point-cloud features for dense 3D prediction.

Overall, example-based methods specify target functions through sparse visual or interaction evidence, whereas open-set methods use language and pretrained priors to broaden the objects and affordances that can be grounded. Recent LLM- and MLLM-based approaches further draw on semantic and commonsense knowledge to acquire generalization capabilities, but often require instruction understanding, intent inference, and task-relevant affordance selection. We therefore discuss these reasoning aspects in Secs. Section 4.2 and Section 4.3. Together, these directions extend affordance perception to long-tail objects, novel queries, and interaction contexts that cannot be exhaustively annotated in advance.

4. Affordance Reasoning

Affordance reasoning studies how an embodied agent determines which affordances are appropriate based on task requirements, object relations, scene context, agent state, or available skills. It resolves ambiguity among possible interactions through relational structure, contextual evidence, or knowledge from large pretrained models. This distinguishes it from affordance perception, which directly grounds candidate interaction regions from observations. For language-conditioned methods, a query used only to localize a region remains perception, whereas a method that leverages knowledge from LLMs or MLLMs, decomposes instructions, or interprets implicit intent to select the relevant affordance is treated as reasoning.

We organize affordance reasoning into three complementary paradigms according to their primary reasoning mechanism, as summarized in Figure 5. Relation-based affordance reasoning models functional knowledge explicitly through semantic or structural relations among objects, parts, actions, attributes, or scene context. Language-centric affordance reasoning uses LLMs or MLLMs to draw on model knowledge, decompose instructions, or interpret intent when determining the relevant affordance. Agentic affordance reasoning further frames the reasoning process as a structured workflow that coordinates specialized tools, evidence acquisition, and intermediate validation or refinement. Table 2 summarizes representative methods across these paradigms.

4.1. Relation-Based Affordance Reasoning

Relation-based affordance reasoning represents an early route for resolving functional ambiguity before the LLM era. Its central idea is to treat affordance as an explicit relation over a predefined set of objects, parts, attributes, actions, or scene configurations. Rather than inferring affordances from isolated object appearance, these methods use semantic properties, object pairings, and surrounding context. Existing approaches encode such dependencies through probabilistic knowledge bases, learned spatial relations, and structured scene graphs.

One line encodes affordance knowledge through probabilistic logic. Zhu et al. [134] harvest object, attribute, and affordance facts from images and metadata and encode them in a knowledge base modeled with a Markov logic network. The resulting representation supports multiple visual inference tasks within a unified framework, including zero-shot affordance prediction. Ardón et al. [137] apply a related formulation to task-oriented grasping, using semantic attributes and context to infer a probability distribution over multiple grasp affordances. The inferred affordances are then associated with prototypical grasping patches on the object, connecting semantic reasoning with spatial grasp selection.

Other approaches focus on learning functional and spatial relations. Fichtl et al. [135] learn relational affordances from robot interaction and bootstrap new object-pair predictors using previously acquired affordance knowledge, reducing the experience required to learn new relations. Related work models spatial relations themselves as constraints on feasible interactions. Mees et al. [164] learn a metric over pairwise spatial relations that generalizes to novel object shapes and sizes from few examples. Spatial-RelNet [138] predicts pixel-wise placement probabilities conditioned on spatial relations, while Mees and Burgard [165] extend this model with language grounding and ambiguity resolution to compose pick-and-place commands into tabletop manipulation tasks. Graph-based models organize scene entities and their dependencies into explicit structures for contextual affordance inference. Chuang et al. [136] adopt graph neural networks to predict and explain whether an action–object pairing is physically appropriate or socially acceptable in the current situation. VAR-Net [140] combines semantic and visual relations among scene objects through graph attention. AR-Net [139] constructs a manipulation relationship graph from raw visual observations, using object attributes and subgraph context to predict manipulation relations between objects.

Taken together, relation-based methods show that appropriate affordances depend on object relations and scene context rather than isolated object recognition. Their explicit relational structure supports interpretable inference, but predefined entities and relation schemas can limit their flexibility under open-ended instructions and implicit goals.

4.2. Language-Centric Affordance Reasoning

Language-centric affordance reasoning uses language, typically through LLMs or MLLMs, to decide the affordance target most relevant to a task. Unlike language-conditioned affordance perception, where text primarily specifies what region to localize, these methods use model knowledge, instruction decomposition, or intent interpretation to determine which object, part, function, or interaction constraint should be grounded. Early systems grounded language reasoning at the skill level: SayCan [11] combines the linguistic plausibility of candidate skills with their estimated executability, while Grounded Decoding [12] injects observation- and state-grounded scores directly into action generation. Subsequent work extends this principle from discrete skill selection to spatial affordance prediction.

Recent methods follow two main paradigms. The first integrates LLM- or MLLM-based reasoning directly into affordance prediction, adapting model knowledge through instruction tuning, multimodal alignment, distillation, or post-training. In 2D, AffordanceLLM [13] and RoboPoint [60] transfer vision–language knowledge to action-conditioned heatmaps or keypoints. Afford-X [144] explores a compact end-to-end model, and Affordance-R1 and AffordanceGrasp-R1 [155,166] use reward-based post-training to strengthen task-oriented reasoning. These approaches differ in output and adaptation strategy, but share the goal of embedding task interpretation into the spatial predictor rather than treating language as a fixed class label. This direction is supported by instruction-driven segmentation, grasping, and visual-question-answering datasets [141,167,168,169,170,171], which are discussed further in Section 6. Integrated reasoning has also expanded toward geometry-aware prediction. For 3D grounding, 3D-ADLLM [143] couples an LLM with a point-level decoder for free-form reasoning queries, ViSPLA [149] iteratively refines such predictions using geometric feedback, and PAVLM [150] aligns point-cloud features with contextual language embeddings. GREAT [147] further uses affordance chain-of-thought supervision to infer geometric properties and interaction intent from interaction images.

The second paradigm separates semantic reasoning from spatial grounding. An LLM or MLLM first translates instructions into task semantics, functional queries, part hypotheses, or augmented textual representations, while delegating spatial grounding to specialized modules. WorldAfford [46] derives affordance knowledge from free-form instructions, while OVAL-Prompt [142] explicitly queries the functional part before segmentation. AffordGrasp [153] further resolves implicit user goals into a target object and part for grasp generation. In 3D, part-aware grounding [172] and Aff3DFunc [151] align enriched functional text with part-level geometry; Fun3DU [148] reasons about the implied object across views and lifts its masks into a scene; and TASA [156] validates predicted concepts through reverse and fallback queries before segmentation. This decomposition preserves specialized spatial representations and makes intermediate semantics more explicit, although errors in the language-derived hypothesis may propagate to the grounder.

A further extension addresses sequential tasks, in which each action changes the scene and hence the next relevant affordance. NaturalVLM [173] selects a fine-grained step from the current observation, high-level instruction, and end-effector state before predicting its affordance. SeqAfford and SeqAffordSplat [145,152] predict sequences of 3D affordance regions in point-cloud and Gaussian-splatting representations, respectively, while TRACER [174] hierarchically decomposes intentions and refines functional regions on deformable objects. Unlike single-step grounding, these sequential methods make affordance selection conditional on task progress and changing state, providing a bridge toward long-horizon interaction.

Overall, integrated methods jointly learn task interpretation and spatial prediction, modular methods translate intent into queries or hypotheses for specialized grounders, and sequential methods update affordance decisions as tasks unfold. Language-centric reasoning thus provides a flexible semantic interface between task intent and spatial affordance grounding.

4.3. Agentic Affordance Reasoning

Agentic affordance reasoning organizes affordance inference as an explicit workflow rather than a single model prediction. It decomposes the reasoning process into coordinated operations and invokes specialized tools or modules for functions such as visual grounding, scene querying, memory retrieval, and geometric verification. Compared with language-centric reasoning, which primarily relies on model knowledge to decide which affordance area should be grounded, agentic reasoning also exposes how this judgment is constructed and how intermediate evidence is acquired. Existing methods can be broadly organized into two regimes: predefined workflows connect specialized operations in a largely fixed sequence, whereas adaptive workflows condition subsequent tool calls, retries, replanning, or termination on accumulated evidence, belief states, or execution feedback. The distinction lies in runtime control flow rather than the number of modules.

Predefined workflows apply a fixed sequence of specialized operations. A4-Agent [163] separates zero-shot prediction into a Dreamer that generates a plausible interaction image, a Thinker that identifies the functionally relevant object part, and a Spotter that converts the resulting description into boxes, keypoints, and a precise affordance mask. AffordBot [54] segments 3D affordance-element candidates and projects their identifiers and types into pre-rendered surround views. An MLLM then selects an instruction-relevant view, identifies the referenced candidates, and estimates their motion type and discretized motion axis. PAP [102] adapts staged reasoning to ultra-high-resolution panoramas, using recursive visual routing to narrow the search space, adaptive gaze to correct local distortion, and cascaded grounding to recover the affordance mask. Some predefined workflows add memory or feasibility checks while retaining fixed control flow. AffordMEM [162] uses cross-scene memory to retrieve examples of small functional regions and in-scene spatial memory to organize candidate instances and their relations for resolving spatial references. SceneTeract [159] instead grounds task decomposition in an agent profile and maps each atomic action to relevant navigability, reachability, interactability, clearance, and visibility checks. It thereby turns a plausible interaction into an agent-conditioned feasibility judgment with step-wise diagnostics. These methods make intermediate decisions more transparent and task-specific, although their reasoning trajectories remain largely determined by manually designed module compositions and execution orders.

Adaptive workflows allow intermediate outcomes to determine what the system does next. Agentic Scene Policies [158] expose open-vocabulary retrieval, spatial-relation queries, and affordance-mediated interactions as tools over a queryable object map. Tool outputs and explicit failure messages are returned to the LLM to guide subsequent tool calls and replanning, connecting explicit affordance reasoning with policy execution. A3R [160] places the adaptive loop within evidence acquisition. It maintains a scene-grounded state comprising the current 3D Gaussian workspace, an affordance belief map, and the action-evidence history. Conditioned on this state, an MLLM iteratively selects 3D instance or part analysis, 2D semantic evidence with 3D lifting, workspace zoom, or termination. Each result updates the corresponding workspace, belief, or history state before the next decision, and the process ends when the policy selects termination or reaches its evidence-acquisition budget. A-Harness [161] instead instantiates a verification-driven adaptive loop. Its evidence store records heterogeneous tool outputs and their provenance, memory provides episodic priors, and a router selects and parameterizes the next skill. An affordance-specific verifier then checks cross-skill consistency, cross-scale stability, and evidence sufficiency. Failed verification identifies the missing or conflicting evidence and triggers a targeted corrective call, while successful verification allows the accumulated evidence to be fused into the final affordance prediction.

Overall, predefined and adaptive workflows form a continuum of agentic affordance reasoning. Predefined methods expose how an affordance judgment is assembled, while adaptive methods additionally revise the inference process using accumulated evidence or feedback. Agentic affordance reasoning therefore concerns how an affordance hypothesis is constructed, supported, and validated before commitment. Once the committed affordance is converted into embodiment-specific motion or closed-loop control, the focus shifts to affordance-guided action.

5. Affordance-Guided Action

Affordance-guided action studies how grounded or inferred affordances are translated into executable robot behavior. At this stage, affordance serves as an interface between task-relevant interaction information and embodiment-specific action generation, providing cues such as interaction types, contact regions, poses, motion directions, trajectories, or spatial value fields. Affordance perception grounds candidate interactions, affordance reasoning determines which affordance is relevant to the task or context, and affordance-guided action uses the resulting information to generate or learn actions under robot-specific execution constraints.

We organize existing methods into two complementary paradigms according to how affordance enters action generation, as shown in Figure 6. Hierarchical affordance-to-action methods preserve affordance as an intermediate interface and pass it to a downstream grasp generator, controller, or planner to generate executable robot actions. Affordance-integrated policy learning instead incorporates affordance into a learned policy or action model as an explicit input, implicit representation, or optimization signal. These paradigms capture the primary way in which affordance contributes to action generation rather than mutually exclusive system architectures. Table 3 summarizes representative methods and the specific role affordance plays in each.

5.1. Hierarchical Affordance-to-Action

Hierarchical affordance-to-action studies use affordance as an explicit interface between perceptual grounding and robot execution. Rather than mapping observations directly to motor commands, they first derive a task-relevant functional structure and then use it to guide downstream grasp generation, motion planning, trajectory optimization, or closed-loop control. Existing methods instantiate this idea through three complementary strategies: primitive-based execution uses predicted affordances to select grasps or parameterize motion primitives; retrieval-based transfer retrieves interaction knowledge from prior examples and adapts it to new objects or scenes; and planner-based execution treats affordance regions, fields, keypoints, or relations as targets or constraints for a downstream planner or optimizer. This modularity preserves interpretability while allowing robot-specific requirements, including kinematics, reachability, and collision avoidance, to be handled during execution.

Following the general formulation in Equation (1), hierarchical affordance-to-action can be expressed as an explicit two-stage mapping from affordance grounding to robot execution:

$$\mathcal{F}=f_{\theta }(o,l,c)=(a,r,\eta ),u=g_{\mathrm{exec}}\left(\mathcal{F},{\mathcal{C}}_{\mathrm{robot}}\right).$$

(2)

Here, $f_{\theta }$ derives the explicit affordance $\mathcal{F}$ from observation o, task information l, and optional context c, such as demonstrations, retrieved memories, or correspondence cues. Within $\mathcal{F}$, a, r, and $\eta$ denote the interaction type, grounded region, and execution-related information (e.g., an action score, motion direction, or value field), respectively. The execution module $g_{\mathrm{exec}}$ then maps $\mathcal{F}$ to an action u under robot constraints ${\mathcal{C}}_{\mathrm{robot}}$, and may take the form of a grasp generator, planner, controller, or optimizer that operates separately from the affordance predictor.

Primitive-based execution converts affordance predictions into actions through grasp estimators or predefined motion primitives. ASPN [175] predicts a 3D affordance map for push and grasp selection without explicit hand–eye calibration, whereas Where2Act [176] couples point-level actionability with predefined action primitives for articulated objects. For task-oriented grasping, Chen et al. [205] combine coarse 6-DoF candidates with an affordance heatmap and grasp confidence to favor functionally relevant regions. AdaAfford [178] adapts predictions through a few test-time interactions that reveal hidden kinematic and dynamic properties, while environment-aware affordance learning [180] incorporates occlusion, surrounding geometry, and robot morphology into actionability estimation. Subsequent methods enrich this interface with temporal and task-dependent action information. VAT-Mart [177] predicts contact locations together with action trajectories for articulated-object manipulation, shifting the representation from where to touch toward how the interaction should unfold after contact, while DefoAfford [181] makes a similar move for deformable objects by learning dense foresightful affordances that anticipate future object states. VRB [10] learns contact points and post-contact trajectories from human videos, projects these cues into 3D, and uses them to parameterize robot actions for imitation learning, exploration, goal-conditioned learning, and reinforcement learning. $A_0$ [185] similarly abstracts contact and post-contact motion into object-centric waypoints for cross-embodiment execution. PRIMP [206] adapts probabilistic motion primitives from simulated functional poses for obstacle-aware execution, whereas BridgeACT [207] couples tool–target grasp regions with 3D interaction dynamics for closed-loop control. Language and foundation models broaden primitive-based execution to novel objects and task instructions. LERF-TOGO [55] ranks grasp candidates using language-conditioned 3D relevance, whereas open-vocabulary part-based grasping [208] grounds task-relevant parts for 6-DoF grasp prediction. OVAL-Grasp [209] further filters candidates through grasp and avoidance regions, while ManipGPT [186] translates prompted affordance masks into contact points and manipulation directions. This execution interface also extends to more complex end effectors. AffordDexGrasp [187] and ScaleADFG [188] guide functional dexterous grasps, whereas DualAfford [179], BiAssemble [189], A3D [191], and BiPreManip [192] model collaborative affordances for bimanual manipulation, including assembly, coordination, and preparatory actions.

Retrieval-based transfer treats affordance as interaction knowledge that can be reused across objects. These methods retrieve a relevant exemplar, align it with the target object or part, and transfer spatial constraints, contact patterns, or motion cues for downstream interaction. DINOBot [183] retrieves a task-matched demonstration, aligns its bottleneck pose through pixel correspondences, and replays the associated trajectory. Robo-ABC [9] constructs a contact-point memory from human videos, retrieves visually or semantically similar reference objects, and transfers their contact points to novel objects through cross-category correspondences. RAM [53] broadens both the memory sources and action representation by retrieving task-relevant demonstrations from heterogeneous robot and human-interaction data, then transferring their 2D affordances into executable 3D contact points and post-contact directions. RAAP [193] further transfers contact points through dense correspondence while estimating action directions from multiple retrieved references. Together, these methods support affordance-guided action on novel objects by reusing prior interaction knowledge rather than learning a task-specific predictor from scratch.

Planner-based execution uses affordance estimates as spatial targets, objectives, or constraints for downstream planning and optimization. Rather than associating each affordance with a predefined action primitive, it lets a separate planner determine a robot-feasible viewpoint, approach motion, grasp configuration, waypoint, or trajectory. HULC++ [15] predicts a language-conditioned 3D interaction location, uses model-based planning to approach it, and then switches to a learned visuomotor policy for local execution. PreAfford [184] is a pre-grasping planner that uses point-level affordance maps to select pushing contacts and directions that improve subsequent graspability, while ACE-NBV [182] extends this paradigm to active view planning, using predicted grasp quality over imagined viewpoints to select the next view. Other methods express functional requirements as spatial fields or optimizable constraints. VoxPoser [16] composes language-derived 3D affordance and constraint maps as objectives for model-based motion planning; ReKep [17] optimizes multistage motions through relational costs over tracked 3D keypoints; and O3Afford [131] grounds object-to-object affordances to generate task-specific constraints for object-transformation planning. This principle also extends to specialized embodiments and learned action models. FSAG [194] predicts finger-specific contact affordances from human demonstrations and converts them into dexterous grasps through kinematics-aware retargeting and motion planning. AFI [195] uses a 3D spatial affordance field to intervene in VLA execution through corrective waypoint generation and trajectory reranking. Across these methods, affordance specifies task-relevant spatial structure, while planning converts it into robot-feasible observation and motion decisions.

5.2. Affordance-Integrated Policy Learning

Affordance-integrated policy learning embeds affordance directly into action prediction or policy training, rather than passing it to a separate execution module. Existing approaches use affordance in three non-exclusive roles: explicit input directly provides grounded affordance information to the policy; implicit representation learns affordance-related features or auxiliary predictions within the action model; and optimization signal filters candidate actions, shapes rewards, or defines goals. These roles describe the primary use of affordance rather than mutually exclusive architectures: a method may, for example, condition a policy on an affordance map while also using it to shape rewards.

Affordance as explicit input directly conditions the action model on grounded affordance cues, such as interaction types, grounded regions, contact points, or motion cues:

$$\mathcal{F}=f_{\theta }(o,l)=(a,r,\eta ),u={\pi }_{\varphi }\left(o,l,p,\mathcal{F}\right).$$

(3)

Here, a, r, and $\eta$ denote the interaction type, grounded region, and optional execution cue, while the policy ${\pi }_{\varphi }$ receives $\mathcal{F}$ together with observation o, task information l, and proprioception p. IntExp [210] is an early example, learning action-specific affordance maps from interaction outcomes and using them to train an actor–critic exploration policy. VLA-based methods employ richer explicit intermediates. RT-Affordance [18] conditions action prediction on projected end-effector poses at key task stages, whereas CoA-VLA [62] represents object, grasp, spatial, and movement affordances in textual and visual forms to guide action prediction. Explicit cues also guide generative policies. Zhang and Gienger [211] condition flow-matching trajectory generation on affordance maps; AffordDP [61] conditions a diffusion policy on transferred 3D contact points and post-contact trajectories and applies affordance guidance during sampling; and DART [200] combines task representations with learned interaction points for articulated-object manipulation. Other methods focus on acquiring diverse affordance cues. UAD [49] distills task-conditioned affordance maps from large vision and vision–language models without manual annotations and uses them as observations for imitation learning. GLOVER++ [201] learns affordance representations from large-scale human-behavior data to modulate attention in policies. Affordance2Action [204] predicts scene-level, task-conditioned affordance regions as visual conditions for an action-chunking diffusion policy. Across these designs, explicit affordances remain inspectable conditions that provide functional, spatial, or motion guidance for action prediction.

Affordance as implicit representation embeds functional information within the action model rather than providing a separately predicted external input. Under Latent affordance conditioning, an affordance-related feature is inferred and fused with the policy inputs:

$$z_{\mathrm{aff}}=h_{\theta }(o,l),u={\pi }_{\varphi }\left(o,l,p,z_{\mathrm{aff}}\right).$$

(4)

Alternatively, Affordance auxiliary supervision trains a shared model to predict both actions and affordances:

$$\begin{array}{c}\hfill (u,\mathcal{F})={\pi }_{\varphi }(o,l,p),\mathcal{L}={\mathcal{L}}_{\mathrm{act}}(u,u^{*})+{\lambda }_{\mathrm{aff}}{\mathcal{L}}_{\mathrm{aff}}(\mathcal{F},{\mathcal{F}}^{*}),\end{array}$$

(5)

where $u^{*}$ and ${\mathcal{F}}^{*}$ denote the ground-truth action and affordance targets, respectively, and ${\lambda }_{\mathrm{aff}}$ balances the affordance supervision. Latent affordance conditioning introduces $z_{\mathrm{aff}}$ as an internal action condition, whereas auxiliary supervision uses $\mathcal{F}$ as a training signal to encourage the shared policy ${\pi }_{\varphi }$ to capture task-relevant interaction structure, without requiring the affordance output at deployment.

These implicit uses of affordance are increasingly adopted in VLA-oriented policies: PALM [63] uses progress-aware affordance reasoning to track long-horizon task evolution; AffordVLA [47] distills affordance knowledge into intermediate visual representations through implicit feature alignment; Afford-VLA [64] learns task-conditioned affordance regions as action-aligned visual intermediates for action generation; and AffordanceVLA [65] trains internal affordance queries through object grounding, interaction localization, and 3D geometric prediction, and allows the action expert to attend to the affordance expert’s representations. These VLA methods integrate affordance learning directly with action prediction, allowing affordance-related features and objectives to be optimized jointly with the robot embodiment, data distribution, and action space.

Affordance as optimization signal directly changes the actions or outcomes favored by a policy through action filtering, reward shaping, or goal specification. Unlike auxiliary supervision, it modifies the feasible action set, behavioral objective, or exploration target rather than only the learned representation.

For filtering and reward shaping, Cruz et al. [212] propose a contextual-affordance-based reinforcement learning framework that predicts the effects of candidate actions and retains only those expected to produce valid states. GRAFF [197] conditions PPO on an affordance map and hand–affordance distances while penalizing distance to the predicted region. VAPO [14] learns affordance centers from teleoperated play and rewards a local SAC policy for approaching them, whereas RLAfford [198] learns point-wise affordances from RL contacts and rewards motion toward the highest-scoring point. Similarly, Nakamoto et al. [213] learn a value function with offline RL and use it to re-rank actions proposed by generalist robot policies at deployment, providing a task-conditioned affordance signal for action selection. In dexterous manipulation, DORA [202] constrains the policy with affordance-aligned grasps and rewards fingertip proximity to positive regions, while AffordDex [203] provides predicted negative-affordance regions to a residual policy and penalizes the hand for approaching them. For goal specification, VAL [196] learns an outcome model, samples latent goals, and optimizes a goal-conditioned policy with off-policy RL. Bharadhwaj et al. [199] instead sample future-image goals from passive videos and train on hindsight-relabeled trajectories through behavior cloning. Together, these methods use affordance to restrict candidate actions, reshape policy objectives, or determine which outcomes should be pursued.

6. Data and Evaluation

6.1. Datasets for Affordance Learning

Affordance datasets have expanded along several dimensions, from isolated objects to full scenes, from RGB images to videos and 3D representations, and from manually annotated masks to interaction-derived heatmaps, language-conditioned regions, contact points, and motion-related supervision. Table 4 summarizes representative datasets across these dimensions, including their setting, category and scale statistics, input modalities, and annotation types. To facilitate the following discussion, we further organize these datasets according to their primary source of affordance evidence and evaluation objective into five broad and partially overlapping groups: object-centric, scene-level, interaction-driven, language- and reasoning-oriented, and action-oriented datasets. These groups are not mutually exclusive, but provide a coherent organization for discussing the principal capability each dataset is designed to assess.

Object-centric datasets ask which part of an object supports an action. Early 2D benchmarks such as UMD [1] and IIT-AFF [3] established dense part-affordance segmentation with closed labels such as grasp, cut, contain, or support. Their 3D counterparts ground the same question in object geometry: 3D AffordanceNet [5] provides probabilistic point-wise affordance annotations and benchmarks full-shape, partial-view, and rotation-invariant estimation; PartAfford [69] contains over 25,000 objects with 24 part-affordance categories; and AffordPose [43] pairs part affordances with 3D hand poses. Together, they support actionable-part localization in both image space and metric object representations.

Scene-level datasets evaluate affordances under broader environmental context. NYUv2 Affordance [98] provides dense pixel-level affordance labels for indoor scenes, whereas ADE-Affordance [136] annotates scene-dependent action–object relations together with explanations and consequences. Panoramic benchmarks such as 360-AGD [101] and PAP-12K [102] introduce wide-field observations for holistic embodied perception. In 3D settings, SceneFun3D [6] annotates functional elements and their motion parameters in full scenes, FunGraph3D [215] complements this representation with open-vocabulary 3D scene graphs that capture objects, interactive elements, and their functional relationships, and AffordBridge [86] links 2D visual signifiers to 3D affordance regions. These benchmarks extend scene evaluation from pixel-level localization to panoramic, graph-structured, and cross-modal 3D grounding.

Interaction-driven datasets infer affordances from observed human pose, contact, and motion, providing evidence of both where and how interactions occur. OPRA [7] pairs product-demonstration clips with target images, action labels, and point annotations of interaction regions, while AGD20K [45] combines exocentric interaction images with egocentric object views and point-derived heatmaps. Egocentric datasets provide complementary forms of interaction-derived supervision: EPIC-KITCHENS Affordance [217] annotates motor actions, grasp types, and mechanical interactions, whereas EPIC-Aff [111] organizes interaction-derived, multi-label annotations into metric affordance maps and spatial action-centric zones. For 3D grounding, PIAD [72] pairs interaction images with object point clouds, PIADv2 [147] extends this setting to open-vocabulary evaluation, and VIDA [127] associates HOI videos with point clouds and 3D affordance annotations. These resources support transferring human interaction evidence to object- and scene-level affordance grounding.

Language- and reasoning-oriented datasets condition affordance grounding on instructions, questions, or implicit goals. LLMaFF [46] pairs natural-language instructions with complex multi-object scenes, InstructPart [168] associates object part masks with task-oriented instructions, and ReasonAff [155] links affordance masks to reasoning-oriented instructions. Datasets organized primarily around question–answer evaluation include A4Bench [167], which covers constitutive and transformative reasoning, and RoboAfford++ [171], which evaluates object recognition, functional-part localization, and free-space localization. In 3D, LASO [75] pairs object point clouds with situated affordance questions and point-wise maps, CompassAD [27] uses implicit-intent queries over functionally competing objects, and Affogato-750K [88] provides open-vocabulary queries with 3D heatmaps. These datasets cover both functional reasoning and language-conditioned grounding across objects, parts, points, and regions.

Action-oriented datasets associate affordance annotations with grasping or manipulation objectives. HANDAL [214] provides 308K annotated frames of 212 real-world manipulable objects, together with category-level 6-DoF pose and scale annotations, affordance segmentation annotations, and reconstructed meshes. RAGNet [169] pairs affordance masks with template- and reasoning-based instructions for general grasping. HOVA-500K [201] derives large-scale affordance-point supervision from human behavior, and A2A-Bench [204] provides task-conditioned functional-region masks in multi-object scenes. ScaleADFG [188] evaluates functional dexterous grasping, whereas AffordSim [218] benchmarks scalable affordance-aware manipulation. Some of these datasets additionally evaluate downstream execution, whereas others remain grounding benchmarks whose annotations are designed to support action.

6.2. Evaluation Metrics

Evaluation protocols depend on the form of affordance output, spanning heatmaps, segmented regions, contact points, motion parameters, and trajectories. We therefore group commonly used metrics into heatmap/distribution, segmentation, and keypoint/motion metrics, while excluding reasoning and manipulation metrics whose definitions vary substantially across tasks and robot embodiments.

Heatmap and distribution metrics. An affordance heatmap assigns an interaction likelihood to each pixel or 3D point. Kullback–Leibler divergence (KLD) measures deviation from the target distribution, similarity (SIM) measures shared probability mass, and normalized scanpath saliency (NSS) evaluates standardized responses at positive ground-truth locations. Mean absolute error (MAE) and mean squared error (MSE) instead compare map values point-wise, with MSE placing greater weight on large deviations. Distribution metrics emphasize whether probability mass falls on the correct functional area, whereas point-wise errors measure global numerical agreement. A method may therefore achieve a low global error without sharply localizing the most actionable region, so reporting complementary metrics is generally more informative than relying on one score. Representative applications include AGD20K [45], which evaluates 2D affordance heatmaps using KLD, SIM, and NSS. Under heatmap-based evaluation in 3D, 3D AffordanceNet [5] reports point-wise MSE, whereas PIAD [72] reports SIM and MAE.

Segmentation metrics. For discrete masks or binarized maps, intersection over union (IoU) measures spatial overlap, while mean IoU averages class-wise IoU across affordance categories. In 2D evaluation, the mean of per-image IoUs, often denoted gIoU, weights each image equally, whereas cumulative IoU (cIoU) aggregates intersections and unions over the dataset and consequently gives larger regions greater influence. Thresholded precision measures the proportion of predictions attaining sufficient overlap: P@50 uses an IoU threshold of 0.5, while P@50:95 averages this success rate over thresholds from 0.50 to 0.95 in increments of 0.05. Image-level benchmarks such as InstructPart and RAGNet report these region-based metrics [168,169]. Their results may differ substantially with region size, aggregation strategy, and the treatment of empty predictions.

For continuous 3D point maps, segmentation-style evaluation generally binarizes the ground-truth scores, commonly at 0.5. The area under the receiver operating characteristic curve (ROC-AUC) and mean average precision (mAP) compare continuous prediction scores against these binary targets: ROC-AUC measures ranking across decision thresholds, while mAP averages category-wise precision–recall performance and is generally more informative when positive affordance points are sparse. Averaged IoU (aIoU) instead thresholds each prediction at multiple values and averages the resulting IoUs across thresholds and samples. These metrics are commonly used by 3D affordance benchmarks [5,72], but comparisons require consistent ground-truth binarization, prediction thresholds, class averaging, and sample-level aggregation.

Keypoint and motion metrics. Keypoints are typically evaluated by Euclidean localization error or accuracy within a distance threshold. Since distances may use pixel, normalized, or metric 3D coordinates, the coordinate system and scale should be reported. Motion-aware protocols additionally evaluate interaction dynamics. SceneFun3D [6], for example, augments mask average precision with requirements on motion type, axis orientation, and axis origin, separating localization from motion-estimation errors.

For trajectories, average displacement error (ADE) compares temporally corresponding positions, whereas dynamic time warping (DTW) allows nonlinear alignment and is less sensitive to execution speed. Lower ADE and DTW values indicate more accurate trajectories. Evaluation protocols may also normalize trajectories before computing these metrics. For example, Yoshida et al. [117] normalize predicted and ground-truth trajectories and set their initial coordinates to zero, isolating trajectory shape from contact-point localization.

7. Challenges and Future Directions

Affordance learning has evolved from visual recognition toward a bridge between perception, reasoning, and action in embodied AI. Despite this progress, important challenges remain in developing scalable, reliable, and actionable affordance learning systems for open-world embodied interaction. Accordingly, we organize the following discussion around affordance representation and action integration, scalable learning and open-world generalization, reliable and autonomous grounding, closed-loop modeling and planning, and embodied evaluation and trustworthiness.

7.1. Affordance Representation and Action Integration

A central question is not only what an affordance representation should predict, but how it should contribute to robot action. Explicit masks, contacts, or trajectories provide interpretable interfaces for planners and policies, but make execution sensitive to prediction errors [9,18,62,118]. Latent features enable flexible end-to-end optimization but often require additional feature encoders and provide limited interpretability [47,65]. Auxiliary objectives introduce action-oriented inductive bias without requiring affordance outputs at deployment, yet improved auxiliary prediction does not necessarily yield higher execution success [64,65]. Since these designs are usually evaluated with different data, architectures, action spaces, and tasks, it remains unclear when each form of integration is most effective.

Future work should compare explicit, latent, auxiliary, optimization-signal, and hybrid formulations through controlled downstream experiments with matched observations, training data, policy backbones, and action interfaces. Comparisons should include no-affordance, predicted-affordance, and oracle-affordance conditions rather than rely on localization performance alone. Intervention tests that remove, perturb, or replace affordance information can further reveal whether a policy causally depends on the intended functional signal. The broader goal is to identify not one universally optimal interface, but the conditions under which each affordance representation most effectively improves action.

7.2. Scalable Learning and Open-World Generalization

Dense affordance annotation is difficult to scale because annotators must infer fine-grained, action-specific interaction regions. Existing methods therefore draw supervision from human–object interactions, demonstrations, pseudo-labels from foundation models, simulation, and robot interaction [7,8,18,49,60,71,219]. These sources broaden coverage but introduce different forms of uncertainty and bias: human bodies may occlude contacts, videos may suffer from motion blur and limited visibility of contact regions, and pseudo-labels may inherit foundation-model hallucinations and semantic preferences. Moreover, many datasets remain concentrated on a small set of object categories, canonical actions, clean observations, and limited object states. Models can consequently perform well in an in-distribution setting while failing to transfer to unseen objects, novel tasks, or complex environments. Scalability and open-world generalization are therefore inseparable: additional data is valuable only when its diversity and quality support transferable functional relations.

Future data pipelines should expand coverage while actively controlling supervision quality. Affordance-guided data generation offers a promising direction by preserving functional correspondences while varying object geometry, pose, and scene configuration. Recent work instantiates this idea by transferring limited demonstrations across novel 3D objects or integrating open-vocabulary prediction with simulation to generate manipulation data [218,220,221]. Generative predictive models may further scale affordance learning by synthesizing diverse interaction trajectories and state transitions [222,223]. The objective is not merely to enlarge the training set but to learn functional relations that remain stable across objects, tasks, states, and environments.

7.3. Reliable and Autonomous Affordance Grounding

Foundation models have transformed affordance grounding from single-model prediction into multi-stage pipelines. A typical workflow may use a vision-language model to interpret the instruction and identify a relevant object or semantic part, a detector or segmentation foundation model to localize that part, and additional 3D, geometric, or verification modules to obtain the final interaction region [49,224]. Modular and agentic systems make these stages explicit and can combine complementary capabilities that no single model provides [102,163]. However, their performance often depends on manually chosen prompts, part vocabularies, model combinations, thresholds, views, execution orders, and fallback rules. A segmentation model may recover visual boundaries rather than the functional contact area, and an early grounding error may propagate through every subsequent stage. This makes it difficult to determine whether gains arise from a general affordance capability or from task-specific pipeline engineering, and complicates reproducibility and fair comparison across rapidly changing foundation models.

Future systems should move from manually assembled feed-forward pipelines toward automatically orchestrated and self-verifying grounding processes. An autonomous grounding system should select tools and evidence sources according to the task, preserve the provenance and uncertainty of intermediate outputs, check semantic, spatial, cross-view, and cross-module consistency, and trigger targeted retries or additional evidence acquisition only when necessary. Adaptive routing and affordance-specific verification provide initial steps in this direction [160,161], but systematic evaluation is still needed. In addition to final grounding accuracy, comparisons should report intermediate failure modes, the number of human corrections, prompt and rule dependence, tool calls, latency, computational cost, and sensitivity to component replacement. The aim is not simply to construct a longer pipeline, but to reduce manual intervention while making its decisions traceable and robust to component errors.

7.4. Closed-Loop Affordance Modeling and Planning

Affordances are state-dependent rather than fixed labels. Robot actions can create occlusions, alter object poses or scene configurations, and invalidate previously correct estimates. In long-horizon tasks [225,226], each outcome also changes which object, region, or interaction is relevant next. Nevertheless, most methods rely on one-shot prediction, allowing early errors to compound without closed-loop correction or continual affordance updating.

Future systems should maintain affordance as a belief updated through observation, action, and execution feedback. Active observation and low-risk exploratory actions can reveal hidden articulation, friction, stability, compliance, and occluded parts, while uncertainty and expected information gain can determine when an agent should gather evidence, act, or revise its hypothesis [160,178,182,227]. For long-horizon tasks, agents must also model affordance transitions, use execution outcomes to revise later choices, track progress, and recover from unexpected effects. Sequential reasoning, relational keypoint constraints, and multi-stage scene representations provide foundations for this direction [17,132,145,152,228]. This would turn affordance into a persistent interface connecting perception, reasoning, action, and feedback over time.

7.5. Embodied Evaluation, Safety, and Trustworthiness

Common metrics such as classification accuracy, mask overlap, point localization, and heatmap similarity measure perceptual agreement but not embodiment-aware validity. A plausible region may be unreachable, unstable, or incompatible with the agent’s morphology, kinematics, or skills, making validity inherently agent-dependent. Recent systems such as SceneTeract [159] explicitly model agent profiles and feasibility, but most benchmarks still overlook embodiment-specific executability. Safety further separates what an agent can do from what it should do: negative affordances, fragile regions, social constraints, and embodiment-specific failures remain underexplored.

Future benchmarks should evaluate affordance as an embodied and trustworthy capability, complementing localization with embodiment-specific feasibility, downstream success, and uncertainty calibration across tasks and robot platforms, with validated simulation providing a scalable evaluation complement [229]. Building on multisensory policy learning [230,231], evaluation should also test whether integrating complementary modalities improves affordance judgments under ambiguous visual evidence. Systems should condition validity judgments on the agent, act conservatively under uncertainty, and account for human intent and social norms [232,233,234], since physical executability alone does not ensure safe or appropriate interaction.