From Engineered Features to Foundation Models: A Decade of Teaching Machines to See, Told Through One Research Path

1. Introduction: Same Questions, New Tools

Imagine asking a computer to recognize a kitchen. In 2017, you would collect thousands of labeled photos of kitchens and other rooms, train a neural network on them, and hope it learned what makes a kitchen a kitchen. In 2026, you would probably start from a large model that has already looked at hundreds of millions of images without any labels at all, and then nudge it toward your task with a small amount of extra training. The question is the same. Almost everything about the answer is different. Some questions have also grown: instead of picking one label from a fixed list, today’s models can recognize things they were never explicitly trained on, describe an image in words, or segment any object they are pointed at.

This paper is about that change. It is not a survey of the whole field. Instead, it follows one research path that happens to sit on both sides of the change, and uses it as a window onto the bigger story.

That path began, long before deep learning, with hand-designed visual tracking for robots [1]. The part of it this paper follows begins with simpler questions about people. Can a phone’s motion sensor tell whether you are walking, sitting, or climbing stairs [2,3,4]? Can the way your eyes move across a page reveal how hard the text is, or even who you are [5,6,7,8]? It moved on to scenes: what do deep networks actually learn when they recognize a room, and do they use context the way people do [9,10]? It then turned to medicine, using deep learning to read and even “stain” tissue images [11,12], and to think about AI in clinical trials [13]. Today it includes adapting large vision models to estimate pasture biomass [14], reading emotion in clinical conversations [15], building AI patients for training therapists and clinicians [16,17], and building AI agents that design user interfaces or record software design decisions [18,19].

Looking at these projects side by side, a pattern appears. In several cases the same kind of problem shows up twice: once solved with the tools of 2017, and once with the tools of 2026. That makes it easier to see what really changed. The rest of the paper tells the story in order: where things stood (Section 2), the turning point (Section 3), where things stand now (Section 4), the six main shifts (Section 5), and the questions that remain (Section 6).

2. Where Things Stood: Around 2017

2.1. When People Designed the Features

Before deep learning took over everything, much of the work in recognizing human behavior was craftsmanship. If you wanted a phone to recognize walking, you did not hand it raw sensor readings and wait. You cut the signal into short windows, then calculated numbers you believed would matter: averages, variations, how strongly the signal repeated. A standard classifier then learned to tell activities apart from those numbers. Our early work on recognizing activities and walking patterns from smartphone accelerometers [2,3] followed exactly this recipe. Because good data was hard to come by, sharing a dataset was a real contribution in itself, which is why we released a database of normal walking recorded with smartphones [4].

Eye-movement research worked the same way. Researchers had already shown that the way a person moves their eyes can identify them [20]. We first studied how the difficulty and topic of a text change where people look and how their pupils react [5]. Then we turned those patterns into a biometric [6]. From the eye movements of forty readers we designed 28 features describing fixations, the path the eyes take across the page, and pupil behavior. A combination of three classical classifiers (a multilayer perceptron, logistic regression, and a logistic model tree) identified readers with 95.3% accuracy and an equal error rate of 2.0%, far below the roughly 11% of the best earlier eye-movement methods.

The interesting twist came next. Text content changes how people read, which makes eye movements harder to use for identification. Instead of treating the text as noise, we used it. Each word was represented by a word2vec vector, a then-new way of placing words with similar meanings close together in a mathematical space, and that representation of the text was combined with how long each reader’s eyes rested on each word. Knowing what someone was reading, together with how their eyes moved, raised identification accuracy to 98.4% [7,8]. Looking back, this was multimodal fusion built by hand, an idea that would later become one of the central themes of the whole field.

2.2. When Deep Networks Arrived, and We Asked What They Knew

In computer vision itself, deep convolutional networks had already changed the game [21,22]. Networks trained on huge labeled collections of photos, including a database of about ten million scene images called Places [23], could recognize objects and scenes with remarkable accuracy. Their inner workings, however, were a mystery. A natural question followed: what had they actually learned?

One striking answer was that networks trained only to name scenes had quietly learned to detect the objects inside them [24]. A network trained to recognize “kitchen” had, without being told, learned something about stoves and sinks. Our own work asked a complementary question [9]. Psychologists had shown that people can grasp a scene’s overall “shape” before recognizing any objects in it, using global properties such as whether it is natural or man-made and whether it is open or enclosed [25]. Can a network trained to name scenes recover those properties too? We took a VGG16 network trained on Places365 [23,26], extracted features from four of its layers for about 7,000 images from the SUN database [27], and trained a simple linear classifier on each layer to predict naturalness and openness, using ratings from human observers as ground truth. On clear-cut images every layer did almost perfectly. The surprise came with ambiguous images, the kind people themselves rate as only moderately open or natural. There, features from the network’s very first convolutional layer still predicted openness with 93% accuracy, while the deeper layers fell close to chance (51–60%). The holistic layout of a scene, it turned out, was best captured not by the network’s most abstract features but by its earliest ones, a result that held even when we kept only the fine detail or only the coarse layout of each image.

A second question was about context. People have strong expectations about scenes. We expect a toothbrush near a sink, not on a stove. Psychologists call these expectations a “scene grammar”, and decades of experiments show that we recognize objects faster when they fit it and react strongly when they do not [28,29]. We tested people and machines on the same images [10]. Twenty participants viewed scenes from the SCEGRAM database, some containing an object that broke the scene’s rules, while we recorded their eye movements. People looked at the out-of-place objects significantly more often, and a simple classifier could tell from their fixations alone whether a scene contained a violation. The deep networks of the day were thrown off by the same violations: the scene-recognition network (AlexNet trained on Places) made about 9 percentage points more top-1 errors on inconsistent scenes (41.5% instead of 32.7%), and the object-recognition network made about 13 points more. We also asked whether the networks’ internal attention maps, drawn from their last convolutional layer, looked where people looked. On inconsistent scenes they largely did not: the maps missed the out-of-place objects that drew human eyes. In short, the networks had absorbed scene context well enough to be confused when it was broken, but not in a way that matched human attention.

3. The Turning Point: Deep Learning Meets the Clinic (2019–2021)

Medicine shows the middle of the story especially clearly. In the late 2010s, a new family of models learned to generate images, not just classify them [30]. They could turn one kind of image into another, such as a sketch into a photo [31,32].

In pathology, this opened a surprising door. Normally, a tissue sample is treated with chemical dyes so that a pathologist can see its structure under the microscope. The most common is hematoxylin and eosin, or H&E. What if a computer could predict what the stained slide would look like, without the chemicals? This idea, called virtual staining, was first shown with fluorescence images of unstained tissue [33]. Our group took a related path with ordinary whole-slide images of non-stained prostate core biopsies: a generative adversarial network learned to turn them into computationally H&E-stained images, and these were evaluated for tumor diagnosis against the conventionally dye-stained slides [11].

We then asked whether the whole workflow could be automated, from unstained slide to diagnosis [12]. The resulting end-to-end system first stains the native non-stained images computationally, then uses a convolutional network (a ResNet) to classify $1024\times 1024$-pixel patches as tumor or non-tumor, and a weakly supervised model to show where the tumor is, without anyone drawing tumor outlines pixel by pixel. On a held-out set of about 17,000 patches from 13 biopsy slides, it reached 86.4% patch-level accuracy, and its tumor maps came close to those of a fully supervised segmentation model (a Dice score of 65.1 versus 70.2). One detail of that work is telling in hindsight: the network started from weights pretrained on ImageNet photos of everyday objects, and this worked better than training on pathology images alone. Borrowing knowledge from a large, general image collection was already the best strategy in 2021; the foundation models of the next section take that idea much further. Around the same time, other researchers showed that deep learning could reach clinical-grade accuracy on whole microscope slides using only weak, slide-level labels [34].

These systems were powerful, but each one was built for one job. A model trained on one kind of tissue for one task did not easily transfer to another, and collecting labeled or paired examples was always the slowest part. At the same time, interest was growing in using AI for whole medical processes, not just single images, such as designing and running clinical trials [13]. That broader view would soon become the norm.

4. Where Things Stand: 2026

4.1. Models That Learn Without Labels

The biggest change of the past decade is that models now learn most of what they know before anyone gives them a task-specific label. Instead of being told “this is a kitchen,” a model is trained on a huge collection of images, sometimes paired with captions, using a training signal that comes from the data itself. There are three main families of such methods, and they learn somewhat different things. Contrastive methods learn to recognize two altered versions of the same photo as a match and other photos as non-matches [35]. Self-distillation methods such as DINO train a “student” network to agree with a slowly updated “teacher” network [36]. Both tend to produce features that separate categories well. Masked image modeling hides large parts of an image and asks the model to fill them in [37]; its features capture fine detail but usually need some fine-tuning before they classify well. Trained at large scale with the Vision Transformer [38], these methods produce general-purpose models now called foundation models [39]. The data is not free of human influence: it is carefully filtered and curated, and caption-based models learn from text that people wrote.

The DINO family is a good example. DINOv2 [40] became a widely used baseline for “frozen” features, used as-is without retraining. Its successor, DINOv3 [41], released in 2025, scales the approach further and improves the detailed, pixel-level features that tasks like segmentation depend on. Other models learned from images paired with their captions, linking vision and language [42], and some can cut out any object you point at [43]. Medicine followed the same path, and there are now foundation models trained on enormous collections of pathology slides [44].

4.2. Adapting Instead of Training from Scratch

With such strong starting points, the everyday question has changed. It used to be “what network should I train?” Now it is “how should I adapt the one I have?” There are several answers. The simplest is linear probing: keep the model frozen and train only a small classifier on top. Others add small trainable pieces: adapter layers inserted into the network [45], learnable “prompt” tokens added to the input [46], or LoRA [47], which learns small low-rank updates to the model’s existing weights. LoRA was first designed for language models but is now common in vision too. All of these are far cheaper than retraining the whole model.

Our work on estimating pasture biomass shows this in practice [14]. The task is to predict five quantities, such as green, dead, and clover dry matter, from a single top-down photo of a pasture, using only 357 labeled images from 19 sites in Australia. Training a large network on so little data is hopeless, and in our experiments even fine-tuning just the last few layers of DINOv3 made results worse than leaving it frozen. Instead, we gave DINOv3 (ViT-Large and ViT-Huge, roughly 300 and 630 million parameters) LoRA adapters whose size grows with depth: the earliest layers, which capture generic visual building blocks, get tiny adapters (rank 2), while the deepest, most task-specific layers get large ones (rank 512). This depth-aware schedule beat uniform LoRA, full fine-tuning, and frozen features, reaching a cross-validated weighted $R^2$ of 0.81. A small team can now specialize a model with hundreds of millions of parameters for a narrow, practical problem with a few hundred examples.

4.3. Many Signals at Once

Remember the 2017 idea of combining eye movements with the meaning of the text? A modern descendant is a framework for analyzing therapy conversations from both sides [15]. On the patient side, a clinical language model recognizes 25 emotion categories in counseling text, despite some categories being sixty times rarer than others. On the provider side, a language model (DeBERTa) and a speech model (WavLM) are joined by cross-modal attention, so the words a therapist says and the way they sound are read together. On carefully annotated sessions this reached a macro-F1 of 0.91, with agreement comparable to that between human raters, and the attention-based fusion beat simply concatenating the two signals by 12 points. The idea is the same as in 2017. The difference is that each signal is now handled by a strong pretrained model, and the combining is learned rather than designed.

4.4. From Giving Answers to Taking Part

Finally, the outputs of vision systems have become much richer. Within computer vision itself, models now produce masks for any object a user points at [43] and full 3D representations of a scene from a handful of photos [48,49]. Beyond vision alone, vision-language models that can look at an image and talk about it [50] have made vision one part of larger, interactive systems. Our recent work reflects this broader shift, and some of it moves past vision into language and software. We built AI patients grounded in real clinical knowledge to help therapists practice [16], and an emotionally expressive virtual patient for training in speech-language pathology [17]. Moving further from vision, we used large language models to turn a written request into a working user-interface prototype [18], and to capture the design decisions made in conversations between developers and AI [19]. In each of these, the output is not a label. It is a conversation, a tool, or a record that a person goes on to use.

5. Six Shifts

Putting the two eras side by side, six changes stand out. Figure 1 shows the overall recipe then and now, Table 1 gives a short summary, and each change is explained below.

1. Where a model’s knowledge comes from.

In 2017, a model’s knowledge came either from a person’s design or from training on labeled photos. Figuring out what that knowledge looked like was a research topic of its own [9,24]. In 2026, it comes mostly from training on huge collections of unlabeled images [41]. In practice, people now usually start by assuming a foundation model’s features are general enough. The work has moved to finding where that assumption breaks, which it still does in many medical, satellite, and other specialized images that look little like the web data these models learned from.

2. How much labeled data it needs.

Labels used to be the scarcest resource, and building a dataset was often the biggest part of a project [4,11]. Now labels are mostly spent on adapting a model and on checking that it works. The hard part has moved from “do we have enough labels to train?” to “do we have enough trustworthy labels to know whether it works?”

3. How it handles context.

We once had to test whether machines used scene context the way people do [10]. Today’s large vision models absorb context from their training data automatically. Pure vision models such as DINO still only produce features, not answers, but vision-language models built on top of such encoders [42,50] can be asked about a scene in plain words. They are not always right; they often get relations such as “left of” or “on top of” wrong, which benchmarks for visual reasoning were designed to expose [51]. So the question has flipped. We no longer ask whether a model has expectations about scenes. We ask when those expectations mislead it, such as when an object is in an unusual place or a setting is rare in its training data.

4. How it combines different signals.

Combining eye movements with word2vec text vectors [7,8] meant designing both the inputs and the way they were combined. Today, each signal is handled by a pretrained model, and a learned attention mechanism does the combining [15]. Using many signals at once has gone from special to normal.

5. How it is adapted to a new task.

Retraining a whole network for every new task has been replaced by training small add-on pieces [47]. In practice, this means a small lab can take a very large model and specialize it for a narrow problem, like estimating pasture biomass [14], without huge computing costs.

6. What it finally produces.

The earlier systems produced labels: an activity, an identity, a scene type, the location of a tumor. Within computer vision, outputs have become far richer: masks for any object [43], 3D models of whole scenes [48,49], and descriptions in natural language [50]. And when vision becomes part of a larger system, the final product is often something people interact with: a practice conversation with a simulated patient [16,17], or, beyond vision, a working interface prototype [18]. That also changes how we judge success. Accuracy on a test set is no longer enough. We now also need user studies, expert reviews, and evidence that the system helps in real use.

6. What Did Not Change, and What Is Still Open

Not everything changed. Some lessons from a decade ago still hold, and some old questions have become more urgent.

Good data still matters.

Large models need fewer labels, but they still need data that looks like the real problem. Clinical images, farm photos, and phone sensor readings are often very different from the web images these models learn from. Carefully collected datasets for specific fields [4] are as valuable as ever.

Human behavior is still a powerful signal.

Ten years ago, eye movements, walking, and other behaviors were studied as signals on their own [2,5,6]. They are again at the center of clinical and educational systems [15,16]. The difference is that now they are one input among many, rather than the whole system.

Trust in medicine is still the hardest part.

Work on computational staining [11] and stain-free tumor detection [12] raised a basic question: when can a doctor trust an image that a computer generated or inferred? Since then, models have become far better at generating realistic images [52]. That makes the question more pressing, not less.

Three open questions.

First, what do these models actually know? We asked this of scene networks in 2017 [9,24], and the answer was not always where we expected: holistic scene layout lived in the earliest layer, not the deepest. It is harder and more important to answer for today’s much larger models. Second, do machines see the world the way we do? In 2018, CNN attention maps missed the out-of-place objects that drew human eyes [10]. Repeating those experiments with foundation models and vision-language models would show whether scale has closed that gap or only hidden it. Third, what about privacy? Our finding that a person can be identified from their eye movements while reading [6,7] matters more now that eye tracking is built into consumer headsets and large models can pick up such signals without being designed to.

7. Conclusions

Comparing 2017 with 2026 shows a field whose questions are familiar and whose methods are almost completely new. Models now learn mostly without labels. They are specialized with small add-ons instead of being retrained. They combine many kinds of signals by default. More and more, they are part of systems that people talk to and work with, rather than tools that simply hand back a label. Yet several of the questions we asked a decade ago are still open: what these models know, whether they see the world as we do, and what our behavior reveals about us. Scale did not answer them. It made them more important, and it gave us much better tools to study them.