Preprint
Concept Paper

This version is not peer-reviewed.

Frozen Small Language Models Pass Private Facts Better than Text

Submitted:

09 November 2026

Posted:

12 November 2026

You are already at the latest version

Abstract
Two copies of a frozen small language model share a task. One sees a private list of four people and their 7-letter codes. The other is asked for one person's code and can answer only from what the first sends. We give the sender a discrete latent channel: a message of 16 vectors, each one of 676 entries in a trained code-book, so the message carries exactly 150.4 bits. The sender's own hidden states choose the entries, and the receiver reads the vectors directly as input embeddings. Neither model is changed. We compare this channel with the best text we could find in 16 tokens (275.2 counted bits): the sender's own message, the raw list cut at 16 tokens, and five compact rewritings of the list. In Qwen3-0.6B the latent message is answered correctly .584 of the time, against .490 for the best text (the raw list) and .296 for the sender's own text (paired gap over the best text +.094, 95% CI [+.037, +.151]). Across three training seeds, the gap is +.103 [+.052, +.154]. In Qwen3-1.7B it is .659 against .490. A message chosen by the 0.6B sender, sent through the 1.7B receiver's code-book, scores .640, level with the 1.7B model talking to itself. On 500 fresh items held back from the start and scored once, the results hold: +.099 [+.046, +.152] pooled over the three 0.6B seeds, and +.151 [+.094, +.208] for the cross-model message. Four message controls (another item's message, zero vectors, random vectors, and a message built from the same list with the codes replaced) all score .000. The trained code-book places each letter where the receiver's own letter embedding points, with one code shared across the seven places. Probes show that the code read from the message appears at the same depth in the receiver as the code read from text, and that the receiver loads more of it ahead of writing.
Keywords: 
;  ;  ;  ;  

1. Introduction

Language models that work together usually talk in text. Text is easy to read and to audit, but it is a narrow channel: a token is one of about 151,000 symbols, and a model’s written message may not carry what its partner needs. A growing line of work lets models exchange hidden states or other latent vectors instead [1,2,3,4,5,6]. These works report gains, but they do not count the information a latent message carries, so a latent message and a text message are not compared at the same size. A recent causal audit [7] also showed that many reported gains disappear under proper controls and that a latent relay helps clearly only when the receiver needs facts that only the sender has.
This experiment asks one question: at the same number of bits, does a trained latent message carry more useful private information between frozen small models than text does? To answer this question, we need three things that earlier work did not combine:
1.
A task where the receiver can only succeed through the message. We use a registry task of the kind the audit found decisive: the sender sees four people and their random 7-letter codes; the receiver is asked for one code. With no message, the receiver scores . 000 .
2.
A message whose size is exact. Each message vector is an entry in a codebook of 676, so 16 vectors carry exactly 16 × log 2 676 = 150.4 bits. A text token counts log 2 151 , 669 = 17.2 bits, so 16 tokens of text are given 275.2 bits.
3.
Controls that rule out answers the message does not cause. We use the audit’s four relays [7]: another item’s message, zero vectors, moment-matched random vectors, and the sender’s message for the same list with every code replaced by a made-up one.
Both models remain frozen throughout. What is trained is a small interface: the code-book (what each entry looks like to the receiver) and a picker that reads the sender’s hidden states and chooses the entries. The text baselines get no training. The claim is therefore that a small trained interface between frozen models carries more than the best text they can exchange at the same size. The interface is structured: its slots follow the list’s layout, and it adds position markers. This was not an arbitrary choice. The less structured channels we tried (a plain codebook, a product codebook, free continuous vectors, choices trained through the receiver’s gradient) did not train on this frozen receiver (Section 5.7). The contribution is a small structured discrete interface that works, not latent messages in general.

1.1. Related Work

Communication between language models. Multi-agent systems built from language models usually coordinate in natural language: agents with distinct roles hold a dialogue [8], or an orchestrating model splits a problem across specialized agents and combines their answers [9,10]. Interlat [1] passes last-layer hidden states through a trained adapter to a fine-tuned actor; it reports no bits and no text at the matched size. LatentMAS [2], KVComm [3], Cache-to-Cache [4], CIPHER [5] and activation communication [6] exchange hidden states, caches or embeddings between models. Gazzaev et al. [11] is closest in setup: a frozen Qwen3-0.6B diffusion model, a trained adapter, facts split across private contexts, and a full control set; its messages are continuous and uncounted, and it names a matched text comparison as future work. HyLaT [12] mixes latent and text messages in a LoRA-tuned model, compared at unmatched token counts. Learned continuous inputs to a frozen model are also the basis of prompt tuning [13] and multimodal adapters that feed visual features to a language model [14,15]; here they carry item-specific facts from another model.
Counting the channel. Anand et al. [16] have frozen models communicate over an exactly counted 12-bit discrete channel found by prompt optimization, with no latent arm and no text arm at matched bits. Wenzel [17] estimates rough bit costs for single dense and sparse vectors passed between 7–8B models, with nothing trained. Discrete bottlenecks are standard in representation learning [18] and in emergent communication between neural agents [19]; we use one to make the message size exact. Our coded-bit count follows the view of a language model as a compressor [20,21].
Audits and probes. Cheng, Das and Ramnath [7] audit LatentMAS, KVComm and C2C with the four relays we use and find that relay helps only when the receiver needs sender-private facts, on a 4-person, 7-letter registry.  [22,23] probe LatentMAS states and find mostly base-model geometry; they argue that probes must be read against the plain model’s own states, which Section 7 does. Our probes are linear [24] with a shuffled-label control [25].
No work compares text at k tokens with a discrete latent message at exactly counted bits, on a private-fact task, between frozen models under 2B parameters, with the audit’s controls.

2. Task, Models and Baselines

Task (T1). Each item is a list of four people, each with a random 7-letter code, for example:
Ines: tlknpod    Grace: ftvqdtu    Lena: dicqszy    Omar: dggjdzy
(one person per line). The names come from a fixed set of 26 with 26 different first letters. The sender sees the list; the receiver sees only the question (“what is the code of Grace?”) and the message, inside a fixed prompt that says its partner saw a list of four people with secret codes. A reply is right if its first 7-letter word is the asked code.
Models. Qwen3-0.6B and Qwen3-1.7B [26], official checkpoints, non-thinking mode, frozen. In the main runs, sender and receiver are two copies of the same model. In the cross-model runs the sender is 0.6B and the receiver 1.7B.
Text baselines at 16 tokens. All on the same 500 scored items, with no training:
  • Sender’s text: the sender is shown the list, told its partner will be asked for one person’s code, and writes a message cut at 16 tokens.
  • Raw list: the sender’s list itself, cut at its first 16 tokens.
  • Compact text: the list rewritten to fit more people into 16 tokens: full names and codes without punctuation (Ines tlknpod Grace ftvqdtu …), first letters and codes (I tlknpod G ftvqdtu …), first letters joined to codes (Itlknpod Gftvqdtu …); the two first-letter forms again with one line in the receiver’s prompt explaining them; and the sender’s own message after one worked example of the first-letter form.
The bar a latent message must beat is the best of these at the same counted size. In short: the compact forms put more codes into the message, but the receiver reads them poorly, so the raw list stays the best text (Table 1).
The compact forms fit more people into 16 tokens (first letters fit three of four), but the frozen receiver does not read them: with first letters it answers . 06 to . 23 of the items whose code is in the message, against . 66 to . 77 with full names. On every model and item set the best text is the raw list, so it is the bar. The latent message has 150.4 bits; the bar has 275.2 counted bits.
Coded bits. Counting a token at 17.2 bits is generous to text. A tighter count is the message’s code length under the model itself: the same frozen model (each model for its own messages) is given only the public task brief in its chat format, with no item, and we sum − log 2 p ( token ) over the message’s tokens. This is the length an arithmetic coder using the model as its predictor would need [20]: an upper bound on the bits a model that already knows the public task needs to encode the message. Table 2 gives the mean of the 500 messages.
The latent message is smaller than the bar in this count as well. Coded bits overstate what text carries: the raw list cut at 16 tokens holds about two people, about 75 bits of the list’s information, yet costs 171.6 coded bits because the model predicts random letters poorly. The count is therefore an upper bound that favors text.

3. Method

3.1. The Channel

The message is 16 vectors placed in the receiver’s prompt where a text message would go, as input embeddings, each at the mean norm of the receiver’s token embeddings. The receiver reads them with no change to its weights.
The list has 32 slots: for each of the four people, a name and seven letters. Each slot role (name, letter 1, …, letter 7) has its own table of 26 learned pieces, shared by the four people. A message vector joins the pieces of two neighboring slots, each piece filling half of the vector, so one person takes four vectors and the list takes 16. Every vector is one of 26 × 26 = 676 combinations, so the message carries exactly 150.4 bits, which is all the information in the list ( 4 × ( log 2 26 + 7 log 2 26 ) ). Each of the 16 positions also adds one learned marker vector, the same for every item, which tells the receiver where it is in the message; it carries no information about the item, so it adds no bits. The vector is rescaled to the token norm after the marker is added.
The channel is built for the task: its slots follow the list’s layout. Section 5.7 shows that less structured channels did not train.

3.2. Training the Code-Book

The code-book (the 8 tables of 26 pieces and the 16 markers) is trained through the frozen receiver. During this training, each item’s own names and letters choose the pieces, the message is built, and the receiver’s cross-entropy on the right code is back-propagated into the pieces and markers. Pieces start at random, and markers at zero; both train together in one run. Training uses 384,000 fresh lists (item seed 1) and 200 development lists (seed 2) to early stopping on the share answered right; AdamW [27], learning rate .05, batch 32, a check every 250 steps, stopped after 2,000 steps without a better check, keeping the best check.
When the item’s own letters choose the pieces, the receiver answers . 608 (0.6B) and . 681 (1.7B). These scores measure the code-book with every choice right and are reported beside the picker below as letters.

3.3. The Picker: The Sender’s States Choose the Message

The sender model reads its list. A picker reads the sender’s hidden states and chooses the 32 pieces; the code-book turns the choices into the 16 vectors.
Input: the frozen sender’s layer-1 hidden states over each person’s line (up to 11 tokens), standardized with statistics from 2,000 training lists. Layer 1 was chosen by an earlier linear probe of the 0.6B sender reading 3-letter codes, where it was the best layer ( . 974 of held-out codes with all three letters right); the same layer is used on 1.7B.
Network: A small network shared by the four people maps a person’s line to 8 × 26 scores, one row per slot; the highest score in each row is the choice.
Training: supervised, with the piece that belongs in each slot (the person’s name and letters) as the target, on 160,000 training lists, AdamW at 3 × 10 − 4 , batch 64, dropout .2, weight decay .05, early stopping on development choices with patience 2,000 steps. The sender, the receiver, and the code-book remain frozen.
Because a message is a list of 32 piece indices, the same choices can be sent through another receiver’s code-book. The cross-model runs send the 0.6B picker’s indices through the 1.7B code-book.

3.4. Cost

Table 3 lists what is trained and how long it takes, on one RTX 3090 Ti, to measure. The code-book is trained once per receiver and then shared; the cross-model result shows that a picker trained on one model’s code-book can address another’s. Neither model’s weights change.

4. Evaluation

Items. Every score is on the screen’s 500 items (seed 0), so each item pairs with its own bar. A further 500 items (seed 3) were set aside when the plan was written; the latent messages were scored on them once, at the end (Section 5.4), and the text baselines were scored on them for their bars. No scored list appears in training or development.
Scoring. Latent messages and compact text are answered 8 times per item with sampling (temperature .7, top-p .8, top-k 20); an item’s score is the share of its 8 replies that are right. The sender’s text and the raw list come from the screen run before any training, with one reply per item at the same settings. A condition’s score is the mean over items.
Gap is the mean over items of (item score − item bar), with its standard error over items, written gap± SE. The 95% interval is gap ± 1.96 SE, and p is from the paired test of gap = 0 .
Controls. Each run scores, on the same items, four relays [7] (Table 4). On intact-no-private we also record how often the reply is the made-up code: the receiver follows the message.
Run checks. Every run checks that its scored items equal the screen’s, that each message is exactly the code-book’s vectors for its choices (largest gap 3 × 10 − 8 ), that loaded weights equal the saved ones (gap 0), and that the 500 messages are distinct. All code is covered by unit tests (512), and deliberate faults planted in each new script were caught by the tests before any run. Text runs use vLLM [28].

5. Results

5.1. Qwen3-0.6B

Table 5 gives the main result. Controls for the picker’s message: swap .000, zero .000, random .000, intact-no-private .000; on intact-no-private the receiver writes the made-up code . 562 of the time. The picker chooses the right name . 9995 of the time and each letter . 984 to . 995 of the time; all 32 choices are right on . 808 of items. Its message scores . 024 ± . 010 below the letters’ message, the cost of its wrong choices (Section 5.6).

5.2. Three Training Seeds

The code-book and the picker were retrained from new starts and new orders of training data (seeds 1 and 2) and scored on the same items (Table 6). Pooled: each item’s mean over the three seeds against its bar; the spread of the picker’s score across seeds is . 020 (sd). All 54 control scores across the three seeds are . 000 .

5.3. Qwen3-1.7B and the Cross-Model Message

The code-book and picker were trained on 1.7B in the same way. The cross-model message is the 0.6B picker (seed 0), reading the 0.6B sender’s states, with its choices sent through the 1.7B code-book to the 1.7B receiver (Table 7). Only the 32 choices cross between the models, not vectors: the two models have different widths (1024 and 2048), and the 1.7B code-book turns the 0.6B picker’s choices into its own vectors. The two code-books agree on which piece means which letter because both are indexed by the list’s own names and letters.
Item by item, the cross-model message is level with the 1.7B model’s own ( − . 012 ± . 009 ). The two pickers make identical choices on . 740 of items, and where they differ they differ by 1.25 pieces on average. All controls score . 000 ; p ≤ 6 × 10 − 7 for every gap in Table 7.

5.4. The Untouched Set

The 500 items of seed 3 were set aside before any work began. The screen was run on them for their own bars (raw list . 464 on 0.6B, . 466 on 1.7B; sender’s text . 290 and . 400 ; the best compact text . 444 and . 404 ), and every frozen latent sender was scored once. The run refused to score them a second time. Scores and bars are both slightly lower on these items, and the gaps are about the same (Table 8). Every control scores . 000 .

5.5. Message Size

The trained 16-vector message holds the four people in order, four vectors each, so its first 8 vectors are the first two people and its first 4 vectors the first person. Cutting it gives messages of 75.2 and 37.6 bits, each compared with the best text at 8 and 4 tokens (the raw list in every case; Table 9). The cut messages are read by a receiver whose code-book was trained on whole messages. At 8 vectors the 0.6B message ties the bar and the cross-model message falls below it.
At 32 tokens the raw list fits whole and scores . 962 (0.6B) and . 970 (1.7B). The 16-vector message already carries the whole list, so a longer latent message adds no information on this task. The latent message wins where the text budget is too small to hold the facts.

5.6. When the Latent Message Misses

We recomputed the picker’s choices for every scored item (they match the run’s record exactly) and sorted items by what it got wrong (Table 10). The picker’s whole cost comes from the 4% of items where it chooses a wrong piece for the asked person. There the receiver writes the wrong letter it was sent: . 83 of the time on both models, against . 02 to . 03 for the true letter. Elsewhere the picker’s message scores as the letters’ message does.
The remaining misses are the receiver’s own: replies with one letter wrong ( . 090 on 0.6B, . 099 on 1.7B), two wrong ( . 095 , . 096 ), three or more ( . 067 , . 077 ), or no 7-letter word, mostly a dropped letter ( . 161 , . 067 ). Another listed person’s code is almost never given ( . 002 , . 003 ). Wrong letters fall late in the code: places 1 and 2 hold 3 to 4% of them, and they peak at place 5.
The picker’s own errors are mostly position slips: of its wrong letter choices (all people), . 75 on 0.6B and . 79 on 1.7B are the true letter of the next or previous place in the same code (about . 08 by chance). Table 11 sets receiver slips (every piece right for the asked person) beside picker errors (a wrong piece sent). With a right message the receiver mostly writes the code and sometimes slips; with a wrong piece it writes what it was sent.
So the message carries the whole list exactly, and what limits the score is the frozen receiver reading it: with every choice right it scores . 60 (0.6B) and . 69 (1.7B), against . 98 and . 99 for the full list as text.

5.7. Less Structured Channels

Before settling on this channel we tried less structured ones on the same frozen receiver, mostly on a smaller form of the task (one 3-letter code):
  • One plain code-book entry per message (151,643 entries, 17.2 bits): . 951 on codes seen in training, . 002 on unseen codes, which land on a trained neighbor’s entry.
  • A product code-book trained through the encoder (3 parts of 53, 17.2 bits): . 106 on unseen codes, . 147 with unused entries reset, below the best single token ( . 260 ). Pieces tied to letter places instead (26 per place, 14.1 bits) scored . 898 .
  • Choices trained by the receiver’s gradient (straight-through [29], Gumbel-softmax [30], an entropy bonus): . 000 ; the encoder sent one message for every code.
  • Free continuous vectors on T1 (no bit limit): no right answers in 30 minutes of training.
  • T1’s slot message without the position markers: . 521 , level with the bar ( + . 031 ± . 029 ). Dropping the trained markers at test: . 000 .
The structure of the channel, pieces tied to slots plus position markers, is what made it trainable here.

6. What the Pieces Encode

The pieces start at random and are trained only through the frozen receiver. The four trained code-books (0.6B seeds 0, 1, 2 and 1.7B) agree on the following.
Letter pieces sit where the receiver’s own letter tokens point. For each place, compared within the half of the vector it fills, a letter piece’s nearest of the 26 letter-token embeddings is its own letter . 77 to 1.00 of the time (chance . 038 ); letter 1 is 1.00 in every code-book.
One letter code serves all seven places. A letter’s piece at one place is nearest the same letter’s piece at another place . 96 to 1.00 of the time; same-letter cosine + . 36 to + . 45 , different letters about 0.
Similar letters cluster as the receiver’s letters do. The 26 × 26 cosine table of a place’s pieces correlates with that of the letter tokens at + . 30 to + . 58 ( p < . 001 by label shuffle, every place and code-book).
Name pieces lean towards their names. The nearest of the 26 name tokens is the piece’s own name . 46 to . 54 of the time.
The vectors sent are not tokens. A whole message vector’s nearest token has cosine . 12 to . 19 and is almost never its name or letter.
The markers mark the place within a person. They are large, 1.11 to 1.30 times the token norm, larger than the content they are added to. Markers at the same place in different people’s blocks are alike (cosine + . 33 to + . 41 ); markers at different places in the same person’s block are less so ( + . 14 to + . 16 ). So a marker mainly says which pair of slots a vector holds (name and letter 1, or letters 6 and 7), not whose it is. They are not tokens either: nearest-token cosine . 13 to . 17 , to unrelated tokens. Without them the message only ties the bar ( . 521 ), and removing them at test gives . 000 .
The 0.6B and 1.7B code-books share their letter geometry. Their vectors cannot be compared directly (different widths), but their 26 × 26 similarity tables can: for the same place, the 0.6B and 1.7B tables correlate + . 39 to + . 72 ( p < . 001 at every place). Two 0.6B seeds correlate + . 59 to + . 85 , so the shared structure is stronger within a model than across models, yet still well above chance. Letters that are alike in one model’s code-book are alike in the other’s.
The code-book learned to reuse the receiver’s existing letter directions, combined into vectors unlike any token.

7. Inside the Receiver

We probed the frozen receiver to see where the asked code is held once it has read the message, and compared this with the same receiver reading the list as text in the same prompt.
Probe. A linear probe [24] per layer (all 29 hidden states) reads the asked code’s 7 letters from the receiver’s state at the last prompt token, from which it writes the first letter. Probes are trained on 4,000 fresh lists (seed 4) and read on the 500 scored items. Conditions: message, text (the full list written into the message slot), no message, another item’s message, and, on 1.7B, the cross-model message. A probe trained with shuffled labels is the capacity control [25].
Depth. On both models the code becomes readable at the same layers with the message and with text: little before layer 16, a sharp rise near layer 20, a peak at layers 21 to 23 (Table 12). A probe trained on text states reads message states from layer 20 on at about half the message probe’s level, so the two share part of the form in which the code is held. Inside the 1.7B receiver the cross-model message and the 1.7B model’s own message give states with cosine . 99 to 1.00 at every layer.
Writing step by step. The receiver writes a code as 3 to 6 tokens (ftvqdtu is written ft, v, q, dt, u). We fed the right code in token by token and probed the state before each of the first four tokens. At each step a letter is next (in the token about to be written), later (in a token after it) or written (already in the input). With text, the receiver holds the next token’s letters and little beyond: it fetches the code from the list as it writes. With the message, it holds the next token and about half of the later letters before writing begins: it loads more of the code at once (Table 13). Shuffled-label probes read . 04 to . 07 at every step; written letters, which are in the input, read . 56 to . 98 .

8. Scope

The experiment covers one synthetic private-fact task, two models of one family (Qwen3-0.6B and 1.7B), and a message of up to 16 vectors. The channel’s slots follow the task’s layout, and the interface is trained: the code-book through the frozen receiver, the picker with the right piece for each slot as its target. Text baselines are not trained. The code-book is the shared language between sender and receiver, including across the two models. The latent message wins when the text budget is too small to hold the facts; at 32 tokens the raw list holds them all, and text wins. What limits the latent score is the frozen receiver’s reading of a message that carries the whole list.

9. Conclusion

A frozen small model’s own hidden states, choosing 16 entries of a small structured code-book, pass private facts to a frozen partner better than the best 16-token text we found, using 150.4 bits against 275.2 counted bits (171.6 to 204.8 coded bits). The result holds across three training seeds, on a larger receiver, from a smaller sender to a larger receiver, and on 500 fresh items scored once, with every message control at zero. The structure (slots that follow the list, plus position markers) is what made the interface trainable; less structured channels did not train on the frozen receiver. Compact rewrites of the list fit more in 16 tokens, but the frozen receiver cannot read them. The trained code-book reuses the receiver’s own letter directions, one code for every place, and inside the receiver the message’s content sits at the same depth as text’s while more of it is loaded at once.

References

  1. Du, Z.; Wang, R.; Bai, H.; Cao, Z.; Zhu, X.; Cheng, Y.; Zheng, B.; Chen, W.; Ying, H. Enabling agents to communicate entirely in latent space. In Proceedings of the Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026, pp. 27106–27129.
  2. Zou, J.; Qiu, R.; Li, G.; Yang, X.; Tieu, K.; Lu, P.; Shen, K.; Tong, H.; Choi, Y.; He, J.; et al. Latent collaboration in multi-agent systems. arXiv preprint arXiv:2511.20639 2025.
  3. Shi, X.; Chiesa, M.; Maguire Jr, G.Q.; Kostic, D. KVComm: Enabling efficient LLM communication through selective KV sharing. In Proceedings of the International Conference on Learning Representations, 2026, Vol. 2026, pp. 84546–84573.
  4. Fu, T.; Min, Z.; Zhang, H.; Yan, J.; Dai, G.; Ouyang, W.; Wang, Y. Cache-to-cache: Direct semantic communication between large language models. In Proceedings of the International Conference on Learning Representations, 2026, Vol. 2026, pp. 43130–43158.
  5. Pham, C.; Liu, B.; Yang, Y.; Chen, Z.; Liu, T.; Yuan, J.; Plummer, B.; Wang, Z.; Yang, H. Let models speak ciphers: Multiagent debate through embeddings. In Proceedings of the International Conference on Learning Representations, 2024, Vol. 2024, pp. 51899–51928.
  6. Ramesh, V.; Li, K. Communicating activations between language model agents. arXiv preprint arXiv:2501.14082 2025.
  7. Cheng, J.; Das, S.; Ramnath, R. When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs. arXiv preprint arXiv:2608.04893 2026.
  8. Rasal, S. Llm harmony: Multi-agent communication for problem solving. arXiv preprint arXiv:2401.01312 2024.
  9. Rasal, S.; Hauer, E.J. Navigating complexity: Orchestrated problem solving with multi-agent llms. arXiv preprint arXiv:2402.16713 2024.
  10. Rasal, S. A Multi-LLM Orchestration Engine for Personalized, Context-Rich Assistance. arXiv preprint arXiv:2410.10039 2024.
  11. Gazzaev, A.B.; Gavrilov, A.; Muravyov, S.; Makarov, I. Hidden Messages Between Diffusion Language Model Agents: Causal Evidence for Latent Communication. arXiv preprint arXiv:2610.28905 2026.
  12. Mou, X.; Wang, S.; Li, Z.; He, Y.; Wei, Z. HyLaT: Efficient Multi-Agent Communication via Hybrid Latent-Text Protocol. arXiv preprint arXiv:2605.25421 2026.
  13. Lester, B.; Al-Rfou, R.; Constant, N. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Proceedings of the 2021 conference on empirical methods in natural language processing, 2021, pp. 3045–3059.
  14. Li, J.; Li, D.; Savarese, S.; Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the International conference on machine learning. PmLR, 2023, pp. 19730–19742.
  15. Rasal, S.; Boddhu, S.K. Beyond Segmentation: Road Network Generation with Multi-Modal LLMs. arXiv preprint arXiv:2310.09755 2023.
  16. Anand, V.; Chandrasekaran, M.; Chaitanya, S. Communication between Frozen Large Language Models via Prompt Optimization in a Referential Game. arXiv preprint arXiv:2609.31989 2026.
  17. Wenzel, M. Latent Communication Between Language Model Agents: Channels, Alignment, and the Limits of Text. arXiv preprint arXiv:2607.14103 2026.
  18. Van Den Oord, A.; Vinyals, O.; et al. Neural discrete representation learning. Advances in neural information processing systems 2017, 30.
  19. Lazaridou, A.; Baroni, M. Emergent Multi-Agent Communication in the Deep Learning Era. arXiv preprint arXiv:2006.02419 2020.
  20. Delétang, G.; Ruoss, A.; Duquenne, P.A.; Catt, E.; Genewein, T.; Mattern, C.; Grau-Moya, J.; Wenliang, L.K.; Aitchison, M.; Orseau, L.; et al. Language modeling is compression. In Proceedings of the International Conference on Learning Representations, 2024, Vol. 2024, pp. 14165–14181.
  21. Shannon, C.E. A mathematical theory of communication. The Bell system technical journal 1948, 27, 379–423.
  22. Lu, W.; Deshpande, S. What Do Latent Agents Actually Represent? Interpreting Hidden-State Communication in Multi-Agent Systems. ICML 2026.
  23. Deshpande, S.; Lu, W. On the Role of Learned Alignment Matrices in LatentMAS. In Proceedings of the 2nd Workshop on Compositional Learning: Safety, Interpretability, and Agents, 2026.
  24. Alain, G.; Bengio, Y. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 2016.
  25. Hewitt, J.; Liang, P. Designing and interpreting probes with control tasks. In Proceedings of the Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), 2019, pp. 2733–2743.
  26. Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388 2025.
  27. Loshchilov, I.; Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
  28. Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y.; Zheng, L.; Yu, C.H.; Gonzalez, J.; Zhang, H.; Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the Proceedings of the 29th symposium on operating systems principles, 2023, pp. 611–626.
  29. Bengio, Y.; Léonard, N.; Courville, A. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv preprint arXiv:1308.3432 2013.
  30. Jang, E.; Gu, S.; Poole, B. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 2016.
Table 1. Text baselines at 16 tokens: share answered right, and the share of items whose asked person is wholly in the message.
Table 1. Text baselines at 16 tokens: share answered right, and the share of items whose asked person is wholly in the message.
At 16 tokens 0.6B 1.7B Asked person in
Sender’s text .296 .434
Raw list .490 .490 .494
Full names, no punctuation .476 .409 .618
First letters .178 .059 .782
First letters joined .001 .000 .798
First letters, prompt line .160 .060 .782
First letters joined, prompt line .001 .001 .798
Sender after a worked example .091 .143
Full list, uncut .984 .990 1.000
No message .000 .000
Table 2. Coded bits at 16 tokens (mean over 500 messages).
Table 2. Coded bits at 16 tokens (mean over 500 messages).
At 16 tokens 0.6B 1.7B
Raw list (the bar) 171.6 204.8
Sender’s text 124.5 200.2
Latent message (exact) 150.4 150.4
Table 3. Trained parameters and measured training time.
Table 3. Trained parameters and measured training time.
0.6B 1.7B
Code-book, trained numbers 0.12M 0.25M
Picker, trained numbers 4.4M 4.7M
Trained share of one model 0.8% 0.3%
Code-book training 34.9 min 48.5 min
Picker: sender states 2.6 min 5.3 min
Picker: training <1 min <1 min
Table 4. Message controls.
Table 4. Message controls.
Control What the receiver gets A non-zero score would mean
Swap another item’s message answers that do not depend on this item’s message
Zero 16 zero vectors answers from the prompt alone
Random 16 random vectors with the message’s mean and spread answers from the presence of vectors
Intact, no private the sender’s message for the same list with every code replaced answers that do not use the private codes
Table 5. Qwen3-0.6B on the 500 scored items. Gaps are paired, over items.
Table 5. Qwen3-0.6B on the 500 scored items. Gaps are paired, over items.
Message Bits Score Gap over the bar (.490) 95% CI p
Latent, picker on the sender’s states 150.4 .584 + . 094 ± . 029 [ + . 037 , + . 151 ] .001
Latent, letters choose 150.4 .608 + . 118 ± . 029 [ + . 061 , + . 175 ] 5 × 10 − 5
Raw list, 16 tokens (the bar) 275.2 .490
Sender’s text, 16 tokens 275.2 .296
Table 6. Three training seeds, Qwen3-0.6B.
Table 6. Three training seeds, Qwen3-0.6B.
Seed Letters Picker Gap over .490 95% CI
0 .608 .584 + . 094 ± . 029 [ + . 037 , + . 151 ]
1 .613 .578 + . 088 ± . 028 [ + . 033 , + . 143 ]
2 .631 .615 + . 125 ± . 028 [ + . 070 , + . 180 ]
Pooled .617 .593 + . 103 ± . 026 [ + . 052 , + . 154 ]
Table 7. Receiver Qwen3-1.7B. Gaps over the bar (.490).
Table 7. Receiver Qwen3-1.7B. Gaps over the bar (.490).
Message Score Gap 95% CI
Letters .681 + . 192 ± . 028 [ + . 137 , + . 247 ]
1.7B picker .659 + . 169 ± . 029 [ + . 112 , + . 226 ]
Cross (0.6B picker) .640 + . 150 ± . 030 [ + . 091 , + . 209 ]
Table 8. The untouched set (seed 3), scored once. Gap over its raw-list bar; last column: the same message on the scored items.
Table 8. The untouched set (seed 3), scored once. Gap over its raw-list bar; last column: the same message on the scored items.
Message Score Gap 95% CI Scored
0.6B picker, seed 0 .540 + . 076 ± . 029 [ + . 019 , + . 133 ] + . 094
0.6B picker, seed 1 .576 + . 112 ± . 029 [ + . 055 , + . 169 ] + . 088
0.6B picker, seed 2 .572 + . 108 ± . 029 [ + . 051 , + . 165 ] + . 125
0.6B picker, pooled .563 + . 099 ± . 027 [ + . 046 , + . 152 ] + . 103
Cross (0.6B to 1.7B) .618 + . 151 ± . 029 [ + . 094 , + . 208 ] + . 150
1.7B picker .623 + . 157 ± . 029 [ + . 100 , + . 214 ] + . 169
Table 9. Message size. Gaps with 95% CI.
Table 9. Message size. Gaps with 95% CI.
Size Bar (0.6B / 1.7B) Picker, 0.6B, pooled Cross, 1.7B
16 vectors .490 / .490 .593 .640
150.4 bits + . 103 [ + . 052 , + . 154 ] + . 150 [ + . 091 , + . 209 ]
8 vectors .222 / .226 .206 .127
75.2 bits − . 016 [ − . 053 , + . 021 ] − . 100 [ − . 139 , − . 061 ]
4 vectors .002 / .000 .086 .049
37.6 bits + . 084 [ + . 064 , + . 104 ] + . 049 [ + . 033 , + . 065 ]
Table 10. Items by what the picker got wrong: n, picker score, letters-choose score.
Table 10. Items by what the picker got wrong: n, picker score, letters-choose score.
Items 0.6B 1.7B
All 32 right 404, .601, .598 431, .695, .690
Others’ slots wrong 74, .659, .633 48, .628, .638
Asked person’s wrong 22, .017, .705 21, .000, .607
Table 11. Typical misses on 0.6B (8 replies each).
Table 11. Typical misses on 0.6B (8 replies each).
Kind Asked Sent Replies
Receiver slip ekztfyz ekztfyz ezktfyz×5, ekztfyz×2, ekztfxz
Receiver slip osfgrdu osfgrdu osfgrrdu×6, osfgrdu, ofgrrdu
Receiver slip sjpfjyl sjpfjyl sjpfjyl×5, sjpjflyl×3
Picker error nwsvniz nwsvviz nwsvviz×7, nwsvvis
Picker error mjemmer mjember mjember×8
Picker error ajnsiap ajnssap ajnssap×5, ajnszap×2, ajnsrap
Table 12. Best layer, share of letters right (chance .038).
Table 12. Best layer, share of letters right (chance .038).
0.6B 1.7B
Message .680 (layer 21) .642 (layer 22)
Cross-model message .639 (layer 22)
Text .425 (layer 23) .475 (layer 22)
Text probe on message .339 .342
None / swap / shuffled .045 / .050 / .045 .046 / .049 / .048
Table 13. Best layer, share right (0.6B / 1.7B), by writing step.
Table 13. Best layer, share right (0.6B / 1.7B), by writing step.
Next Later
Message, before the first token .996 / .991 .583 / .541
Message, before the second token .963 / .946 .476 / .488
Text, before the first token .930 / .963 .286 / .349
Text, before the second token .706 / .733 .199 / .224
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.