SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture

Source-first digest for monthly 2026_05 rank 4, rank_id p011.

• Routing status: success
• PDF extraction: not used

Motivation / Background

Most recent multimodal systems still split understanding and generation into different mechanisms. Perception usually depends on pretrained vision encoders, while image generation usually depends on VAEs or other latent bottlenecks. SenseNova-U1 argues that this split is not only an implementation detail: it creates separate objectives, representation spaces, and deployment paths that make unified multimodal intelligence harder to obtain.

The paper's answer is NEO-unify, a native pixel-word architecture that removes separate VEs and VAEs, uses lightweight patch interfaces, and trains understanding and generation inside one Mixture-of-Transformers backbone. The intended outcome is a model that can read, reason, draw, edit, interleave text and images, and show early VLA/world-model behavior without routing across separate modality-specific systems. The high-level scope is summarized in Figure 1.

Claims And Evidence

Scores are support-from-paper scores, not independent reproduction scores. I cap claims when evidence is mainly qualitative, when baselines are mostly benchmark-specific, or when the result is inherited from a smaller NEO-unify ablation rather than measured directly on the final SenseNova-U1 variants.

Core Technical Idea

SenseNova-U1 makes three architectural moves:

• Near-lossless visual interface. Images/noise are converted into 32 by 32 patch tokens through two convolutional layers with GELU and 2D sinusoidal positions. The generation stream predicts pixel patches directly with an MLP-like head, rather than decoding through a VAE.
• Native pixel-word sequence modeling. Text tokens and image tokens are projected into one embedding space, delimited by image boundary tokens, and processed jointly.
• Native Mixture-of-Transformers. Clean understanding tokens and noisy generation tokens share attention context but use token-type-routed projections, normalizations, and feedforward blocks. This keeps the architecture unified while decoupling parameters enough to reduce interference.

The architecture is shown in Figure 2, and the two released variants are summarized in Table 1.

Table 1. Model configurations. This table comes from the LaTeX source table tab:model_config; it matters because the paper's main comparison is not only small versus large, but dense symmetric streams versus stream-wise MoE.

The training objective combines text understanding and pixel-space visual generation:

For understanding, the model uses autoregressive next-token loss over text conditioned on multimodal context:

For generation, it applies rectified-flow style pixel-space modeling. A clean image \(\mathbf{x}\) and Gaussian noise \(\boldsymbol\epsilon\) are interpolated with a resolution-adaptive noise scale \(\sigma_R\):

The model predicts a clean signal and converts it into a velocity:

The generation loss is:

For image-conditioned generation and editing, the paper uses a unified classifier-free guidance rule with separate text and image-context strengths:

The reported practical setting is \(\gamma = 4\), \(\gamma_{\mathrm{img}} = 1\), timestep shift 3.0, and global CFG renormalization.

Method Details

Training Recipe

The model is trained in a staged sequence, with pure understanding first, generation pretraining second, joint mid-training third, and unified SFT fourth. Table 2 compresses the stage table recovered from the flattened LaTeX source.

Table 2. Training stages. The important design is not simply "more data"; Stage 2 deliberately lets the generation branch learn pixel synthesis before the unified model is jointly optimized.

After SFT, the paper adds generation post-training. Dynamic resolution warmup gates candidate resolution sampling by difficulty:

Text-rendering RL uses OCR overlap:

For one reward group, OCR and style rewards are combined as:

The text-rendering RL run uses 600 epochs, learning rate \(1\times10^{-5}\), KL coefficient \(\beta=0.01\), \(N=48\) prompts per epoch, \(K=16\) samples per prompt, 10-step flow matching, guidance 4.0, and a 200-epoch resolution warmup. The later general RL stage interleaves text/style and aesthetic reward groups; the 8B variant trains for 1,600 epochs and the A3B variant for 200.

Inference Infrastructure

The paper keeps the user-facing model unified but splits serving into two specialized engines because understanding and generation have different runtime bottlenecks. Figure 3 shows the disaggregated LightLLM/LightX2V serving setup, and Figure 4 shows the hybrid attention kernel.

The infrastructure supports separate and colocated deployment. For \(2048 \times 2048\) generation with SenseNova-U1-8B-MoT, both modes support TP2+CFG2; in separate mode, the paper reports per-step latencies of 0.415 seconds on 5090 and 0.443 seconds on L40S GPUs.

Data Construction

The understanding corpus and generation corpus are both heavily filtered. Figure 5 shows the understanding pipeline, Figure 6 summarizes the corpus distribution, and Figure 7 shows the generation filtering pipeline.

For interleaved data, the paper reports a compact corpus spanning video, lifestyle, infographics, and reasoning. Lifestyle is about 44%, infographics about 29%, video about 19%, and reasoning about 8%, with the reasoning subset including explicit chain-of-thought traces.

Experiments And Results

Understanding, Text, Agent, And Spatial Results

The paper evaluates understanding with EvalScope under an LLM-as-a-judge llm_recall strategy, using gpt-4o-mini-2024-07-18 as judge. The setup uses temperature 0.6, top-p 0.95, top-k 20, repetition penalty 1.0, 40,960 max sequence length, 600 second timeout, and internal thinking enabled.

Table 3 condenses the multimodal understanding and spatial-intelligence table recovered from flattened LaTeX. The strongest evidence is not that SenseNova-U1 wins every column; it is that an encoder-free unified model stays close to or above strong understanding-only VLMs across diverse categories.

Table 3. Selected multimodal and spatial benchmarks. Asterisks mark Qwen variants that used 128 frames for best performance under the paper's EASI note; the standard protocol otherwise uses 32 frames on VSI-Bench.

Table 4. Selected text, instruction, and agent results. The instruction-following and agent rows are the clearest wins for SenseNova-U1 over Qwen3VL variants; dense reasoning-focused Qwen3.5 still leads on several knowledge rows.

Image Generation

For general generation, the paper reports GenEval, DPG-Bench, OneIG, TIIF, CVTG-2K, LongText-Bench, WISE, IGenBench, and BizGenEval. Figure 8 and Figure 9 show qualitative examples, while Table 5 and Table 6 summarize the strongest quantitative claims.

Table 5. Selected generation benchmarks. SenseNova-U1 is particularly strong on GenEval and TIIF. On DPG-Bench it is competitive but not the top overall row because Qwen-Image reports 88.32 and Seedream 4.5 reports 88.63.

Table 6. Text-centric generation. The 8B variant is the strongest SenseNova-U1 text-rendering model in these rows, especially on CVTG-2K and LongText-Bench.

Table 7. Infographic and business visual generation. IGenBench and BizGenEval are important because they test layouts, charts, data constraints, text rendering, and domain knowledge rather than only natural-image aesthetics.

Image Editing

The editing results are more nuanced. SenseNova-U1 is competitive among unified open-source models, but it does not beat the strongest dedicated editing systems such as Qwen-Image-Edit-2511 or Seedream in the general editing table. The paper explicitly attributes part of the gap to editing data limitations.

Table 8. General editing. The unified native model clears older unified baselines, but dedicated editors still lead, especially on ImgEdit and GEdit overall.

Table 9. Reasoning editing. CoT helps substantially on RISEBench. For example, A3B logical score moves from 7.1 to 20.0, and overall rises from 25.3 to 30.0.

Interleaved Generation And Unified Reasoning

Interleaved generation is where the paper most directly tests whether the model can coordinate language and images over multiple steps. Table 10 covers OpenING and VBVR-Image preview; Table 11 covers generation-aids-understanding and bidirectional synergy.

Table 10. Interleaved generation. These rows support the claim that SenseNova-U1 can sustain multi-step image-text coherence and reasoning-through-generation better than the compared unified baselines.

Table 11. Unified reasoning. Uni-MMMU measures generation assisting understanding. RealUnify separates Understanding Enhances Generation and Generation Enhances Understanding, which is why it is central evidence for C7.

Ablation Studies

The ablations address three questions: whether the encoder-free visual interface preserves detail, whether MoT co-training creates manageable interaction between objectives, and whether data scale improves both generation and unified capabilities. Figure 10, Figure 11, Figure 12, Figure 13a, Figure 13b, Figure 13c, and Figure 13d show the visual evidence.

Figures 13a-13d. Data-scaling curves. These curves cover DPG-Bench, WISE, GEdit-Bench, and RISEBench. The paper interprets them as evidence that diverse data scale improves generation quality and understanding-generation synergy.

Preliminary VLA And World Modeling

The final experiments are qualitative visualizations rather than full embodied evaluations. They are still useful because they show what the authors mean by a path from passive multimodal understanding toward acting and predicting state transitions.

Practical Takeaways

1. The real contribution is architectural unification, not just benchmark breadth. SenseNova-U1 tries to remove the VE/VAE split and force pixel-word understanding, image synthesis, editing, and interleaving through one native modeling interface.

2. Parameter decoupling inside a unified backbone is the pragmatic compromise. The model is "single architecture" at the sequence and attention level, but projections, normalizations, and feedforward blocks are stream-routed by token type. That choice is what makes the unification claim more credible than a purely shared-backbone design.

3. Text rendering and structured visual generation are unusually central. The strongest generation evidence is not only natural-image quality; it is GenEval, TIIF, CVTG-2K, LongText-Bench, IGenBench, BizGenEval, and WISE, which stress layout, text, knowledge, and reasoning.

4. Editing is good but not solved. SenseNova-U1 beats many unified baselines, but dedicated editors still lead on general editing. The paper's own limitation is that editing supervision is still dominated by public resources and lacks broader pipelines and preference optimization.

5. The VLA/world-model story should be treated as roadmap evidence. The figures are promising, but they are not a substitute for benchmarked embodied control or closed-loop world-model evaluation.

6. For follow-up work, the most interesting tests are interference and transfer. A strong reproduction would vary the MoT decoupling, VE/VAE removal, data mixture, and post-training rewards to test whether the claimed synergy survives outside this training stack.

Reference Coverage

• Additional local references for validation and navigation: evidence-data, evidence-inference, evidence-training, fig-vla, fig-world-modeling, table-editing.
• Additional local references for validation and navigation: table-infographic, table-reasoning-editing, table-text-agent.