From a Request to a Deliverable

After working on a content generation system for a while, I increasingly feel that the hardest part is not "making a model write something."

If all you need is for a large model to write a script or some copy from a topic, that is no longer surprising. The harder problem is different: a user gives you a requirement, uploads a mix of videos, images, documents, and scattered notes, and the system needs to return not just an answer, but a content deliverable that can continue into editing, layout, rendering, or even direct delivery.

This system also uses RAG underneath, but it is not traditional knowledge-base Q&A. Retrieval is only the entrance. It finds materials related to the task; after that, the system still needs task planning, content generation, asset matching, format cleanup, and engineering post-processing. In other words, the question here is not "is the answer correct?" but "can the final result be delivered?"

Below is a breakdown of the current RAG + Agent architecture for content generation.

What This System Actually Does

First, the positioning: this is a RAG + Agent system for content generation and delivery.

Traditional RAG is closer to Q&A: the user asks a question, the system retrieves relevant snippets from a knowledge base, puts them into a prompt, and lets the model answer. The key metrics are usually retrieval accuracy and answer correctness.

A content generation system faces a more complex task. For example, a user says, "Help me create a short video introducing an energy storage product for overseas markets," and uploads several video clips plus product documents. The system cannot just return generic marketing copy. At minimum, it needs to generate a short video script, storyboard plan, asset usage suggestions, and possibly an executable editing plan.

That means both the input and output are more complex than Q&A:

• The input is not only a text question, but also videos, images, documents, and plain text materials.
• The output is not a paragraph of text, but a business-facing content deliverable.

So the system cannot stop at "retrieval + answer." It needs to identify what the user wants to generate, retrieve context from multiple sources, let the model plan and generate, and finally let engineering code turn the result into a stable deliverable.

Compressed into one chain, it looks like this:

User requirement and materials -> task intent detection -> multi-source context retrieval -> reranking and assembly -> model planning and generation -> engineering post-processing -> delivered content.

What the Overall Pipeline Looks Like

Expanded, the full flow looks like this:

By responsibility, I split it into four layers:

Layer	Role	Representative capabilities
Code orchestration layer	Task detection, flow control, context assembly, post-processing	Business services, Agent scheduling, toolchain
Retrieval pipeline layer	Query rewriting, retrieval, merging, reranking	Qwen series, Embedding, Reranker
Large model generation layer	Agent planning, content generation	Gemini Pro tier
Tool and asset layer	Vectorization, multimodal asset understanding, web search	Gemini Flash tier, Qwen Embedding, Bing / Serper

Among these four layers, the most time-consuming parts are not "calling the model," but the retrieval pipeline and engineering orchestration. Model capabilities are public and everyone can call them. What really affects the result is whether the earlier steps can organize materials into usable context, and whether the later steps can turn model output into a controllable product.

Let's walk through the pipeline.

Retrieval Pipeline: Why It Is Not One Search

Many RAG demos use a very direct retrieval flow: "user input -> vector search -> topK." That can work in Q&A scenarios, but in content generation, retrieval quality becomes unstable.

User Wording Is Not Good Retrieval Input

User input is usually task-oriented, not search-oriented.

For example, when a user says, "Help me write a short video script introducing an energy storage product for overseas markets," the sentence itself has no industry dimension, no selling point dimension, and no structural dimension. If you use it directly for vector retrieval, the results are often scattered. You might retrieve generic "how to make short videos" content, but miss materials that can truly support the output, such as overseas user pain points for energy storage products.

So the first step is Multi-Query rewriting: use a mid-sized generation model to rewrite the original requirement into multiple retrieval queries, expanding recall from different angles.

That requirement might become queries like:

energy storage overseas market trends
overseas energy storage user pain points
energy storage product short video script structure
new energy brand overseas marketing copy
energy storage product application scenarios

This allows a single retrieval stage to cover industry background, user pain points, content structure, expression style, and application cases. Content generation does not need a single piece of evidence; it needs a material package that can support creation.

I use a mid-sized Qwen model for this step because it only rewrites queries and does not need complex reasoning. It is cheap, fast, and good enough.

Context Is Scattered Across Sources

The context needed for one generation task usually does not live in a single database.

We use three parallel retrieval paths:

• Private knowledge base vector retrieval: internal knowledge, product documents, and methodologies. This is the core path, built with Qwen Embedding.
• User asset library retrieval: videos, images, documents, and text uploaded by the user for this task. This path is special and worth discussing separately.
• Web search: real-time and public information, such as recent industry data and competitor updates, through Bing or Serper API.

These three paths run in parallel and are merged later. The private knowledge base keeps the content aligned with the business, web search fills the freshness gap, and the user asset library provides the raw materials that must be used in this generation. Missing any one of them makes the result thinner.

Multimodal Assets Need to Be Structured First

This is the biggest difference between this system and ordinary text RAG.

Uploaded videos and images cannot be directly searched as text. To make them retrievable later, they need multimodal preprocessing before entering the asset library:

• Videos generate summaries, scene tags, and keyframe descriptions.
• Images generate tags, subject descriptions, and style descriptions.
• Documents extract titles, paragraphs, summaries, and keywords.
• Plain text is chunked directly.

After preprocessing, the results are written into the vector store with Embedding and can be retrieved online later.

This step uses the Gemini Flash tier vision-language model. It has high throughput and low cost, making it suitable for tagging and summarization. It is also in the same family as the Gemini Pro tier model used later for generation, so the API is consistent and easier to integrate.

So the retrieval pipeline here is not simply topK retrieval. It rewrites queries, performs multi-source recall, and first turns multimodal assets into text representations that can be searched.

After Retrieval: Why Rerank Is Needed

Three retrieval paths solve the breadth problem, but they also introduce noise: different sources have different quality, different credibility, and duplicate semantics can appear repeatedly.

If all of these results are sent directly to the large model, the problems are obvious:

• Precise internal knowledge and generic web articles get mixed together, making it hard for the model to judge importance.
• Duplicate information wastes tokens.
• Irrelevant materials dilute the model's attention.

So a Rerank step is needed between retrieval and assembly.

Rerank does not generate content. It answers one question: among these candidate snippets, which ones are most worth putting into the final context?

Its role is different from vector retrieval. Vector retrieval is "rough recall," optimized for speed and breadth. Rerank is "fine ranking," comparing the query with each candidate snippet more carefully and moving truly relevant snippets to the front.

I use a Qwen-series Reranker here. A dedicated reranker is cheaper and more stable than using a generation model for temporary scoring, and it is less likely to improvise.

After Rerank, there is another easily underestimated step: context assembly. This is done in code. The goal is to turn a pile of retrieved snippets into prompt context the model can actually understand and use.

During assembly, I care about:

• Deduplication, to avoid wasting tokens on repeated information.
• Segmentation, to separate internal knowledge, user assets, and web information.
• Source labeling, so the model can judge credibility. Internal knowledge and web search results do not have the same trust level.
• Length control, so the prompt stays within the context window while preserving the most important evidence.

The model does not see the raw database. It sees the prompt assembled by code. If context assembly is rough, even a strong model can easily drift.

Generation and Delivery: The Model Only Produces a Draft

Once the context is assembled, the large model generation stage begins.

This stage has two phases.

Agent planning turns a broad task like "generate a short video" into an executable structure:

• Topic planning: what angle should this content take?
• Script structure: how should the opening, body, and ending be organized?
• Content splitting: how many sections or shots are needed?
• Storyboard planning: what does each shot say, and which asset should be used?
• Output type decision: short video, podcast, graphic post, or article.

Content generation turns that plan into concrete text:

• Body copy and marketing copy
• Short video script
• Storyboard descriptions
• Podcast narration script
• Graphic post content

I use the Gemini Pro tier for these two steps. Complex planning, long-context understanding, and multimodal generation all rely on it.

But in this system, the model output is not treated as the final product. It is more like a high-quality draft.

The scripts, storyboards, and copy generated by the model are not delivered directly to the user. They go through another engineering post-processing layer:

• Short video orchestration: combine storyboards, assets, and subtitles into an editing plan or even a rendered video.
• Graphic post layout: apply templates and adjust formatting.
• Podcast editing: split narration into usable segments.
• Article formatting: normalize structure and remove redundancy.
• Toolchain rendering: export final files users can use.

This layer is the most engineering-heavy part of the architecture. The model generates core content; business code turns that content into a stable, controllable, deliverable product. Without this layer, even a great model output is still just markdown, not what the user actually wants.

Model Division of Labor: Why Not Use One Model for Everything

From a capability perspective, a flagship model can handle more stages. So why put so many different models in the pipeline instead of letting one model do everything?

The core reason is that a production system must consider quality, cost, speed, and stability at the same time. Different stages have different task types, and each task type has a more suitable model.

Here is the current model division of labor:

Model family	Main responsibility	Stage
Qwen series (mid-sized)	Multi-Query rewriting	Before retrieval
Qwen Embedding	Knowledge base and asset library vectorization and retrieval	Vector retrieval
Qwen Reranker	Reranking retrieved results	Rerank
Gemini Flash tier	Multimodal asset preprocessing (video / image tagging and summarization)	Asset understanding
Gemini Pro tier	Agent planning and final content generation	Generation
Bing / Serper	External information from web search	External retrieval

The trade-off is straightforward:

• Embedding should use a dedicated vector model. Generation models are not designed for retrieval. Using them for vectorization lowers recall quality and costs more.
• Using a generation model for Rerank is expensive and slow. Rerankers are optimized for ranking and are faster and more stable.
• Multimodal asset understanding fits vision-language models. Tagging and summarization need throughput and low cost; they do not need a flagship model every time.
• Only complex planning and final generation need the flagship model. Expensive compute should be used where it matters most.

Overall:

Gemini handles multimodal understanding and generation, Qwen handles retrieval enhancement, and the code layer handles orchestration and final delivery.

With this split, cost, stability, and controllability are all better than forcing everything onto one model. The flagship model is mainly used where reasoning and generation quality matter most. Earlier stages that can run in batch, run in parallel, or be solved by cheaper models are handled by lightweight specialized models.

The Real Difficulties in This Architecture

If you only look at the architecture diagram, the pipeline seems like a few modules connected together. In practice, the hard parts mostly live in engineering details.

1. Structuring multimodal assets

Videos and images cannot be directly searched as text. They must first produce tags, summaries, keyframe descriptions, and scene information. The quality of preprocessing directly determines whether later retrieval can find the right materials.

2. Merging multi-source retrieval results

Private knowledge, user assets, and web search have very different quality and credibility. How to merge, deduplicate, label sources, and rank them affects the final context at every step.

3. Context control

You cannot put every retrieved result into the model. Within the token limit, you need to keep the most important information, while segmenting and labeling sources so the model can distinguish internal knowledge from web information.

4. Agent workflow stability

Different output types have completely different flows. Short videos, podcasts, graphic posts, and articles each require separate planning and post-processing chains. Complexity grows linearly with the number of supported deliverable types.

5. Engineering delivery of generated results

The model produces a content draft, but the draft still needs layout, editing, formatting, and rendering before it becomes something users can use. This layer often takes more engineering work than "getting the model call working."

These problems are unlikely to be solved by simply "switching to a stronger model." They need to be refined in the pipeline, data structures, prompt organization, and post-processing rules.

Conclusion

After building this system, my view of AI content generation has become more engineering-oriented.

The large model is certainly core, but it only handles the middle of the chain: understanding context, planning, and generating content. Before that, materials, knowledge, and real-time information need to be organized into usable context. After that, generated results need to become deliverable short videos, podcasts, graphic posts, or articles. The parts outside the model often decide whether the system can run reliably.

So I now prefer to think of it as a content production pipeline: retrieval prepares the materials, context assembly puts them in front of the model, the Agent plans and generates, and engineering post-processing turns the draft into a product. Models will keep getting stronger, but straightening out this pipeline and stabilizing every handoff point is the longer-term value of this kind of system.