After working on a content generation system for a while, I've found the hard part isn't making a model write something. That stopped being interesting a while ago. The hard part is when a user hands over a requirement plus a pile of videos, images, documents, and random notes, and the system has to return something that can be edited, laid out, rendered, and shipped — not just a text answer.

The system uses RAG under the hood, but it's not the knowledge-base Q&A kind. Retrieval is just the entry point: it pulls out materials related to the task. After that come planning, generation, asset matching, formatting, and engineering post-processing. So the question I care about isn't "did it answer correctly," it's "can we actually deliver the result."

Here's the current pipeline, as it actually runs.

How it differs from normal RAG

To be clear: this is a RAG + Agent system for content generation and delivery.

Traditional RAG is Q&A. The user asks, the system pulls snippets from a knowledge base, stuffs them into a prompt, and the model answers. The metrics people watch are retrieval accuracy and answer correctness.

Content generation is messier. Say a user asks for "a short video about an energy storage product going overseas," and uploads a few video clips and product docs. The system can't just return some generic marketing copy. It needs a script, a storyboard plan, asset suggestions, and ideally an editing plan someone can actually run.

The inputs and outputs are both bigger than Q&A:

• Input includes videos, images, documents, and plain text, not just a question.
• Output is a business deliverable, not a paragraph.

So the pipeline can't stop at "retrieve + answer." It has to figure out what the user wants, pull context from several sources, let the model plan and generate, and then let code turn that into a stable deliverable.

Squashed into one line:

User requirement and materials → intent detection → multi-source retrieval → rerank and assembly → model planning and generation → engineering post-processing → delivered content.

The full pipeline

Expanded, it looks like this:

By responsibility, I break it into four layers:

Layer	Role	Representative capabilities
Code orchestration layer	Task detection, flow control, context assembly, post-processing	Business services, Agent scheduling, toolchain
Retrieval pipeline layer	Query rewriting, retrieval, merging, reranking	Qwen series, Embedding, Reranker
Large model generation layer	Agent planning, content generation	Gemini Pro tier
Tool and asset layer	Vectorization, multimodal asset understanding, web search	Gemini Flash tier, Qwen Embedding, Bing / Serper

The most time-consuming part isn't calling the model. It's the retrieval pipeline and engineering orchestration. Model capability is a commodity now; what matters is whether the earlier steps can turn materials into context the model can use, and whether the later steps can turn model output into a controllable product.

Retrieval: why one search isn't enough

A lot of RAG demos do "user input → vector search → topK." That works for Q&A, but retrieval quality falls apart in content generation.

The user's sentence is not a good query

Users write task descriptions, not search queries.

"Help me write a short video script about an energy storage product going overseas" has no industry angle, no selling-point angle, no structural angle. If you feed that sentence straight into vector search, you get scattered results: generic "how to make short videos" articles, but not "overseas energy storage user pain points" — the material that would actually carry the script.

So the first step is Multi-Query rewriting: a mid-sized generation model rewrites the requirement into several queries to widen recall.

The sentence above becomes something like:

energy storage overseas market trends
overseas energy storage user pain points
energy storage product short video script structure
new energy brand overseas marketing copy
energy storage product application scenarios

One retrieval pass now covers industry background, user pain points, content structure, tone, and use cases. Content generation doesn't need one piece of evidence; it needs a material kit.

I use a mid-sized Qwen model here. It's only rewriting, not reasoning, so cheap, fast, and good enough.

Context is scattered across sources

The context that supports one generation task rarely lives in one place. We run three retrieval paths in parallel:

• Private knowledge base: internal knowledge, product docs, methodology. This is the core path, indexed with Qwen Embedding.
• User asset library: the videos, images, documents, and text uploaded for this task. More on this below.
• Web search: real-time and public info, like industry data and competitor moves, through Bing or Serper.

They run in parallel and merge later. The private base keeps content on-business, web search fills the freshness gap, and the user's assets provide the raw material this specific job must use. Drop one, and the result gets thin.

Multimodal assets need structure first

This is the biggest difference from plain text RAG.

Uploaded videos and images can't be searched as text. Before they can be recalled, they go through multimodal preprocessing:

• Videos get summaries, scene tags, and keyframe descriptions.
• Images get tags, subject descriptions, and style descriptions.
• Documents get titles, paragraphs, summaries, and keywords extracted.
• Plain text gets chunked.

Then everything is embedded into the vector store and can be retrieved online.

This step uses a Gemini Flash tier vision-language model: high throughput, low cost, good for tagging and summarization. It's in the same family as the Gemini Pro tier used later for generation, so the API surface is consistent and integration is simpler.

So retrieval here isn't topK. It's query rewriting, multi-source recall, and turning multimodal assets into searchable text first.

After retrieval: rerank and context assembly

Three-way recall buys breadth and brings noise. Sources differ in quality and trust, and the same meaning can be recalled twice. If you dump all of it into the model:

• internal conclusions and generic web articles blend together, and the model can't tell which matters
• duplicates waste tokens
• irrelevant material dilutes attention

So between recall and assembly sits a Rerank step. Rerank doesn't generate anything. It answers: which of these candidates deserve to be in the final context?

Vector search is coarse recall, tuned for speed and coverage. Rerank is fine ranking: it compares the query against each candidate more carefully and pushes the truly relevant snippets to the front.

I use a Qwen-series Reranker. A dedicated reranker is cheaper and steadier than asking a generation model to score, and it improvises less.

After rerank comes the easily underrated part: context assembly, done in code. Its job is to shape retrieved snippets into prompt context the model can actually use. I watch four things:

• deduplicate, so repeated info doesn't burn tokens
• partition internal knowledge, user assets, and web info so the model knows what's what
• label sources, because internal docs and random web pages are not equally trustworthy
• cap length, so the prompt fits the window while keeping the strongest evidence

The model never sees the raw database. It sees the prompt that code assembled. If that assembly is sloppy, even a strong model drifts.

Generation and delivery: the model only drafts

After context is assembled, generation starts. Two phases.

Agent planning turns "make a short video" into an executable structure:

• topic: what angle does this content take?
• script structure: opening, body, ending
• content split: how many sections or shots
• storyboard: what each shot says and which asset it uses
• output shape: short video, podcast, graphic post, or article

Content generation turns the plan into text:

• body and marketing copy
• short video script
• storyboard descriptions
• podcast narration
• graphic post content

These two use Gemini Pro tier. Complex planning, long context, and multimodal generation all live here.

But the model's output isn't the product. It's a high-quality draft. Scripts, storyboards, and copy still pass through engineering post-processing:

• short video orchestration: storyboards, assets, and subtitles into an editing plan or even a rendered video
• graphic layout: templates and formatting
• podcast editing: narration into usable segments
• article formatting: normalize structure, remove redundancy
• toolchain rendering: final files the user can actually use

This is the most engineering-heavy layer. The model produces the core content; business code turns it into something stable, controllable, and deliverable. Without it, a great model output is still just markdown.

Model division of labor

A flagship model could technically cover more stages. But production has to balance quality, cost, speed, and stability, and different stages suit different models. Current split:

Model family	Main responsibility	Stage
Qwen series (mid-sized)	Multi-Query rewriting	Before retrieval
Qwen Embedding	Knowledge base and asset library vectorization and retrieval	Vector retrieval
Qwen Reranker	Reranking retrieved results	Rerank
Gemini Flash tier	Multimodal asset preprocessing (video / image tagging and summarization)	Asset understanding
Gemini Pro tier	Agent planning and final content generation	Generation
Bing / Serper	External information from web search	External retrieval

The trade-offs are simple: Embedding needs a real vector model — generation models aren't built for retrieval, so they're worse and more expensive. Rerank with a generation model is slow and pricey; a reranker is built for it. Multimodal tagging should use a vision-language model with throughput, not a flagship. And the expensive model only shows up for planning and final generation, where it actually earns its cost.

In one line:

Gemini handles multimodal understanding and generation, Qwen handles retrieval enhancement, and the code layer handles orchestration and final delivery.

That split is cheaper, more stable, and more controllable than forcing one model to do everything.

The hard parts

On paper the pipeline looks like a few modules connected together. In practice the hard parts are all in the engineering details.

Structuring multimodal assets. Videos and images can't be searched as text. Tags, summaries, keyframe descriptions, and scene info all have to be generated first. Preprocessing quality decides whether retrieval finds the right material.

Merging multi-source recall. Internal knowledge, user assets, and web search have very different quality and trust. Merge, dedupe, label, and rank — each choice shapes the final context.

Controlling context. You can't send everything. Within the token limit you have to keep the most important info, partition it, and label sources so the model can tell internal knowledge from web noise.

Agent workflow stability. Short video, podcast, graphic post, and article have totally different flows. Each planning-to-post-processing chain has to be tuned separately, so complexity grows with every deliverable type you add.

Engineering the output. The model drafts; the product still needs layout, editing, formatting, and rendering. That layer usually takes more work than wiring up the model call.

None of this gets solved by swapping in a stronger model. It's ground out in the pipeline, data structures, prompt organization, and post-processing rules.

Conclusion

After building this, I've gotten more engineering-minded about AI content generation.

The model is the core, but only the middle: understand context, plan, generate. Before it, materials and knowledge and real-time info have to become usable context. After it, the output has to become a deliverable video, podcast, graphic post, or article. Whether the system holds up usually depends on everything outside the model.

I think of it as a production line now: retrieval preps the materials, context assembly lays them out in front of the model, the Agent plans and generates, and engineering post-processing turns the draft into a product. Models will keep improving, but the longer-term value is getting this line straight and making every handoff solid.