Reduce RAG costs on Amazon Bedrock through intelligent compression

Introduction to RAG cost challenges

In the modern world of generative AI, Retrieval-Augmented Generation (RAG) architectures have become the foundation of enterprise applications that require precise and contextual answers. However, as these systems evolve and are widely adopted, the costs associated with token processing increase exponentially. Each query sent to a large language model (LLM) consumes a significant number of tokens, and in the context of RAG, retrieved documents can add thousands of additional tokens to each request. This economic reality is quickly becoming a major issue for companies looking to scale AI solutions without blowing their cloud infrastructure budget.

Amazon Web Services has identified this critical challenge and developed an innovative solution available on the Amazon Bedrock platform : context-aware compression, also known as Query-Aware Compression . This technology revolutionizes the way RAG systems manage retrieved documents, removing irrelevant information and keeping only the content essential to generating the answer. The result is a dramatic reduction in the number of tokens processed, which directly translates into substantial cost savings, without compromising the quality or accuracy of the model-generated answers.

What is Query-Aware Compression and how does it work?

The fundamental principle of contextual compression

At the heart of Query-Aware Compression is an elegant principle: not all information in a retrieved document is relevant to a given query. Traditionally, RAG systems send the entire content of retrieved documents to the language model, regardless of how much of that content actually contributes to the answer. This approach is not only expensive, but also potentially detrimental to the quality of the answers, as LLM models can be distracted by irrelevant information or suffer from the phenomenon known as “lost in the middle” , where information in the middle of a long context is ignored or processed incorrectly.

Query-Aware Compression solves this problem through a two-stage process. In the first stage, the system analyzes the user's query and the retrieved documents to identify passages with the highest semantic relevance. In the second stage, it applies compression algorithms that eliminate redundant or irrelevant content, keeping only the fragments that have a high probability of contributing to the formulation of a correct and complete answer. This approach dramatically reduces the number of tokens sent to the LLM model, while maintaining the semantic accuracy of the provided context.

Technical architecture of the solution on Amazon Bedrock

The technical implementation of Query-Aware Compression on Amazon Bedrock involves using a dedicated compression model that acts as an intermediary between the retrieval step and the generation step in the RAG pipeline. This compression model, often a smaller and more cost-effective language model, evaluates each text segment in the retrieved documents and assigns it a relevance score relative to the current query. Segments with scores below a certain threshold are discarded, and the rest are concatenated and sent to the main generation model.

In the Amazon Bedrock ecosystem, this architecture benefits from native integration with services such as Amazon Bedrock Knowledge Bases , Amazon OpenSearch Serverless , and Amazon S3 . The data flow begins with storing documents in S3, indexing them in OpenSearch Serverless using embeddings generated by models available in Bedrock, retrieving relevant documents at query time, compressing the retrieved context, and finally sending the compressed context to the LLM model to generate the final response. Each component in this pipeline is managed and automatically scaled by AWS infrastructure, providing an end-to-end experience with no operational friction.

The quantifiable economic benefits of smart compression

Reducing costs with tokensContext Window

The most direct and quantifiable benefit of Query-Aware Compression is the reduction in the number of input tokens processed per query. Large language models, such as Anthropic's Claude or Amazon's Titan models, store input and output tokens separately. In typical RAG applications, input tokens come largely from retrieved documents, which can represent 70-90% of the total tokens processed per query. By compressing these documents, organizations can reduce input costs by 30-80% , depending on the type of documents and the specificity of the queries.

To understand the financial impact in concrete terms, consider a typical enterprise scenario: a company processes 1 million RAG queries per month, with each query retrieving an average of 5 documents at an average of 500 tokens per document. Without compression, this translates to 2,5 billion monthly input tokens from retrieved documents alone. With a 60% compression rate, this figure drops to 1 billion tokens, generating substantial savings that can reach tens of thousands of dollars per month, depending on the model used and its price per token.

Improving latency performance and response quality

Beyond the direct financial savings, intelligent compression brings significant benefits in terms of latency and response quality . LLM models process tokens sequentially, and a shorter context means lower processing times. In enterprise applications where latency is critical, such as customer support chatbots or real-time assistance systems, this latency reduction can significantly improve the end-user experience and allow scaling to a larger number of concurrent users without proportionally increasing the infrastructure.

In terms of response quality, research has shown that LLM models generate higher quality responses when the context provided is concise and relevant. Removing informational noise from the RAG context reduces the likelihood that the model will generate hallucinations or be distracted by contradictory or irrelevant information. Therefore, Query-Aware Compression is not only a cost optimization, but also an improvement in the intrinsic quality of the RAG system , providing a dual value proposition for engineering teams and business stakeholders.

Practical implementation on Amazon Bedrock

Configuring the RAG pipeline with compression

Implementing Query-Aware Compression within Amazon Bedrock Knowledge Bases is surprisingly accessible, even for teams that don’t have extensive experience with RAG optimizations. AWS has integrated this functionality directly into the existing Bedrock APIs, allowing compression to be enabled with a simple parameter configuration. The setup process involves defining a compression model that will be used to evaluate the relevance of text segments, setting the minimum relevance threshold for including segments in the compressed context, and configuring the chunking strategy for the source documents.

From a code perspective, integration is done through the AWS SDK in popular languages ​​such as Python, JavaScript, or Java. A typical example of a Python implementation involves using the boto3 client to call the Bedrock API with compression parameters specified in the request body. The key parameter is retrievalConfiguration , which accepts a compression configuration object that includes the compression type ( ENABLED or DISABLED ) and the compression model used. AWS recommends using a smaller, more efficient model for the compression step to maintain an optimal cost-benefit ratio across the entire pipeline.

Advanced compression optimization strategies

For teams looking to get the most out of compression, there are several advanced strategies worth exploring. The first strategy is adaptive compression based on query type : simple factual queries can benefit from more aggressive compression rates, while complex queries that require the synthesis of multiple sources of information may require richer context. Implementing a query classifier that dynamically adjusts the compression threshold can further optimize the cost-performance ratio of the RAG system.

The second advanced strategy is hierarchical compression , where documents are compressed initially at the paragraph level and then at the sentence level, applying a double relevance filter. This approach is particularly effective for long, information-dense documents, such as technical reports, legal contracts, or medical documentation. A third strategy involves using feedback from the generation model to continually refine the compression model, creating a continuous improvement loop that adjusts thresholds and compression strategies based on the quality of the generated responses.

Comparison with alternative approaches to RAG cost optimization

Query-Aware Compression vs. reducing the number of retrieved documents

A common alternative approach to reducing RAG costs is to simply reduce the number of documents retrieved for each query, thereby reducing the total number of input tokens. This strategy is simple to implement, but has a fundamental disadvantage: it reduces the system's recall , that is, the probability that relevant information will be included in the context provided to the model. In enterprise applications where accuracy is critical, reducing recall is not an acceptable option.

Query-Aware Compression offers a superior alternative: instead of reducing the number of documents retrieved, you retrieve a similar or even larger number of documents, but compress each document to extract only the relevant information. Thus, you maintain or improve the system's recall while reducing the number of tokens processed . This is a clear value proposition and superior to the naive approach of document reduction.

Query-Aware Compression vs. Document Summarization

Another popular RAG optimization technique is to pre-process documents by generating summaries and storing these summaries instead of the full documents. This approach reduces retrieval costs, but has its own limitations: summaries are generated offline, without knowledge of future queries, and may omit specific details that are relevant to certain types of queries. In addition, generating summaries for a large corpus of documents requires a substantial initial cost and must be repeated whenever the documents are updated.

Query-Aware Compression is an online and adaptive approach : compression is done at query time, with exact knowledge of the user's query, which allows for a much more precise selection of relevant information. The disadvantage is that it adds latency and marginal cost to the compression step, but these costs are generally smaller than the savings generated by reducing the tokens sent to the main generation model.

Enterprise and industry verticals use cases

Applications in the financial and legal fields

In the financial sector, RAG systems are used to analyze annual reports, assess the risks of complex contracts, and answer regulatory questions. Documents in these areas are notorious for their length and information density, making them ideal candidates for intelligent compression. An investment bank that analyzes hundreds of financial analysis reports daily can see cost reductions of 50-70% by implementing Query-Aware Compression, while maintaining the accuracy of answers needed for critical financial decisions.

In the legal field, law firms and corporate legal departments are increasingly using RAG systems for case law research, contract analysis, and case preparation. Legal documents, with their specific structure and dense technical language, benefit enormously from contextual compression. The ability to quickly extract relevant clauses from a hundreds-page contract, without processing the entire document with an expensive LLM model, represents a fundamental transformation in operational efficiency in the legal industry.

Applications in healthcare and scientific research

In the healthcare industry, RAG systems powered by Amazon Bedrock are used to help physicians quickly find relevant information from medical literature, clinical guidelines, and patient histories. Medical articles and clinical trials are extremely dense documents, with sections on methodology, results, and discussion that may or may not be relevant to a specific query. Query-Aware Compression enables medical RAG systems to quickly extract relevant clinical information, reducing processing costs and improving response speed in time-critical scenarios.

In scientific research, academic institutions and R&D companies manage massive corpora of scientific publications. The ability to efficiently query these corpora, at controllable costs, democratizes access to scientific knowledge synthesis and accelerates the pace of discovery. By reducing RAG costs with intelligent compression, institutions with smaller budgets can access scientific literature synthesis capabilities that were previously reserved only for organizations with substantial financial resources.

Considerations for production implementation

Monitoring and evaluating compression quality

Implementing Query-Aware Compression in production requires a robust monitoring strategy to ensure that cost savings do not come at the expense of response quality. AWS recommends implementing an evaluation framework that simultaneously measures cost per query, latency, compression ratio, and response quality . Quality metrics can include Retrieval Augmented Generation Assessment (RAGAS) scores, explicit user feedback, and automated assessments based on more powerful LLM models used as judges.

Native AWS tools, such as Amazon CloudWatch and AWS X-Ray , can be configured to trace each stage of the compressed RAG pipeline, providing granular visibility into system performance. Automated alerts can notify engineering teams when compression rates deviate significantly from expected values ​​or when quality scores fall below predefined thresholds, allowing for rapid intervention to adjust compression parameters.

Security and compliance considerations

In enterprise deployments, it is essential to ensure that the compression process does not introduce security vulnerabilities or violate compliance requirements. All data processed within the RAG pipeline on Amazon Bedrock benefits from encryption in transit and at rest , granular access controls provided by AWS IAM , and full auditing through AWS CloudTrail . Organizations in regulated industries, such as finance or healthcare, must ensure that the compression model used does not expose sensitive data and that all processing occurs within the geographic and organizational boundaries imposed by applicable regulations.

Query-Aware Compression on Amazon Bedrock represents an important step in maturing the RAG ecosystem for enterprise-scale use. By addressing one of the most pressing issues in the adoption of RAG systems at scale, AWS is giving organizations a concrete tool to control their generative AI costs without sacrificing the quality or capabilities of the systems they deploy. The combination of reduced token costs, improved latency, and increased response quality makes this technique one of the most valuable recent innovations in the generative AI infrastructure space.

Looking ahead, the evolution of contextual compression techniques will bring even more optimizations. Domain-specific compression models, adaptive compression algorithms that learn from user feedback, and the integration of compression with advanced re-ranking and semantic filtering techniques are active research and development directions that promise to bring even more substantial cost reductions. For organizations investing in building RAG infrastructure on Amazon Bedrock today, adopting Query-Aware Compression is not just a tactical cost optimization, but a strategic decision that positions these organizations to scale efficiently and sustainably in the era of generative AI.

Reduce token processing costs by up to 80% by implementing Query-Aware CompressionImprove response quality by removing informational noise from the RAGS contextScale enterprise RAGS applications without proportional increase in infrastructure costsBenefit from native integration with the AWS ecosystem for monitoring and securityDynamically adapt the compression rate depending on the type and complexity of queriesMaintain compliance with industry regulations with native AWS security controls.