When Your LLM Bill Outgrows Your MVP: A Bedrock Cost Architecture Case Study

Generative AI prototypes have a well-known superpower: they can go from zero to impressive demo in a weekend. They also have a less-celebrated habit of quietly compounding token costs until the monthly invoice lands and someone in finance asks an uncomfortable question.

This post walks through a real engagement with a mid-size engineering team whose Amazon Bedrock spend had begun scaling faster than their user base. We'll cover how they got visibility into what was actually driving costs, the architectural patterns that brought spending back under control, and the AWS-documented best practices that informed every decision along the way.

The Situation: Fast Growth, Faster Burn

The team had shipped an AI-powered product on top of Amazon Bedrock. MVP traction was real — usage was climbing, the product was working, and the engineering team was shipping features. The problem surfaced gradually: Bedrock credits were depleting at a rate that made the unit economics look increasingly uncomfortable as they projected toward scale.

The instinct in this situation is usually to reach for a cost-cutting lever immediately — swap the model, reduce output length, something. The team resisted that impulse, which turned out to be the right call. Without knowing which workloads were expensive and why, any optimization is essentially guesswork that risks degrading the features users actually value.

What they needed wasn't a quick fix. They needed a structured diagnostic followed by targeted architecture changes.

Why LLM Cost Problems Are Non-Obvious

Token costs feel deceptively simple on paper: you pay per input token and per output token, rates vary by model, done. In practice, the cost surface of a real application is far more complex, and several factors conspire to make the root cause hard to identify without proper instrumentation.

The frontier model default problem. When you're moving fast, you reach for the most capable model available. Claude Sonnet handles your classification task? Great, ship it. The problem is that frontier models are priced for frontier work. AWS documentation on effective cost optimization for Amazon Bedrock is explicit about this pattern: using a frontier model for tasks that belong in a smaller model or a rule-based system inflates costs disproportionately. Classification, extraction, and structured output tasks frequently fall into this category.

Repeated context you're paying for over and over. Many applications send the same long system prompt — sometimes thousands of tokens — with every single API call. At scale, you're essentially re-paying for that context on every invocation. This is one of the highest-leverage cost problems in production LLM systems, and it's invisible until you start measuring at the token level.

No per-feature cost visibility. By default, Bedrock costs appear as a single line item. A team with five AI-powered features has no way of knowing whether 80% of their spend comes from one feature or is spread evenly. This makes prioritization impossible.

Synchronous calls doing asynchronous work. Real-time inference is more expensive than batch inference. Teams often default to synchronous API calls even for workloads — nightly summaries, bulk document processing, background classification — where the user doesn't need a response in under two seconds.

Step One: Instrument Before You Optimize

The single most important principle in this engagement was establishing measurement infrastructure before writing a single optimization recommendation. Every suggestion made without real token-level data is an estimate, and estimates in this domain are frequently wrong by an order of magnitude.

The instrumentation baseline required two things:

Model Invocation Logging is a free Bedrock feature that captures every model call — model ID, input tokens, output tokens, latency, and invocation metadata — and ships it to S3, CloudWatch Logs, or both. Enabling it takes under an hour:

Navigate to Bedrock Console → Settings → Model Invocation Logging
Enable logging and select your target S3 bucket and/or CloudWatch Logs group
Validate by running a test invocation and confirming the log entry appears in CloudWatch within approximately 60 seconds

Once logs are flowing, you can query them with Athena (for S3) or CloudWatch Logs Insights (for CloudWatch) to get per-model token counts broken out by any dimension you've tagged.

Application Inference Profiles are the second half of the visibility layer. Where Model Invocation Logging tells you what was called, Application Inference Profiles let you attribute costs to specific features or business units via cost allocation tags. You create one profile per major feature using the CreateInferenceProfile API or the Bedrock console, attaching tags like cost_center and feature_name.

import boto3

bedrock = boto3.client('bedrock', region_name='us-east-1')

response = bedrock.create_inference_profile(
    inferenceProfileName='document-summarization-prod',
    description='Inference profile for the document summarization feature',
    modelSource={
        'copyFrom': 'arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0'
    },
    tags=[
        {'key': 'cost_center', 'value': 'product-team'},
        {'key': 'feature', 'value': 'document-summarization'},
        {'key': 'environment', 'value': 'production'}
    ]
)

profile_arn = response['inferenceProfile']['inferenceProfileArn']
print(f"Created profile: {profile_arn}")

With both pieces in place, you can write a CloudWatch Logs Insights query like this to see token consumption broken down by feature:

fields @timestamp, modelId, inputTokenCount, outputTokenCount
| filter ispresent(inputTokenCount)
| stats sum(inputTokenCount) as totalInputTokens,
        sum(outputTokenCount) as totalOutputTokens,
        count(*) as invocations
  by modelId
| sort totalInputTokens desc

This instrumentation baseline took the team approximately 30–60 minutes to establish. Everything that followed was grounded in actual data rather than architectural assumptions.

Step Two: Decompose the Workload, Model the Costs

With telemetry flowing, the next phase was a structured workload intake: cataloguing every LLM call site in the application, characterizing its behavior, and attaching a cost model to each one.

For each call site, the relevant dimensions are:

| Dimension | Why It Matters | |---|---| | Call volume (daily/monthly) | Determines absolute cost impact of any change | | Average input token count | Identifies caching candidates | | Output type | Generative, classification, extraction, or agentic — each has different optimization paths | | Latency requirement | Real-time vs. async determines whether batch inference is viable | | Model currently in use | Identifies frontier-model-for-simple-task mismatches |

Once this table exists, you can calculate current monthly cost per call site as:

monthly_cost = avg_input_tokens × input_rate 
             + avg_output_tokens × output_rate
             × monthly_call_volume

And project post-optimization cost under each applicable lever. The output of this phase is a prioritized savings table that makes the highest-ROI change immediately obvious. In practice, the top opportunity is almost always either prompt caching on a high-volume long-context feature, or moving a classification workload off a frontier model.

Step Three: Apply the Right Optimization Layer

With a cost model in hand, the team had three primary architectural levers to work with, each documented in AWS's Well-Architected GenAI Lens.

Prompt Caching for Repeated Context

For any call site sending a long, repeated system prompt — RAG context, document contents, detailed instructions — prompt caching is typically the single highest-impact optimization available. AWS documentation for GENCOST03-BP03 quantifies the impact: up to 90% reduction in input token costs and up to 85% reduction in latency for eligible workloads. Cache retention is five minutes, which covers the vast majority of interactive session patterns.

Implementation requires marking cacheable portions of your prompt with cachePoint blocks:

import boto3
import json

bedrock_runtime = boto3.client('bedrock-runtime', region_name='us-east-1')

# System prompt with cache checkpoint after the stable, expensive context
system_prompt_with_cache = [
    {
        "text": """You are a document analysis assistant. Your role is to answer 
        questions about the provided document accurately and concisely.
        
        [DOCUMENT CONTENT - 4000 tokens of stable context here]
        
        Always cite specific sections when answering questions."""
    },
    {
        "cachePoint": {
            "type": "default"
        }
    }
]

response = bedrock_runtime.converse(
    modelId='anthropic.claude-3-5-sonnet-20241022-v2:0',
    system=system_prompt_with_cache,
    messages=[
        {
            "role": "user",
            "content": [{"text": "What are the key findings in section 3?"}]
        }
    ]
)

The minimum token threshold for cache eligibility varies by model family, so verifying that your system prompt meets the threshold before deployment is an important implementation detail.

Intelligent Prompt Routing for Mixed Workloads

Not every query in a given feature requires the same model capability. A user asking a simple factual question from a knowledge base doesn't need the same model as a user asking for a nuanced multi-step analysis. Amazon Bedrock Intelligent Prompt Routing addresses this directly: it provides a single serverless endpoint that dynamically routes requests between frontier and cheaper models within the same family — for example, between Claude Sonnet and Claude Haiku — based on the assessed complexity of each request.

The operational benefit is significant: you get a single API endpoint, no routing logic to maintain in application code, and automatic fallback to the frontier model when the cheaper model isn't confident enough to handle the request. This pattern is particularly effective for customer-facing features with high call volume and variable query complexity.

Batch Inference for Asynchronous Workloads

Any call site that doesn't require a real-time response is a candidate for Bedrock's batch inference, which processes requests asynchronously at lower cost than on-demand invocation. Common candidates include:

Nightly document summarization pipelines
Bulk content classification or tagging jobs
Offline data enrichment workflows
Scheduled report generation

A typical batch trigger pattern uses EventBridge Scheduler to kick off a batch inference job at a defined interval:

import boto3
import json

bedrock = boto3.client('bedrock', region_name='us-east-1')

# Submit a batch inference job
response = bedrock.create_model_invocation_job(
    jobName='nightly-document-classification-2024-01-15',
    modelId='anthropic.claude-3-haiku-20240307-v1:0',
    inputDataConfig={
        's3InputDataConfig': {
            'inputDataType': 'S3Prefix',
            's3Uri': 's3://your-bucket/batch-inputs/2024-01-15/'
        }
    },
    outputDataConfig={
        's3OutputDataConfig': {
            's3Uri': 's3://your-bucket/batch-outputs/2024-01-15/'
        }
    },
    roleArn='arn:aws:iam::123456789012:role/BedrockBatchInferenceRole'
)

print(f"Batch job ARN: {response['jobArn']}")

Rule-Based Replacement for Deterministic Tasks

The final lever — and often the most impactful per-dollar — is recognizing that some tasks don't need an LLM at all. If a call site is doing something deterministic (input validation, format checking, keyword classification with a fixed taxonomy, regex-matchable extraction), replacing it with a rule-based system eliminates that cost entirely while typically improving latency and reliability. AWS guidance on cost optimization explicitly recommends auditing for these cases before applying more sophisticated optimization techniques.

Putting It Together: The Architecture Decision Record

Each major change in this engagement was documented in an Architecture Decision Record (ADR) that captured the current state, the target state, the rationale, and the trade-offs. Mermaid diagrams illustrated the before and after for each affected call site.

This documentation discipline matters beyond the immediate engagement. LLM cost architecture decisions have downstream implications for model upgrade paths, latency budgets, and feature development. An ADR that explains why a workload was moved to batch inference prevents a future engineer from unknowingly reverting it to synchronous calls.

The final deliverable — a 10–15 page report covering workload decomposition, per-lever cost modeling, a prioritized action plan with console and API instructions, instrumentation blueprints, and architecture diagrams — required roughly 9–16 hours of diagnostic and design work across the five phases described above. That scope reflects what it actually takes to produce recommendations grounded in data rather than generic advice.

Key Takeaways

If your Bedrock costs are scaling faster than your user metrics, the path forward follows a consistent pattern:

Instrument first. Enable Model Invocation Logging and create Application Inference Profiles with cost allocation tags before drawing any conclusions. Thirty minutes of setup work eliminates weeks of guesswork.
Decompose before optimizing. Map every LLM call site to its volume, token profile, output type, and latency requirement. The highest-cost feature is almost never the one you'd guess.
Match the model to the task. Frontier models are priced for frontier work. Classification, extraction, and structured output tasks frequently belong on smaller models or in rule-based systems.
Cache repeated context aggressively. If you're sending the same multi-thousand-token system prompt on every call, prompt caching is likely your single highest-ROI change — up to 90% reduction in input token costs on eligible workloads.
Let Intelligent Prompt Routing handle complexity variance. For high-volume features with variable query complexity, dynamic routing between model tiers delivers cost reduction without requiring application-level routing logic.
Move async workloads to batch. Any pipeline that doesn't need a real-time response has no reason to pay real-time inference prices.

The mid-size engineering team in this case study didn't have a spending problem — they had a visibility problem. Once the instrumentation was in place and the workload was properly decomposed, the path to sustainable unit economics became straightforward. The same pattern applies to almost every team hitting this inflection point: the answer is almost never "use AI less." It's "use the right AI for each job, and measure everything."

For teams working through similar challenges, the AWS Well-Architected GenAI Lens cost optimization pillar (GENCOST01–03) provides the authoritative framework. Model Invocation Logging documentation and Application Inference Profile setup guides are available in the Amazon Bedrock documentation.

Ai Genai Case Study: LLM costs scaling beyond MVP, burning credits fast