NEWDr FRIDAY, a wellness app we built and run on Amazon Web Services (AWS), is now on Google Play. View on Google Play →
Cloud

Three lessons from running a health app on AWS serverless: alarms, payments and shared accounts

By Ajit NK•11/20/2026

We recently took a wellness app from agreed scope to Google Play in 37 days, with the backend running on Amazon Web Services (AWS) in the Mumbai Region. The full case study is here. This post is about something else: three mistakes and near-misses from the build that taught us more than the parts that went smoothly. Each one is small, each one is easy to repeat, and each fix is a few lines.

The stack is plain: Amazon API Gateway, AWS Lambda, Amazon DynamoDB, Amazon CloudWatch and Amazon SNS, defined with the AWS Cloud Development Kit (AWS CDK). One engineer built it.

Lesson 1: A new alarm is judged against the past the moment it is created

We wanted a heartbeat alarm: a scheduled Lambda function publishes a metric every five minutes, and if the metric goes quiet, the team gets an e-mail. We set the alarm to alarm after four quiet five-minute periods, with missing data treated as breaching. Our reasoning was that four periods need 20 minutes, and the first heartbeat would arrive within five.

The reasoning was wrong. CloudWatch looks back over the previous four periods as soon as the alarm exists, finds no data, counts all four as breaching, and goes to ALARM within seconds. An alert e-mail went out at deployment, for a system that was working.

The fix is to treat missing data as missing, and to send notifications for the "insufficient data" and "OK" states too:

const alarm = new cloudwatch.Alarm(this, 'CollectorSilent', {
  metric: heartbeatMetric,                       // Sum of the heartbeat, 5-minute periods
  threshold: 1,
  comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
  evaluationPeriods: 4,
  datapointsToAlarm: 4,
  treatMissingData: cloudwatch.TreatMissingData.MISSING,
});
alarm.addAlarmAction(new actions.SnsAction(topic));
alarm.addInsufficientDataAction(new actions.SnsAction(topic));
alarm.addOkAction(new actions.SnsAction(topic));

A new alarm now starts in "insufficient data", which is not a state change, so it stays quiet until the first heartbeat. If the heartbeat later stops, the alarm moves back to insufficient data and sends the e-mail. We added a stack test that fails if this setting changes, because the mistake is easy to reintroduce.

Lesson 2: In a shared AWS account, every custom metric needs the environment in its name

Our production, user-acceptance, test and staging environments live in one AWS account. CloudWatch has no idea which environment a metric came from. We wrote a nightly credit-expiry job that published two custom metrics with the same names in every environment. When we proved the job on the test environment by causing eight deliberate errors, the new production alarm fired, and an alert e-mail went out. The alarm was reading the same metric name that the test run had just written.

The same flaw could also have gone the other way: a healthy test heartbeat could hide a real silence in production.

The fix is to put the environment in the metric name, passed from the stack, never typed in the function:

// simplified: production gets no suffix; every other environment gets its own
const metricSuffix = envName === 'prod' ? '' : `-${envName}`;

const expireRun = new cloudwatch.Metric({
  namespace: 'DrFriday/Credits',
  metricName: `ExpireRun${metricSuffix}`,
  statistic: 'Sum',
  period: cdk.Duration.hours(24),
});

Then we made the check automatic. A script reads the synthesized CloudFormation for every environment and fails if a custom metric name does not carry the environment. A convention that depends on remembering will be forgotten.

Lesson 3: "Exactly once" for a payment needs a database transaction, and a real database to prove it

Customers buy scan credits on a web page. The payment provider confirms by webhook, but webhooks can arrive twice, late or out of order. The page also checks the order status, and an hourly job compares the provider's records with ours. Three paths can try to grant the same credits, and only one may succeed.

The rule is that whoever arrives first wins, and everyone else finds the work already done. We implemented it as one DynamoDB transaction:

try:
    dynamodb.transact_write_items(TransactItems=[
        {"Put": {                               # 1. claim the payment id, once
            "TableName": payments_table,
            "Item": {"pk": {"S": f"PAYMENT#{payment_id}"}},
            "ConditionExpression": "attribute_not_exists(pk)",
        }},
        {"Update": {                            # 2. add the credits
            "TableName": users_table,
            "Key": {"phone": {"S": phone}},
            "UpdateExpression": "ADD credits :n",
            "ExpressionAttributeValues": {":n": {"N": str(credits)}},
        }},
        {"Put": {                               # 3. write the history line
            "TableName": ledger_table,
            "Item": ledger_item,
        }},
    ])
except dynamodb.exceptions.TransactionCanceledException as err:
    reasons = [r.get("Code") for r in err.response.get("CancellationReasons", [])]
    if "ConditionalCheckFailed" in reasons:
        return "already processed"            # someone else got there first
    raise                                       # anything else is a real problem

Two things only showed up when we tested against a real DynamoDB table, not the in-memory fake we used for unit tests:

  • Our history table was keyed by patient and second. Two grants in the same second silently overwrote one history line while the balance stayed correct. A conditional write that moves to the next free second fixed it.
  • Real DynamoDB cancels a transaction when two writers touch the same item at once, and our code first treated that as a failure. The in-memory fake never produced that case. We now retry that path a few times, and we proved it with concurrent calls against the real table.

The takeaway is about testing as much as design: for anything involving money, a test that passes against a simulator is not enough.

What we would do again

  • Treat every alarm change as something to test, because alarms fail quietly.
  • Put environment names into anything that is shared, and check them in code.
  • Test money paths on the real service, with concurrent calls.

None of this is clever. It is the kind of detail that is cheap to get right on day one and expensive after an incident.

If you are planning a health or wellness app on AWS, get in touch and we can compare notes on demo data. Dr FRIDAY is a wellness tool and does not give medical advice.

💡

Need Expert Help?

Our AWS certified professionals can help you implement these strategies in your organization.