Abstract:
A seemingly harmless schema change in Amplify Gen2 nearly broke our app in a higher environment. Here is how a frustrating debug session led to a deeper understanding of Amplify’s auto-generated queries, the pitfalls of table scans, and the importance of testing real-world edge cases across environments.
The Setup
We have done dozens of Amplify Gen2 deployments in multiple environments, and this one seemed no different, until my admin user suddenly lost access to expected features. Everything worked in my local sandbox. But in a higher environment? I was missing key permissions even though I was authenticated.
Amplify Gen2 generates AppSync schemas and queries based on your data models. That’s usually helpful, but it is easy to miss what Amplify is creating behind the scenes.
Environment Matters
One of Amplify’s quirks is that its behavior in different environments is not always consistent:
- Sandbox(local dev): Fast deployments, destructive schema updates, and data wipes are common
- Higher environments: Safer, slower to deploy, but more opaque and harder to debug
This divergence tripped us up. A debug panel I added showed the data was definitely there, but the query returned nothing. My sandbox worked, but it had only one user. The higher environment had two or more. I hit the classic “It works on my machine” problem.
The Problem Was the Query
After trying a few variations and going in circles for a bit, (“The data is missing!” declared Claude) I discovered something strange: switching from limit: 1 to limit: 2 made the query work. Testing the query in AppSync was key to cutting out the client and narrowing down on the problem. Being able to see the query and test directly was essential. That’s when it clicked.
DynamoDB’s scan with a filter will return the first item it finds but it needs to scan everything in the table, limit: 1 can produce a false negative if there is more than 1 item in the table. It can depend on how the data is ordered in the table. Classic trap.
We Tried Adding a GSI (and Wiped Everything)
To fix this, we added a secondary index on cognitoId because that is the field we really wanted to filter on. That should have helped but instead, some of our sandbox tables were wiped. We had tried to add secondary indices to multiple tables at the same time to try and save deployment time.
Turns out this is a known behavior, Amplify’s default sandbox deploy mode can drop and recreate tables when schema changes like GSIs (Global Secondary Index) are introduced. It is faster to build and deploy but frustrating when you have hundreds of thousands of rows of data (and forgot to back up first). Not catastrophic, but certainly annoying. There’s even a GitHub issue confirming this behavior.
Thankfully, the higher environment was not affected when the changes were made there. Still, the index did not seem to help until I realized:
Amplify creates new query methods for GSI, you have to use them after you deploy the indices in the schema.
I found the new query in our schema and tried it directly in AppSync. Success!
The Fix: Use the Right Query
Here’s what was happening:
# ❌ This does table scans
listUsers(filter: { cognitoId: { eq: "..." } }, limit: 1)# ✅ This uses the new GSI directly
listUserByCognitoId(cognitoId: "...", limit: 1)The second query is only available because we added a GSI for cognitoId. It was not obvious unless you inspected the generated schema in AppSync, which is how Amplify Gen2 exposes these auto-created queries.
Lessons Learned
- Sandbox ≠ Higher environment behavior
- Adding secondary indexes (and other schema changes) in a sandbox may drop your table which clears your data (GitHub issue)
- Following schema changes, Amplify Gen2 creates hidden queries behind-the-scenes
- Table scans are inefficient and unreliable for filtering, search by GSI is much faster
- Always test with multiple real-world records in every table
- AI tools (including Claude in Cursor) struggle with Amplify-specific debugging
- AppSync query console is a critical troubleshooting tool
- Amplify docs don’t highlight auto-generated queries well enough
Final Thoughts
Even with AI-assisted tools, real debugging still takes context, patience, and experience. Once we deployed the fix, it worked—and the tables in our higher environment stayed intact.
Have you run into something like this? Let’s trade war stories, reach out here.
Matt Pitts, Sr Architect
No responses yet