The Security Check Is Not the Security Boundary
A security command that returns a clean report is not the same thing as a secure application.
Shopify has added shopify app security check to Shopify CLI and AI Toolkit. The command scans an app locally for common security patterns across its code, extension configuration, and dependencies. When used through a coding agent, the workflow also asks the agent to investigate questions that static inspection cannot answer cleanly.
That split is the interesting part. The tool is not pretending that one scan can prove security. It is building a small control loop around a recurring developer task: inspect, explain, fix, review, and inspect again.
The command is only the first pass
Shopify’s developer announcement says the check runs locally and reports the affected file, line, severity, impact, remediation example, and documentation for each finding.
The CLI reference describes the default pass as a deterministic scan. It can catch questions with an exact answer, such as whether a credential is committed or an API version is obsolete. The command can also print instructions for a coding agent to perform the next stage of review.
The recommended invocation through AI Toolkit is:
/shopify Check my app for security vulnerabilities
The agent runs the command, investigates checks that require understanding the application’s behavior, records its findings, and presents the combined result. Shopify’s security-check guide gives the example of determining whether a database query is scoped to one shop. That is not merely a string pattern. It is a question about data flow, identity, and authorization.
Deterministic checks and agent review have different jobs
The two stages should not be collapsed into one vague claim that AI scans the code.
| Stage | What it is good at | What it cannot establish by itself |
|---|---|---|
| Deterministic scan | Fixed patterns, configuration checks, committed credentials, known unsafe constructs | Whether application behavior is safe in every execution path |
| Agent review | Contextual questions such as tenant isolation and missing authorization logic | A proof that the agent understood every path or made a correct change |
| Human review and tests | Confirming intent, regression behavior, and deployment impact | Eliminating every unknown unknown |
The table is not a ranking. It is a boundary map.
The deterministic scan is valuable because it is repeatable. The agent review is valuable because application security is full of questions that depend on context. Neither stage should inherit the other stage’s confidence.
The failure mode is false closure
The dangerous version of an AI security tool is not necessarily one that finds nothing. It is one that produces a polished report that convinces a team the work is finished.
Shopify’s own documentation is explicit: the command is limited to source code and should be part of a layered security approach. The announcement also tells developers to start with critical and high-severity findings, review the changes, and run the check again.
That wording matters. The product is not a security boundary around the app. It is a verifier inside the development loop.
A clean result can still miss runtime configuration, infrastructure mistakes, authorization behavior that depends on external services, supply-chain compromise, deployment drift, and bugs outside the checks it knows about. The agent stage adds useful reasoning, but it also adds another component whose conclusions need review.
The output is designed to become an artifact
The CLI stores results under .shopify/app-security/. It can be configured with a --blocking threshold so findings at or above high, medium, or low severity produce a non-zero exit code.
That turns the check from a suggestion into a possible release gate:
shopify app security check --blocking high
The gate is only useful if the repository and pipeline treat it as evidence rather than decoration. A practical workflow should preserve the report, associate it with the commit that produced it, and make the exit status visible in CI. It should also define what happens when a finding is waived, when the scan cannot inspect a generated path, or when an agent review is incomplete.
The command supports --include-dir for shared code outside the app directory, --exclude for intentional exclusions, and --list-files to show exactly what would be scanned. Those flags are not minor conveniences. They determine the scope of the verifier.
If the backend lives in a sibling directory and the scan never includes it, a passing report says only that the selected files passed. If Git-ignored build output contains behavior that reaches production, the default scan may not see it. Scope must be reviewed alongside findings.
What builders should change
Treat an AI-assisted security check as a control loop with explicit handoffs.
- Pin the scanned scope. Review the file list and use
--include-dirfor shared backends or libraries. - Make severity blocking intentional. Choose a threshold that matches the release risk instead of accepting the default, which never fails on findings.
- Separate scan evidence from agent conclusions. Store which findings came from deterministic checks and which came from agent investigation.
- Review every proposed fix. The fastest way to create a new vulnerability is to accept a broad code change because its explanation sounds security-aware.
- Run regression tests after remediation. Authorization and tenant isolation fixes can change legitimate behavior.
- Keep layered controls. Continue using dependency checks, secret scanning, tests, runtime monitoring, platform protections, and human review.
The command is also open source. Shopify points developers to the security engine in the Shopify CLI repository, which gives operators something concrete to inspect and improve rather than a black-box promise.
The verifier belongs inside the loop
Shopify’s new check is a good example of where agent security tooling is heading. The useful unit is not an LLM that reads a repository and declares it safe. The useful unit is a loop that combines repeatable checks, contextual investigation, reviewable evidence, and a release decision.
The uncomfortable truth is that the command cannot be the security boundary because it does not control the whole system. It can identify a missing authorization check. It cannot, by itself, guarantee that every request reaches the check, that the fix is deployed, or that the next feature does not bypass it.
Use the tool as a verifier. Keep the boundary in the architecture, the tests, the permissions, and the review process.