Dreamer's Build Pipeline, Round Two: An Env Var That Never Left the Host, and a Shebang Line You Can't See Is Broken

The last SSR post ended with a working Function URL and three fixed bugs. It did not end with a working platform. Two more turned up almost immediately after, both in the exact part of the pipeline I'd just finished trusting: getting a project's own configuration into its own build.

Neither one threw an error where I expected it to. One didn't throw an error at all.

Table of contents

1. Problem 1: The Env Var That Never Left the Host
2. Problem 2: A File That Was Right There
3. Final Result
4. Closing Thoughts

Problem 1: The Env Var That Never Left the Host

This one started as a question, not a bug report: does the build actually get the project's env vars, or just the deployed function?

The two paths through Dreamer's build engine handle this completely differently, and I'd only ever tested one of them properly. A STATIC build runs npm install and npm run build as a plain child process inside the build task, and that process inherits the task's environment automatically, no extra wiring needed. A DYNAMIC (SSR) build hands the Dockerfile to Kaniko instead, since there's no Docker daemon on Fargate. Kaniko builds in its own isolated environment. It does not inherit anything from the process that invoked it. It never has.

deployment-engine.ts already puts a project's env vars into the build task's container environment, right alongside the platform's own internal ones like AWS_ACCESS_KEY_ID and REDIS_URL, flattened into the same array with no separator between the two. That's exactly what a STATIC build needs. For a DYNAMIC build it does nothing at all, because Kaniko never touches that array.

I went looking for the place that was supposed to compensate for that and it wasn't there. The generated Dockerfile template had no ARG lines. The Kaniko invocation had no --build-arg flags. A repo-wide grep for build-arg came back with a single result: the frontend's own Dockerfile, building Dreamer itself, not anything Dreamer builds for someone else.

The reason this hadn't shown up as an obvious break is that it wasn't a full break. Lambda's runtime environment variables were set correctly, since those go through a completely different path at deploy time, after the image already exists. Anything read from process.env inside a server component or an API route at request time worked fine. What silently failed was anything read during the build itself: NEXT_PUBLIC_ variables, which Next.js inlines into the client bundle at next build time, and anything a page reads from process.env during static generation. The build succeeded either way. It just quietly shipped undefined for half the config.

The fix needed two things Kaniko's isolation doesn't give you for free. First, a way for the build script to know which of the container's env vars were the project's own and which were platform internals, since a name-based diff against reserved prefixes felt fragile the moment I considered relying on it. So the same vars go in twice now: flattened, for STATIC builds that still need that, and a second time as one JSON-encoded manifest, so the DYNAMIC path has an explicit list instead of a guess.

Second, the values had to reach Kaniko without getting written as literal text into a Dockerfile that ends up sitting in the build context. ARG NAME plus ENV NAME=$NAME goes in for every var name, right before the install and build steps. The actual values travel separately, as --build-arg NAME=value on the Kaniko command itself.

ARG NEXT_PUBLIC_API_URL
ENV NEXT_PUBLIC_API_URL=$NEXT_PUBLIC_API_URL
ARG DATABASE_URL
ENV DATABASE_URL=$DATABASE_URL
RUN npm ci
RUN npm run build

The one line I went back and changed after writing it the first time was the log statement announcing the Kaniko command. It printed the full argv, which meant every --build-arg value, secrets included, would have landed verbatim in build logs the first time anyone configured a real one. Redacted to --build-arg=NAME=*** before it ever gets logged.

There's a leftover tradeoff I decided not to solve rather than pretend didn't exist: ARG/ENV values do get written into the builder stage's intermediate layers, even though Kaniko only pushes the final stage to ECR. Not a leak into the shipped image, but not nothing either, and worth knowing if a DATABASE_URL is sitting in there rather than just a public API base URL.

Problem 2: A File That Was Right There

The first real deploy after that fix didn't get anywhere near testing whether the env vars worked. The build task itself wouldn't start.

Task stopped at: 2026-08-26T05:06:16.794Z
exec /home/app/main.sh: no such file or directory

main.sh is two lines. #!/bin/bash, then exec node script.js. It's the image's entrypoint, unchanged in months, and the message reads like it isn't there.

First guess was a stale task definition pointing at an old image digest instead of the tag. Checked the task definition's JSON directly. It referenced the tag, not a pinned digest, so that was ruled out immediately rather than argued about.

Second guess came from actually looking at the ECR console instead of just the task logs. The image's artifact type read "Image Index," not "Image." An Image Index is a manifest list, meant for images that target more than one platform. Opening it showed two entries. One was a normal amd64/linux manifest. The other listed its architecture and OS as unknown, unknown.

That matched something I'd read about before but never actually hit: recent Docker Desktop versions wire plain docker build through the BuildKit/buildx driver by default, and that driver attaches build provenance and SBOM attestations even for an ordinary two-step docker build and docker push, not just an explicit buildx build --push. An attestation isn't a runnable image. It's a JSON manifest describing how the build happened, sitting in the same index as the real image, which is exactly why it shows up as unknown/unknown rather than a real platform. It was a completely plausible explanation for a container that can't be found: whatever's resolving the pull could be landing on the wrong entry in that list.

Rebuilding with --provenance=false fixed the ECR side of it cleanly. Artifact type went back to a plain "Image," one manifest, amd64/linux. Redeployed the task.

Same error. Same message, same container, same everything.

That's the point where continuing to iterate against ECS stopped making sense, since every cycle costs a task launch and a few minutes of logs to find out whether a guess helped. Reproducing it locally instead was faster and, it turned out, actually correct in a way the previous test hadn't been:

docker run --rm --entrypoint sh builder-image:latest -c "ls -la /home/app/ && cat main.sh"

main.sh was there. Right size, right permissions, and printed exactly the two lines it was supposed to. Nothing about that output explained the failure, because that test never actually explained anything. Overriding the entrypoint to sh -c "cat ..." reads bytes off disk. It never touches the shebang line the way the container's real ENTRYPOINT ["/home/app/main.sh"] does when the kernel parses it looking for an interpreter to hand the script to.

So I ran it without the override, on the same Windows machine, nowhere near AWS:

docker run --rm builder-image:latest
exec /home/app/main.sh: no such file or directory

Identical error, entirely local. That single result did more than either of the two ECR theories combined. It meant this had nothing to do with architecture, nothing to do with ECS's pull path, and nothing to do with the attestation manifest, which was a real thing worth fixing on its own merits but never the actual cause.

What's invisible in a normal terminal is a trailing carriage return. cat main.sh prints #!/bin/bash and a plain newline swallows the difference completely if there's a stray \r sitting right before it. The kernel doesn't swallow anything. It reads the shebang line literally, character for character, to find the interpreter, so a line ending in \r\n instead of \n sends it looking for a file named /bin/bash with a carriage return stuck to the end. That path has never existed on any Linux system. Exec fails, and the error it gives back, "no such file or directory," is completely accurate about a file that was never the one actually missing. main.sh was there. /bin/bash was there. The one specific byte connecting them wasn't.

Building from D:\Nextgen\dreamer-core explains the rest. Somewhere between a Windows checkout, a possible core.autocrlf setting, or an editor resave, main.sh had picked up CRLF line endings while nothing else in the repo had. It's a two-line file nobody had opened in months, which is exactly the kind of file that goes unnoticed.

Fixed it in two places instead of one, on purpose. Locally, so the current image builds clean:

git config core.autocrlf false
(Get-Content -Raw main.sh) -replace "`r`n","`n" | Set-Content -NoNewline main.sh

And in the Dockerfile itself, so this can't come back from some future checkout without anyone noticing it happened:

COPY . .
RUN sed -i 's/\r$//' main.sh && chmod +x main.sh

The second one is the fix I actually trust. The first one only holds until the next person clones the repo on Windows without knowing to check.

Final Result

Same task definition, same image name, one line different in the Dockerfile. The build task launched, ran, and finished, for the first time carrying the project's own env vars into the Kaniko build that had been silently missing them since the SSR pipeline first shipped.

Closing Thoughts

Neither of these was where I expected to spend time. The env var gap was invisible by construction, since the platform's own internal config sat in the exact same array as a project's, and nothing about that array told you which was which unless you went and checked. The shebang bug was invisible for a completely different reason: every individual piece of evidence, the file's presence, its permissions, its printed contents, looked correct in isolation, and the one thing that wasn't correct doesn't render as a visible character in a terminal at all.

Both times, the test that actually mattered was the one that used the real thing instead of an approximation of it. Grepping the whole repo for --build-arg instead of assuming the pipeline handled it because the runtime side clearly did. Running the container with its actual entrypoint instead of a sh -c override that only ever proved the file existed, never that it could run. The approximations weren't wrong exactly. They just answered a slightly different question than the one that was actually failing, and it took getting an identical failure on a machine with no ECS involved at all to see where the two questions had split apart.