Day 90 of DevOps, Day 40 of Azure. Both tasks today were about a permission that exists, reads back correctly, and grants nothing.
On Linux, that is the ACL mask, which can clip an entry you just created down to nothing while ls -l reports something else entirely. On Azure it is an access policy written against the wrong one of a principal's two GUIDs, which ARM accepts without complaint and which authorises nobody.
One Ansible task, one Azure task. Set three different ACLs on three servers, then build a Key Vault and prove an encrypt and decrypt round trip. The tasks come from the KodeKloud Engineer platform.
Task 1: Three ACLs on three servers
The task: a different file and a different ACL on each app server. Group tony read on one, user steve read and write on another, group banner read and write on the third.
Step 1: Three plays, not one
- hosts: stapp01 become: yes tasks: - name: Create empty file file: path: /opt/security/blog.txt state: touch - name: Set ACL for group tony acl: path: /opt/security/blog.txt entity: tony etype: group permissions: r state: present - hosts: stapp02 ... Three plays in one playbook, each targeting one host. The alternative is one play with when: inventory_hostname == ... on every task, which reads worse and runs every task against every host just to decide to skip it.
Worth the contrast with Day 86, where one play served three hosts because the only thing varying was a value that {{ ansible_user }} supplied per host. Here the filenames and the ACL types differ, so the plays differ.
The output confirms plays run in sequence: PLAY [stapp01] finishes before PLAY [stapp02] starts. Within a play, Ansible parallelises across hosts, as Day 85 showed. Between plays it does not.
Step 2: Two things the short module name hides
acl is ansible.posix.acl, from the ansible.posix collection. Not ansible.builtin, and not shipped in ansible-core, so it needs ansible-galaxy collection install ansible.posix on a control node that does not already have it. The lab had it. A fresh one would fail at task resolution, which has nothing to do with ACLs and reads as it does.
That is three naming patterns in one batch, none visible in the playbook. Day 87's yum resolves to ansible.builtin.yum, which is itself a redirect to the dnf module. Day 89 used the fully qualified ansible.builtin.blockinfile. This one resolves outside core entirely.
And state is doing real work. Its default is query, which "gets the current ACL without changing it, for use in register operations". So a task that omits state reads the ACL, changes nothing, reports no change, and looks like it worked. state: present is not boilerplate.
Step 3: The mask, and why ls -l disagrees with getfacl
This is the finding worth keeping, and it is sitting in plain sight in the output:
# stapp02 -rw-rw-r--+ 1 root root story.txt user:steve:rw- group::r-- mask::rw- getfacl says the owning group has r--. ls -l shows the group bits as rw-. Those contradict each other, and both are correct.
When a file has extended ACLs, the group permission bits in ls -l are the mask, not the owning group's entry. The mask is a ceiling applied to every named user, every named group and the owning group, and setfacl recalculates it by default to cover whatever those entries ask for. steve asked for rw, the mask widened to rw-, and ls -l now reports rw- in the group field while the owning group still has only r--.
The + at the end of the mode string is the only clue any of this is happening. One character.
Two consequences, and the second is the one that catches people.
Reading ls -l on a file with an ACL tells you the ceiling, not the group's access. On this file, it looks like "group can write", which is wrong in two different ways at once.
And an ACL entry can exist and grant nothing. A mask::r-- with a user:steve:rw- entry means steve has the entry and cannot write, because the mask clips it. getfacl flags that with an #effective: comment. It did not happen here, because the module's recalculate_mask defaults to default and setfacl widened the mask to fit. Set the mask explicitly, or pass recalculate_mask: no_mask, and you can produce exactly that: a permission present, correct in intent, and ineffective.
What the docs recommend
Verify ACLs with getfacl, which is what this task did on all three servers.
ls -la alone would have shown the + and the mask and none of the entries, which on stapp02 would have read as "group can write" and been wrong. The task asked for an ACL, and getfacl is the only command that shows one.
Worth saying because yesterday's verification checked content and skipped the mode and ownership the task also named. This one checked the thing that was genuinely hard to get right, with the command that shows it.
One other thing to fix: state: touch on the file tasks means this playbook reports changed on every run forever, for Day 84's reason. The ACL tasks are idempotent. The file creation is not, unless modification_time: preserve and access_time: preserve are added.
Task 2: A Key Vault and a round trip
The task: a vault, an RSA-4096 key, an access policy granting exactly Get, List, Encrypt and Decrypt, and an encrypt and decrypt of a file.
Step 1: The whole task was one GUID
az account show --query user.name -o tsv -> a1107d7d-9373-4903-a81c-21b8d7531bce token "oid" claim -> 18970e52-5cf3-4f58-ad2c-7fbffab9e2b8 The lab identity is a service principal, and for a service principal user.name is the application ID. az keyvault set-policy --object-id wants the object ID, which is the identifier of the service principal object in the directory. Two different GUIDs for the same principal.
Pass the app ID to --object-id, and ARM accepts it. It creates a policy entry pointing at nothing. The vault then reports an access policy that looks entirely correct, grants access to nobody, and every crypto call returns Forbidden.
Both values appear in one az keyvault show response, which makes it concrete:
"accessPolicies": [ { "objectId": "18970e52-..." } ], "systemData": { "createdBy": "a1107d7d-...", "createdByType": "Application" } ARM's audit trail records the app ID. The access policy records the object ID. Same principal, same JSON document, two values you cannot substitute for each other.
Which is this morning's ACL mask at a different scale. An entry clipped by the mask is present and ineffective. A vault policy written against the app ID is present and ineffective. Both read back fine, and the only way to find out is to attempt the operation it was supposed to permit.
Step 2: Getting the object ID when Graph is closed
Every obvious route goes through Microsoft Graph, and the lab identity cannot read it. az ad sp show is denied. az keyvault set-policy --spn resolves the app ID through Graph. az role assignment list --all returned [].
The access token already contains the answer. Its oid claim is the object ID of the authenticated principal:
OID=$(az account get-access-token --query accessToken -o tsv \ | python3 -c "import sys,base64,json; p=sys.stdin.read().strip().split('.')[1]; p+='='*(-len(p)%4); print(json.loads(base64.urlsafe_b64decode(p))['oid'])") The padding line matters. JWTs use base64url with the padding stripped, so b64decode fails on most tokens without '='*(-len(p)%4) added back.
Worth keeping for any Azure task that needs "the object ID of whoever I am" with Graph closed. The payload also carries appid, tid and sub, which covers most identity questions a script will ask.
--no-self-perms on the create solves the same problem from the other end, because by default az keyvault create writes a policy for the caller, which means resolving the caller's object ID, which means Graph. Without the flag, that step either warns and produces a vault with no policies or fails the create outright and leaves half a vault behind.
Step 3: The order of operations is forced
set-policy --key-permissions get list create encrypt decrypt # then create the key set-policy --key-permissions get list encrypt decrypt # then narrow set-policy --key-permissions replaces that category's permission array. It does not add to it.
The task asks for exactly Get, List, Encrypt and Decrypt, and none of those four permits creating a key. So grant create temporarily, create the key, then narrow. The other order gives Forbidden on keyvault key create, and the error names the operation rather than the policy, which sends you off looking at RBAC.
Fourth command in this challenge that replaces rather than merges, after az resource tag, the S3 notification configuration and the lifecycle policy. With one mercy: set-policy is scoped per object ID and per category, so it replaces one cell of the matrix rather than the whole table.
And then re-verify, because the round trip was first proven while create was still in the policy, which proves nothing about the state being left behind. That is the AWS Day 38 lesson with the terms swapped: there, configuration was right and behaviour untested. Here, behaviour was tested against a configuration that no longer existed.
What the docs recommend
Two corrections to my own notes on soft delete, and both matter more than the thing I got right.
I wrote that soft delete "cannot be disabled; it has been mandatory since 2020". The first half is documented: "Once soft-delete is enabled on a key vault, it can't be disabled." The 2020 date is not on that page, which instead says it is on by default for new vaults and that customers are "strongly encouraged to turn on soft delete enforcement for their vaults via Azure Policy". That reads as a default rather than an absolute, so I am recording the date as unverified.
The default retention is 90 days, not 7: "You can set it anywhere from 7 to 90 days, with 90 days being the default." So --retention-days 7 was necessary to meet the requirement rather than a restatement of the default, which I had backwards on.
And it is permanent: "The retention interval can only be configured during key vault creation and can't be changed afterwards." That vault has a seven-day window for its entire life, and changing it means a different vault.
The name reservation I flagged is confirmed and broader than I stated: the name cannot be reused until retention expires, a vault with the same name can't be created in the same location, and "The DNS record corresponding to the vault is also retained during the retention interval." Which is why the discovery step ran az keyvault list-deleted before creating anything.
One last thing that is worth knowing before using this pattern for real. RSA cannot encrypt more than its modulus allows, and for a 4096-bit key with RSA-OAEP the ceiling is 470 bytes of plaintext. The file here was 26 bytes. Beyond the limit, the answer is wrapKey and envelope encryption, encrypting a symmetric data key in the vault and the actual file locally with AES, which is how Azure Disk Encryption works and why "encrypted with Key Vault" rarely means the bytes went to Key Vault.
What I learned
Both halves are the same failure in two systems: a permission that is syntactically fine, visible when you look, and does nothing.
The ACL mask is the more elegant one, because the system is actively telling you something misleading. ls -l reports the mask in the group field, so a file where the group has read-only displays as read-write, and the only signal that the number you are reading is not the number you think is a single + at the end of the mode string.
The Key Vault GUID is the more expensive one. ARM takes the wrong identifier without complaint, the vault displays a policy that looks correct, and every operation fails with Forbidden, which points at permissions in general rather than at one wrong field. Two GUIDs, same principal, both in the same response, and the hint the task gives you points at the wrong one.
The thing both have in common is that reading the configuration back is not a check. In both cases, it reads back exactly as intended. The only test is attempting the operation the permission was supposed to allow, which is what I eventually did in one case and what the mask happened to spare me in the other.
So here is the Day 90 question. Which of your access rules have you confirmed by reading them, rather than by watching someone successfully use them?
Day 90 down. Ten to go.

