Skip to content

Troubleshooting

Findings from real deployments, each with its symptom, cause, and fix. Every entry here was observed against a live instance, not constructed.

A proxy or gateway decodes encoded slashes (%2F)

Symptom. Every by-path project or group operation fails, while listing collections works. Without this role's handling you would see an unexplainable 404 on a resource that exists; with it you see either:

a proxy in front of GitLab is decoding URL-encoded slashes: GET
/api/v4/projects/root%2Fansible-demo was redirected to
/api/v4/projects/root/ansible-demo, which is not a GitLab API route. ...

or the run proceeds with this warning:

[WARNING]: a proxy in front of GitLab decodes URL-encoded slashes, so
'root/ansible-demo' was resolved to numeric id 1 through the projects
collection and the GET proceeded. This costs one extra request per resource
and cannot cover endpoints outside /projects and /groups; fix the proxy to
pass request paths through unchanged.

Cause. GitLab addresses projects and groups by URL-encoded full path — /api/v4/projects/group%2Fproject. Infrastructure between the client and GitLab (most commonly a Kubernetes gateway or ingress) normalizes the request path, decoding %2F into a literal / before GitLab sees it. The decoded path is not a valid API route, so the request is redirected and then 404s. Every by-path API client breaks identically behind such a gateway — python-gitlab, Terraform's GitLab provider, and glab included. This was observed live behind a Kubernetes gateway whose responses carried no identifying headers (bare HTTP/2, 307, stripped server header).

The acid test. Run this against your instance; repeat after each infrastructure change:

curl -sS -o /dev/null -w '%{http_code}\n' \
  -H "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "$GITLAB_URL/api/v4/projects/<group>%2F<project>"

200 means the path arrived intact. 307 (or 301/308) means something is still rewriting it.

What the role does about it. By-path requests are never blindly redirected. When the redirect matches the decoded-slash signature on a project- or group-scoped path, the role resolves the identifier to its numeric id through the collection endpoint (which carries no encoded slash), reissues the operation, and warns once per task with the message above. The resolution is cached for the task, costs one extra read per resource, and covers only /projects/... and /groups/... paths — which is the entire by-path surface this role uses today. Any other redirect is a named failure instructing you to point target.url at the canonical GitLab address.

The real fix is in the gateway, not the role: it must forward the request path unmodified. Where that is controlled depends on the data plane — for Envoy-based gateways the relevant behaviour is path normalization and its escaped-slash handling (path_with_escaped_slashes_action; it must keep encoded slashes unchanged rather than unescape or reject them); nginx-based ingresses decode the URI whenever a rewrite or a proxy_pass with a URI part is in play. GitLab's own reference architectures assume the encoded path reaches Workhorse untouched.

Status. Observed live on a Kubernetes gateway; the role's fallback kept that deployment fully functional (discovery, snapshot, converge, idempotence, and check mode all validated through it) until the gateway was fixed, at which point the acid test returned 200 and the fallback went dormant without any role change. The entry is retained because the pattern recurs across ingress and gateway stacks; the fallback remains in place as insurance.

Dependency proxy settings fail: "singleton is not exposed by this instance"

Symptom. Declaring dependency_proxy or dependency_proxy_ttl_policy on a group fails (strict mode) or skips (compatible mode) with:

GroupDependencyProxySettings singleton is not exposed by this instance: the
parent is readable but the settings object is null, which usually means the
owning feature is disabled in the instance configuration; it cannot be
created through the API (enabling the feature is documented at
https://docs.gitlab.com/administration/packages/dependency_proxy/)

Every other group setting on the same declaration converges normally.

Cause. The dependency proxy is an instance-level feature read from GitLab's deployment configuration at boot, not a database setting. When it is off, GitLab returns null for every group's dependencyProxySetting and dependencyProxyImageTtlPolicy GraphQL objects and refuses the update mutations outright — even for an administrator token. No API (REST, GraphQL, or Rails console) can enable it, so the role cannot route around the refusal; it names the cause instead. Helm chart deployments ship with the feature disabled by default; Omnibus enables it by default.

The acid test. Ask GitLab directly for any existing group:

curl -sS -H "Authorization: Bearer $GITLAB_TOKEN" \
  -H "Content-Type: application/json" "$GITLAB_URL/api/graphql" \
  -d '{"query":"query { group(fullPath: \"<group>\") { dependencyProxySetting { enabled } } }"}'

"dependencyProxySetting":null with a readable group means the feature is off. A mapping (even {"enabled":false}) means the feature is on and the role will converge the declared settings.

The real fix is in the deployment, not the role. On the Helm chart set global.appConfig.dependencyProxy.enabled: true (plus its object-storage bucket, gitlab-dependency-proxy by default) and redeploy; on Omnibus the feature is on unless gitlab_rails['dependency_proxy_enabled'] = false was set. GitLab's documentation: https://docs.gitlab.com/administration/packages/dependency_proxy/.

Status. Observed live on the reference Helm deployment with the chart default. Enabling the feature in the deployment made the identical declaration converge with no role change, from the same apply that created the group.

A 404 carries "version skew is a likely cause"

Symptom. A failure on an instance outside the role's verified profile set ends with:

; version skew is a likely cause: the endpoint is declared by the v19.2-ee
catalogue while the instance runs 17.10.1-ee, so the surface may not exist
on this GitLab version or may have been moved or removed upstream

Cause. Every path the role calls is declared by its committed catalogue, so a 404 has three candidate causes the status alone cannot separate: the resource genuinely does not exist, access hides it, or the instance's GitLab version does not serve the endpoint. When the instance is outside the verified set, the role names the third candidate — hedged, appended after GitLab's own error detail, never replacing it. Instances matching a verified profile never see the fragment.

What to do. Read the first half of the message normally (a typo'd path or a permissions gap is still the common case). If those check out, the endpoint likely post-dates your GitLab version or was removed from it — the audit-event-destination migration in this role's own history was found by exactly such a 404. See GitLab version support for the support model.