When you run systemctl start myapp.service and get this in the journal:
myapp.service: Failed at step EXEC spawning /opt/myapp/bin/server: Permission denied
myapp.service: Failed with result 'exit-code'.
…and the binary is 755, owned by the right user, and runs perfectly fine from the command line — you’ve stumbled into one of systemd’s most disorienting failure modes. The log points at the binary. The binary looks fine. So you chmod things that don’t need to be chmoded, check things that are already correct, and stay stuck. Here’s what’s actually going on, the exact diagnostic steps I use, and how to confirm the fix actually held.
Context: The Setup That Broke
I was deploying a compiled Go binary as a system service on Ubuntu 22.04 (systemd 249). The unit file was nothing exotic:
[Unit]
Description=My App
After=network.target
[Service]
Type=simple
User=myapp
ExecStart=/opt/myapp/bin/server
WorkingDirectory=/opt/myapp
Restart=on-failure
[Install]
WantedBy=multi-user.target
The binary had chmod 755 applied. The myapp user existed, had a home directory, and was in the right groups. Running /opt/myapp/bin/server manually after sudo su - myapp worked without any complaint. The process started, bound its port, logged its startup line — everything was correct.
Then I ran sudo systemctl start myapp.service and it died instantly with Permission denied. No helpful follow-up message. Just the error and the service sitting in failed state.
The Wrong Path: Chasing the Binary
The natural first move when you see “Permission denied” on a binary is to check the binary. So I spent the next 30 minutes doing exactly the wrong things.
ls -l /opt/myapp/bin/server showed -rwxr-xr-x. 1 myapp myapp 18234720 Jun 08 12:34 server. Executable by everyone, owned by myapp. I ran chmod +x on it anyway — no change. I double-checked that the myapp user existed with id myapp. I verified the unit file path was spelled correctly. I ran sudo -u myapp /opt/myapp/bin/server directly from the shell — it started right up. I restarted the service three more times, which is the sysadmin equivalent of blowing on a game cartridge.
None of it helped because the file permissions were never the problem.
The part that finally clicked: the journal log says “Failed at step EXEC spawning /opt/myapp/bin/server” — and I was reading that as “couldn’t execute the file at that path”. But the actual issue was one level higher. The kernel hadn’t even gotten to the file yet.
What systemd Does Before execve() — and Where Permission Denied Actually Comes From
When systemd launches an ExecStart command, PID 1 prepares the new process and eventually calls execve(). Before execve() can succeed, the kernel must traverse every directory in the path and verify that the calling context has execute (x) permission on each directory component.
This is the standard Linux path traversal permission check. To reach /opt/myapp/bin/server, the kernel must traverse /opt, then enter /opt/myapp, then enter /opt/myapp/bin, and finally reach the file. Each directory lookup requires execute permission on that specific directory for the calling process. If any directory in the chain denies the traverse, the kernel returns EACCES — and systemd logs it as “Permission denied” pointing at the final path, which is deeply misleading because the final path is not what failed.
The critical twist is the ordering: the User= switch in the service unit happens after the exec setup, not before the path check. So the effective user context during the initial path traversal is not the User=myapp value you specified. This is why running the binary manually as myapp works fine — you are myapp from the start, the traverse checks succeed because the owner (myapp) has execute on every directory. But systemd’s PID 1 spawn path sees a different effective context for that initial walk.
The Fix: Use namei to Audit the Entire Path
The tool that solves this in one shot is namei with the -l flag. It prints permission and ownership information for every component in the path:
namei -l /opt/myapp/bin/serverSample output:
f: /opt/myapp/bin/server
drwxr-xr-x root root /
drwxr-xr-x root root opt
drwx------ myapp myapp myapp <- 700: no world-execute
drwxr-xr-x myapp myapp bin
-rwxr-xr-x myapp myapp server
There it is. The /opt/myapp directory was 700 — owner-only permissions. When I created that directory during initial setup I’d locked it down out of habit, thinking I was being tidy with application directory security. The myapp user can traverse it fine as the file owner. The PID 1 spawn context cannot.
The fix:
chmod 755 /opt/myappIf you want the directory’s file listing to stay hidden from other users (no world-read) but still allow path traversal, just add world-execute without world-read:
chmod o+x /opt/myapp # world-execute, no world-readAfter the chmod, clear systemd’s failure state and restart:
sudo systemctl daemon-reload
sudo systemctl reset-failed myapp.service
sudo systemctl start myapp.service
Two notes: reset-failed is necessary because a service in failed state is refused on a plain start until explicitly cleared. Systemd keeps failed services sticky by design so logs don’t get trampled. daemon-reload is only strictly required if you modified the unit file, but it’s cheap to run and a good habit after any change in the service stack.
Verification: Confirming the Fix Held
Check the service status:
sudo systemctl status myapp.serviceYou want to see:
● myapp.service - My App
Loaded: loaded (/etc/systemd/system/myapp.service; enabled)
Active: active (running) since Tue 2026-06-09 14:32:01 UTC; 3s ago
Main PID: 18402 (server)
Tasks: 6 (limit: 9830)
Memory: 12.1M
CGroup: /system.slice/myapp.service
└─18402 /opt/myapp/bin/server
If you still see failed, the specific exit code in the journal pinpoints which layer is failing:
sudo journalctl -u myapp.service -n 30 --no-pagerThe two codes that matter most:
status=203/EXEC—execve()itself failed. This is always a permissions, path, or interpreter issue. You haven’t fully fixed it yet — runnamei -lagain and look more carefully.status=1/FAILURE— The process started but exited non-zero. This is a different problem category entirely. Your application launched successfully and then crashed due to its own logic. Check the application’s own log output in the same journal block.
Also watch for No such file or directory appearing alongside 203/EXEC. This usually means either the ExecStart path contains a typo, or the binary has a broken shared library dependency. Test with:
ldd /opt/myapp/bin/serverAny line showing not found is a library that needs to be installed or added to LD_LIBRARY_PATH via an Environment= directive in the unit file.
Edge Cases and Gotchas
Shell scripts with a shebang add a second permissions check. If your ExecStart points to a shell script rather than a compiled binary, systemd must execute both the script file and the interpreter named in the shebang. A script starting with #!/usr/bin/env python3 works interactively but can fail under systemd’s stripped PATH environment. Use absolute paths in every shebang: #!/usr/bin/python3, #!/bin/bash. This removes the variable.
Scripts created on Windows introduce another silent killer: CRLF line endings. A shebang of #!/bin/bash will fail because the kernel looks for an executable named bash followed by a carriage return character — which doesn’t exist. The error is usually 203/EXEC with a vague message. Diagnose with:
file myscript.shIf it reports CRLF line terminators, convert it:
dos2unix myscript.shSELinux and AppArmor produce errors that look identical to filesystem permission denials. If namei -l shows clean permissions on every path component and the binary has correct execute bits, check the security module audit logs before doing anything else:
# RHEL / CentOS / Fedora (SELinux):
sudo ausearch -m avc -ts recent
Ubuntu / Debian (AppArmor):
sudo journalctl -k | grep -i apparmor
To quickly confirm the hypothesis, put SELinux into permissive mode temporarily: sudo setenforce 0, then try starting the service. If it starts, SELinux policy is the blocker. The proper fix is a context relabel (restorecon -Rv /opt/myapp) or a custom policy module — not chmod 777.
systemd sandbox directives can block previously-working services after an OS upgrade without any change to your application code. Options like ProtectSystem=strict, DynamicUser=yes, PrivateTmp=yes, and NoNewPrivileges=true restrict what filesystem paths the service process can see. If a unit file you inherited worked before and now fails after an upgrade, scan the [Service] section carefully for these options. Add explicit ReadWritePaths= entries for the paths your binary writes to, and ReadOnlyPaths= for paths it only reads.
noexec mount options appear on hardened servers and in certain container base images. If the filesystem partition your binary lives on was mounted with noexec, execve() will always fail regardless of the file’s permission bits:
mount | grep noexec
findmnt --target /opt/myapp
If noexec appears, either remount without that flag or relocate the binary to a standard path like /usr/local/bin that is not subject to the noexec restriction.
TL;DR Fix
namei -l /path/to/your/ExecStart/binaryFind the directory component showing 700 or missing world-execute, fix it with chmod 755 or chmod o+x, then: