The disk-space full is more like to happen on the PA-200, 220 platform, which has a smaller disk size than other platforms. Symptoms may vary by filling up disk-space depends on which directory.
Here are symptoms of disk-space getting filled up are:
- Symptom
- Web interface not loading
- Certain daemons processes not starting
- Incomplete tech support bundles
- System log alerts with "Disk usage for / exceeds limit, X percent in use, cleaning file system"
- Unable to download PAN-OS software images or dynamic updates
- GlobalProtect HIP checking fails
- GlobalProtect HIP checking fails
- Cause
The PAN-OS file system is divided into various directories. To view partitions and associated disk-space, use the command below:show system disk-space
Filesystem Size Used Avail Use% Mounted on
/dev/md2 3.9G 3.8G 588K 100% / << root partition is full
/dev/md5 7.6G 4.2G 3.0G 59% /opt/pancfg .
/dev/md6 3.8G 3.0G 666M 82% /opt/panrepo
tmpfs 2.0G 210M 1.8G 11% /dev/shm
cgroup_root 2.0G 0 2.0G 0% /cgroup
/dev/md8 88G 2.4G 81G 3% /opt/panlogs
tmpfs 12M 0 12M 0% /opt/pancfg/mgmt/lcaas/ssl/private
The PAN-OS file system has a default mechanism to rotate and clear disk-space. In some cases, manual intervention may be required to clear disk space.
- Resolution
Option 1: Enable Aggressive Cleaning
This will automatically truncate all old log files (entries under all *var/log/pan directories matching *.1, ... *.4, *.log.old) if the 95% occupancy alarm is tripped.
> debug software disk-usage aggressive-cleaning enable
> debug software disk-usage aggressive-cleaning disable
To verify the changes or if it is already enabled use the command below.
> show system state | match aggressive-cleaning
cfg.debug-sw-du.config: { 'aggressive-cleaning': True, }
In PAN-OS versions 8.1.0+, cleanup can be done manually using the command below:
> debug software disk-usage cleanup ?
+ deep cleanup with deleting backup logfile
* threshold percentage threshod of size of system partition to kick off cleanup
> debug software disk-usage cleanup deep threshold 90
Note that this command has to be run manually each time to bring the disk usage to below 90% and is not persistent across reboots.
Caveat: Enabling aggressive clean up may clear up logs, rendering logs unavailable for analysis.
† Tip: Create an alert for the disk-full on syslog.
The firewall generates a warning alert on the system logs. If you have a syslog system, you may parse a word like 'Disk usage for / exceeds limit' and create an alert when detected.
"Disk usage for / exceeds limit, 95 percent in use, cleaning filesystem."
Option 2: Check and Delete Unnecessary Core Files
show system files
/opt/dpfs/var/cores/:
total 4.0K
drwxrwxrwx 2 root root 4.0K Jun 10 20:05 crashinfo
/opt/dpfs/var/cores/crashinfo:
total 0
/var/cores/:
total 115M
drwxrwxrwx 2 root root 4.0K Jun 10 20:15 crashinfo
-rw-rw-rw- 1 root root 867M Jun 12 13:38 devsrvr_4.0.3-c37_1.gz
-rw-rw-rw- 1 root root 51M Jun 12 13:39 core.20053
/var/cores/crashinfo:
total 16K
-rw-rw-rw- 1 root root 15K Jun 10 20:15 devsrvr_4.0.3-c37_0.info
Delete unnecessary core files. A core file can be deemed unnecessary if investigation around the core file is complete or they are very old files.
(This example deletes a device server core file from the management-plane.)
> delete core management-plane file devsrvr_4.0.3-c37_1.gz
Option 3: Delete Rotated Files and Files with Extension .old
> delete debug-log ?
> cp-log Remove cp-log at /opt/var.cp/log/pan/
> dp0-log Remove dp0-log at /opt/var.dp0/log/pan/
> dp1-log Remove dp1-log at /opt/var.dp1/log/pan/
> dp2-log Remove dp2-log at /opt/var.dp2/log/pan/
> mp-global Remove mp-global at /opt/mp-global/
> mp-log Remove mp-log at /var/log/pan/
delete debug-log mp-log file *.1
delete debug-log mp-log file *.2
delete debug-log mp-log file *.3
delete debug-log mp-log file *.4
delete debug-log mp-log file *.old
show system disk-space
Option 4: Clear Any packet-diag Logging If Enabled
debug dataplane packet-diag show setting
DP dp0:
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
Enabled: no
Match pre-parsed packet: no
--------------------------------------------------------------------------------
Logging
Enabled: no <<< No is the default value and means not enabled.
debug dataplane packet-diag show setting
DP dp0:
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
Enabled: no
Match pre-parsed packet: no
--------------------------------------------------------------------------------
Logging
Enabled: yes <<< Means packet-diag is enabled.
To clear out packet-diag setting and the logs, run below commands:
debug dataplane packet-diag clear all
Packet diagnosis setting set to default.
debug dataplane packet-diag clear log log
dataplane debug logs cleared
Option 5: Delete Any Debug pcaps or Debug-filter pcaps
delete pcap directory *
Option 6: Delete Old Content and Antivirus Update Packages
delete anti-virus update
Apart from these options, there have been various improvements in the latest PAN-OS release to best utilize disk-space.
※ Last resort: Open a case with Palo Alto TAC
After you open a case, you will need a TAC engineer who has the root permission. Some of the outsourced TAC engineers don't have the root access.
The Palo Alto TAC engineer should execute the following steps. The debug tac-login is only intended for use by a TAC engineer, and access is restricted outside of TAC.
debug tac-login challenge
admin@firewall> debug tac-login challenge
Please use the following string as your challenge:
VTBaCbIyxzgxMBwxpjD8OPAxVDJvOFk0TnyrOU0unABCDu85ky1taIMxuwZZYq+u
debug tac-login response
admin@firewall> debug tac-login response
Please enter challenge response (^C to cancel):
-----BEGIN RESPONSE-----
JLAjLUyT7/V3K+gKrDE/TXyqvennFNoiVyekBBBkdyhi336yIaib+6Z2A3hCBBQjWcanWqedGx1
55SKUqMO6X0DWa1i0KG+g2vWgJpshvdbW8tivCC8gpVpgvnjpbVq9NLwechbeCf1eG0KAI/vPs6U
Uy9mJax/xsjXyFzhTzBSHycn5EzuYbJ7F/frDB8v2h6bJdl7t3T4TLk6/pAbb26CIJgt4YjieADo
3Ey6APGj3cqM98EnbspurweuIBE/gaPqlu5AB4K6O+yOzI40GKtLjY6wkXrfYuClTnsXnj8WaDDt
H/0e7kcwnI/d9tdUgp3/wpzGto2FyVMdxD1YNw==
-----END RESPONSE-----
* The above change and response codes are modified and scrambled, so it is not the real ones.
[root@firewall ~]# << TAC got into the root access on Linux
- Check the disk-space and directories on Linux with the root access.
[root@firewall]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 1.9G 1.6G 206M 89% /
/dev/sda5 6.6G 5.1G 1.3G 81% /opt/pancfg
/dev/sda6 1.9G 1.2G 619M 67% /opt/panrepo
tmpfs 1.4G 210M 1.2G 16% /dev/shm
/dev/mmcblk0p8 5.6G 5.6G 0 100% /opt/panlogs
- Go to Threat logs directory, and delete the old files and folders.
[root@firewall]# cd /opt/panlogs/logdb/threat/1
or
- Go to Traffic logs directory, and delete the old files and folders.
/opt/panlogs/logdb/traffic
- Check the current directory.
[root@firewall]# pwd
/opt/panlogs/logdb/threat/1
- Check the files or directories.
[root@firewall]# ls -ah
. 20200801 20200803 20200805 20200807 20200809 20200811 20200813 20200815 20200817 20200819 20200821 20200823 20200825 20200827
.. 20200802 20200804 20200806 20200808 20200810 20200812 20200814 20200816 20200818 20200820 20200822 20200824 20200826 20200828
[root@firewall]# ls -alh
total 120K
drwxrwxrwx 30 root root 4.0K Aug 28 19:55 .
drwxr-xr-x 3 root root 4.0K Oct 24 2019 ..
drwxrwxrwx 2 root root 4.0K Aug 2 02:15 20200801
drwxrwxrwx 2 root root 4.0K Aug 3 02:15 20200802
drwxrwxrwx 2 root root 4.0K Aug 4 02:15 20200803
...
snipped
- Check the disk-space of the folders, and sort the files or directories with numeric data present inside.
- The 'sort -n' option shows the largest file at the bottom, and 'sort -nr' shows the largest file at the top.
[root@firewall]# du -hs * | sort -n
4.4M 20200828
4.9M 20200801
...
snipped
...
5.0M 20200826
5.0M 20200827
6.4M 20200825
[root@firewall]# du -hs * | sort -nr
6.4M 20200825
5.0M 20200827
5.0M 20200826
...
snipped
...
4.9M 20200801
4.4M 20200828
- Delete the old files and folders.
[root@firewall]# rm -rf 2019*
- Logout from the Linux mode.
[root@firewall]#exit
logout
admin@firewall>
* Additional Information
For additional information, please review the following articles:
How to Delete Unnecessary Downloaded Software Versions
How to delete configurations through the CLI
How to Delete Saved Configuration Files
No comments:
Post a Comment