The Linux audit system can be used to collect important system events. It is often used for compliance with PCI DSS 3.1. But it is also useful for debugging certain problems, for example where you have to monitor an unknown number of processes to show a certain behavior. In our case an unknown process kept killing other processes form time to time.

Architecture

Audit works on events, which are recorded by the Linux kernel itself of are generated by user space programs. The events are processed inside the Linux kernel and pass through several lists of rules.

The first list is exlucde, which can be used to filter our certain events. Processing too many events can lead to performance issues.

After that the event if processed by a second list depending on its type:

1. File events go though the fstype list
2. Events from user application go through user.
3. Process related events from fork() and exec() are processed through task.
4. System call related events go through the exit list.

Rules

Rules should be added to files in /etc/audit/rules.d/. They are concatenated by augenrules to a single file /etc/audit/audit.rules. This file is then loaded into the Linux kernel with auditctl -R or by augenrules --load.

Security

In strictly controlled environments auditing is required for conformance with regulatory policies. For example any action performed by an administrator should be logged. If failed access to files containing sensitive informations should be logged.

To prevent an attacker from changing the audit rules to hide his traces, the audit rules can be locked. This is done by calling auditctl -e 2. Any later attempt to load a new rule set will then be blocked. This can only be reset by rebooting the system, after which the audit starts again with an empty rule set.

Message types

The Audit framework uses different message types. You can get a list via ausearch -m. See the Appendix for list of message types and more information.

Keys

Each audit event can be tagged with a key, which can be specified via -k $KEY. This allows to group related rules by using a common key. This simplified later reporting and searching.

System calls

System calls are identified by numbers, which are system and architecture dependant. On multi-architecture systems (amd64 and i386) they might have different numbers. The audit tools know how to lookup the names, so stick to using them as often as possible.

If you want to filter on additional arguments you need to know which arguments is used for what. Ryan Chapman has a nice Linux System Call Table for x86_64. For example sys_kill is number 62 and expects the process ID as a0 and the signal number as a1.

Examples

Monitor any change to /etc/shadow:

-w /etc/passwd -p wa -k passwd_change

Catch all location invoking sudo:

-w /usr/bin/sudo -p x -k sudoer

Catch processes sending SIGTERM(15) successfully to other processes:

-a always,exit -F arch=b64 -S kill -F a1=15 -F success=yes -k killer

Many more examples are available in /usr/share/doc/auditd/examples/rules/.

Processing

Audit events are broadcasted vie NETLINK_AUDIT. There are two commonly used user space applications, which log those events to files.

auditd

By default auditd logs to /var/log/audit/audit.log. The file is not rotated by logrotate, but by auditd itself. This can be configured via /etc/audit/auditd.conf.

audispd

In addition to that the optional dispatcher daemon audispd can be used. It is implemented as a separate process to make auditd more secure and to move the load to a separate process. It is configured via /etc/audisp/audispd.conf and can be extended by plugins in /etc/audisp/plugins.d/:

• af_unix.conf can be used to write Audit events to a UNIX socket.
• syslog.conf can be used to forward Audit events to syslog.

journal

The systemd journald also listens for Audit events by default. This is controlled by systemd-journald-audit.socket. It can be disabled by masking the socket using systemctl mask systemd-journald-audit.socket.

Reporting

The log entries written by auditd to /var/log/audit/audit.log may need post-processing: The often consist of multiple lines and need further filtering.

There are two tools work with the collected information:

• aureport can be used to summary information.
• ausearch can be used for filtering.

Examples

Show often used programs:

aureport --summary --interpret  --executable

Show login attempts used today:

aureport --summary --interpret --auth --start yesterday --end today

Show used tags:

aureport --summary --interpret  --key

Show all events in human readable format with time stamps and strings hex-decoded:

ausearch -if /var/log/audit/audit.log -i

Show specific audit event 7148743:

ausearch -i -a 7148743

Show successful usages of syscall kill:

ausearch -i -sc kill -sv yes

Links

• Audit System Architecture
• RedHat knowledge base

Appendix

Message types

Each event uses a specific message type.

• Linux Audit Specification
• RedHat documentation
• RHEL Audit System Reference.

Here is my list of message types grouped by category:

• User and group account management:
  • ADD_USER: user-space user account is added.
  • USER_MGMT: user-space management data.
  • USER_CHAUTHTOK: user account attribute is modified.
  • DEL_USER: user-space user is deleted.
  • ADD_GROUP: user-space group is added.
  • GRP_MGMT user-space group management data.
  • GRP_CHAUTHTOK: group account attribute is modified.
  • DEL_GROUP: user-space group is deleted.
• User login live cycle events:
  • CRYPTO_KEY_USER: cryptographic key identifier used for cryptographic purposes.
  • CRYPTO_SESSION: parameters set during a TLS session establishment.
  • USER_AUTH: user-space authentication attempt is detected.
  • LOGIN: user log in to access the system.
  • USER_CMD: user-space shell command is executed.
  • GRP_AUTH: group password is used to authenticate against a user-space group.
  • CHUSER_ID: user-space user ID is changed.
  • CHGRP_ID: user-space group ID is changed.
  • PAM Authentication:
    • USER_LOGIN: user logs in.
    • USER_LOGOUT: user logs out.
  • PAM account:
    • USER_ERR: user account state error is detected.
    • USER_ACCT: user-space user account is modified.
    • ACCT_LOCK: user-space user account is locked by the administrator.
    • ACCT_UNLOCK: user-space user account is unlocked by the administrator.
  • PAM session:
    • USER_START: user-space session is started.
    • USER_END: user-space session is terminated.
  • Credentials:
    • CRED_ACQ: user acquires user-space credentials.
    • CRED_REFR: user refreshes their user-space credentials.
    • CRED_DISP: user disposes of user-space credentials.
• Linux Security Model events:
  • DAC_CHECK: record DAC check results.
  • MAC_CHECK: user space MAC (Mandatory Access Control) decision is made.
  • USER_AVC: user-space AVC message is generated.
  • USER_MAC_CONFIG_CHANGE
  • SELinux Mandatory Access Control:
    • AVC_PATH: dentry and vfsmount pair when an SELinux permission check.
    • AVC: SELinux permission check.
    • FS_RELABEL: file system relabel operation is detected.
    • LABEL_LEVEL_CHANGE: object’s level label is modified.
    • LABEL_OVERRIDE: administrator overrides an object’s level label.
    • MAC_CONFIG_CHANGE: SELinux Boolean value is changed.
    • MAC_STATUS: SELinux mode (enforcing, permissive, off) is changed.
    • MAC_POLICY_LOAD: SELinux policy file is loaded.
    • ROLE_ASSIGN: administrator assigns a user to an SELinux role.
    • ROLE_MODIFY: administrator modifies an SELinux role.
    • ROLE_REMOVE: administrator removes a user from an SELinux role.
    • SELINUX_ERR: internal SELinux error is detected.
    • USER_LABELED_EXPORT: object is exported with an SELinux label.
    • USER_MAC_POLICY_LOAD: user-space daemon loads an SELinux policy.
    • USER_ROLE_CHANGE: user’s SELinux role is changed.
    • USER_SELINUX_ERR: user-space SELinux error is detected.
    • USER_UNLABELED_EXPORT: object is exported without SELinux label.
  • AppArmor Mandatory Access Control:
    • APPARMOR_ALLOWED
    • APPARMOR_AUDIT
    • APPARMOR_DENIED
    • APPARMOR_ERROR
    • APPARMOR_HINT
    • APPARMOR_STATUS
    • APPARMOR
• Audit framework events:
  • KERNEL: record the initialization of the Audit system.
  • CONFIG_CHANGE: Audit system configuration is modified.
  • DAEMON_ABORT: Audit daemon is stopped due to an error.
  • DAEMON_ACCEPT: auditd daemon accepts a remote connection.
  • DAEMON_CLOSE: auditd daemon closes a remote connection.
  • DAEMON_CONFIG: Audit daemon configuration change is detected.
  • DAEMON_END: Audit daemon is successfully stopped.
  • DAEMON_ERR: auditd daemon internal error is detected.
  • DAEMON_RESUME: auditd daemon resumes logging.
  • DAEMON_ROTATE: auditd daemon rotates the Audit log files.
  • DAEMON_START: auditd daemon is started.
  • FEATURE_CHANGE: Audit feature changed value.
• Networking related:
  • IPSec:
    • MAC_IPSEC_ADDSA
    • MAC_IPSEC_ADDSPD
    • MAC_IPSEC_DELSA
    • MAC_IPSEC_DELSPD
    • MAC_IPSEC_EVENT: IPSec event, when one is detected, or when the IPSec configuration changes.
  • NetLabel:
    • MAC_CALIPSO_ADD: NetLabel CALIPSO DoI entry is added.
    • MAC_CALIPSO_DEL: NetLabel CALIPSO DoI entry is deleted.
    • MAC_MAP_ADD: new Linux Security Module (LSM) domain mapping is added.
    • MAC_MAP_DEL: existing LSM domain mapping is added.
    • MAC_UNLBL_ALLOW: unlabeled traffic is allowed.
    • MAC_UNLBL_STCADD: static label is added.
    • MAC_UNLBL_STCDEL: static label is deleted.
  • Message Queue:
    • MQ_GETSETATTR: mq_getattr and mq_setattr message queue attributes.
    • MQ_NOTIFY: arguments of the mq_notify system call.
    • MQ_OPEN: arguments of the mq_open system call.
    • MQ_SENDRECV: arguments of the mq_send and mq_receive system calls.
  • Netfilter firewall:
    • NETFILTER_CFG: Netfilter chain modifications are detected.
    • NETFILTER_PKT: packets traversing Netfilter chains.
  • Commercial Internet Protocol Security Option:
    • MAC_CIPSOV4_ADD: user adds a new Domain of Interpretation (DoI).
    • MAC_CIPSOV4_DEL: user deletes an existing DoI.
• Linux Cryptography:
  • CRYPTO_FAILURE_USER: decrypt, encrypt, or randomize cryptographic operation fails.
  • CRYPTO_IKE_SA: Internet Key Exchange Security Association is established.
  • CRYPTO_IPSEC_SA: Internet Protocol Security Association is established.
  • CRYPTO_LOGIN: cryptographic officer login attempt is detected.
  • CRYPTO_LOGOUT: cryptographic officer logout attempt is detected.
  • CRYPTO_PARAM_CHANGE_USER: change in a cryptographic parameter is detected.
  • CRYPTO_REPLAY_USER: replay attack is detected.
  • CRYPTO_TEST_USER: cryptographic test results as required by the FIPS-140 standard.
• Process:
  • BPRM_FCAPS: user executes a program with a file system capability.
  • CAPSET: any changes in process-based capabilities.
  • CWD: current working directory.
  • EXECVE; arguments of the execve system call.
  • OBJ_PID: information about a process to which a signal is sent.
  • PATH: file name path information.
  • PROCTITLE: full command-line of the command that was used to invoke the analyzed process.
  • SECCOMP: Secure Computing event is detected.
  • SYSCALL: system call to the kernel.
• Special system calls:
  • FD_PAIR: use of the pipe and socketpair system calls.
  • IPC_SET_PERM: information about new values set by an IPC_SET control operation on an IPC object.
  • IPC: information about a Inter-Process Communication object referenced by a system call.
  • MMAP: file descriptor and flags of the mmap system call.
  • SOCKADDR: record a socket address.
  • SOCKETCALL: record arguments of the sys_socketcall system call (used to multiplex many socket-related system calls).
• Systemd:
  • SERVICE_START: service is started.
  • SERVICE_STOP: service is stopped.
  • SYSTEM_BOOT: system is booted up.
  • SYSTEM_RUNLEVEL: system’s run level is changed.
  • SYSTEM_SHUTDOWN: system is shut down.
• Virtual Machines and Container:
  • VIRT_CONTROL: virtual machine is started, paused, or stopped.
  • VIRT_MACHINE_ID: binding of a label to a virtual machine.
  • VIRT_RESOURCE: resource assignment of a virtual machine.
• Device management:
  • DEV_ALLOC: device is allocated.
  • DEV_DEALLOC: device is deallocated.
• Trusted Computing Integrity Measurement Architecture:
  • INTEGRITY_DATA: data integrity verification event run by the kernel.
  • INTEGRITY_EVM_XATTR: EVM-covered extended attribute is modified.
  • INTEGRITY_HASH: hash type integrity verification event run by the kernel.
  • INTEGRITY_METADATA: metadata integrity verification event run by the kernel.
  • INTEGRITY_PCR: Platform Configuration Register (PCR) invalidation messages.
  • INTEGRITY_RULE: policy rule.
  • INTEGRITY_STATUS: status of integrity verification.
• Intrusion Prevention System:
  • Anomaly detected:
    • ANOM_ABEND
    • ANOM_ACCESS_FS
    • ANOM_ADD_ACCT
    • ANOM_AMTU_FAIL
    • ANOM_CRYPTO_FAIL
    • ANOM_DEL_ACCT
    • ANOM_EXEC
    • ANOM_LINK
    • ANOM_LOGIN_ACCT
    • ANOM_LOGIN_FAILURES
    • ANOM_LOGIN_LOCATION
    • ANOM_LOGIN_SESSIONS
    • ANOM_LOGIN_TIME
    • ANOM_MAX_DAC
    • ANOM_MAX_MAC
    • ANOM_MK_EXEC
    • ANOM_MOD_ACCT
    • ANOM_PROMISCUOUS
    • ANOM_RBAC_FAIL
    • ANOM_RBAC_INTEGRITY_FAIL
    • ANOM_ROOT_TRANS
  • Responses:
    • RESP_ACCT_LOCK_TIMED
    • RESP_ACCT_LOCK
    • RESP_ACCT_REMOTE
    • RESP_ACCT_UNLOCK_TIMED
    • RESP_ALERT
    • RESP_ANOMALY
    • RESP_EXEC
    • RESP_HALT
    • RESP_KILL_PROC
    • RESP_SEBOOL
    • RESP_SINGLE
    • RESP_TERM_ACCESS
    • RESP_TERM_LOCK
• Miscellaneous:
  • ALL: Matches all types.
  • KERNEL_OTHER: record information from third-party kernel modules.
  • EOE: end of a multi-record event.
  • TEST: success value of a test message.
  • TRUSTED_APP: The record of this type can be used by third party application that require auditing.
  • TTY: TTY input was sent to an administrative process.
  • USER_TTY: explanatory message about TTY input to an administrative process is sent from user-space.
  • USER
  • USYS_CONFIG: user-space system configuration change is detected.
  • TIME_ADJNTPVAL: system clock is modified.
  • TIME_INJOFFSET: Timekeeping offset is injected to the system clock..