LFCS Phase 1 Part 18: Understanding /usr Directory - The Heart of Your Linux System

When you run commands like ls, grep, or python, where do these programs actually live? When you install software with apt or dnf, where does it go? The answer: /usr. This massive directory is the heart of your Linux system, containing the vast majority of your installed programs, libraries, and documentation. In this comprehensive guide, we'll explore /usr in detail and understand why it's structured the way it is.

What is /usr?

The /usr directory stands for "Unix System Resources" (though historically it meant "user").

The Heart of Your System

Think of /usr as the warehouse where all your software lives:

Why /usr Exists

Historically: Early Unix systems had limited disk space. The root filesystem (/) was kept small, and larger programs were put on a separate mounted filesystem (/usr).

Modern usage: While we no longer have these space constraints, the structure remains because it provides:

• Clear organization
• Separation of essential vs non-essential programs
• Easy read-only mounting for security
• Network sharing capabilities

Exploring /usr Structure

Let's see what's inside /usr:

ls -l /usr

Typical output:

drwxr-xr-x.  2 root root  49152 Nov  5 18:14 bin
drwxr-xr-x.  2 root root    172 Jun 25  2024 etc
drwxr-xr-x.  2 root root      6 Jun 25  2024 games
drwxr-xr-x. 48 root root   4096 Oct 28 15:13 include
drwxr-xr-x. 71 root root   4096 Oct 28 15:13 lib
drwxr-xr-x.202 root root  69632 Oct 28 15:13 lib64
drwxr-xr-x. 27 root root   4096 Oct 28 15:06 libexec
drwxr-xr-x. 14 root root    142 Jun 25  2024 local
drwxr-xr-x.  2 root root  20480 Oct 28 15:13 sbin
drwxr-xr-x.293 root root  12288 Oct 28 15:13 share
drwxr-xr-x.  6 root root     57 Jun 25  2024 src
drwxr-xr-x.  3 root root     20 Oct 28 15:06 tmp

Let's explore each of these in detail.

/usr/bin - User Commands and Applications

/usr/bin is where most of your command-line programs live.

What's Inside

ls /usr/bin | head -20

Examples you'll find:

• Text editors: vim, nano, gedit
• Programming languages: python3, perl, ruby, gcc
• File utilities: find, grep, awk, sed
• Network tools: curl, wget, ssh, scp
• Compression: tar, gzip, bzip2, zip
• Package managers: dnf, apt, yum

Count Your Commands

ls /usr/bin | wc -l

Typical result: 1,000 to 2,000+ programs!

Finding Where Commands Live

Use which to find a command's location:

which ls
# Output: /usr/bin/ls

which python3
# Output: /usr/bin/python3

which grep
# Output: /usr/bin/grep

The /bin Symlink

On modern Linux distributions:

ls -l /bin

Output:

lrwxrwxrwx. 1 root root 7 Jun 25  2024 /bin -> usr/bin

What this means: /bin and /usr/bin are the same location! The system has consolidated binaries into /usr/bin for simplicity.

/usr/sbin - System Administration Binaries

/usr/sbin contains programs used for system administration, typically requiring root privileges.

What's Inside

ls /usr/sbin | head -20

Examples:

• Disk management: fdisk, mkfs, parted
• Network configuration: ip, ifconfig, route
• User management: useradd, usermod, userdel
• Service management: systemctl, service
• Package management: dnf, rpm, apt-get
• Firewall: iptables, firewalld

System vs User Binaries

The /sbin Symlink

Just like /bin, on modern systems:

ls -l /sbin

Output:

lrwxrwxrwx. 1 root root 8 Jun 25  2024 /sbin -> usr/sbin

Again, /sbin and /usr/sbin are the same location!

/usr/lib and /usr/lib64 - Shared Libraries

Libraries are collections of precompiled code that programs can use. Think of them as "code libraries" that programs "check out" when they run.

What Are Shared Libraries?

Analogy: If programs are recipes, libraries are common ingredients:

• Instead of every recipe including "how to make tomato sauce from scratch"
• Recipes just say "add tomato sauce" (use the library)
• One library serves many programs (saves space and effort!)

/usr/lib vs /usr/lib64

ls -ld /usr/lib /usr/lib64

Example Libraries

ls /usr/lib64/lib*.so.* | head -10

Common libraries:

• libc.so.6 - C standard library (fundamental!)
• libm.so.6 - Math library
• libssl.so - OpenSSL library
• libpython3.so - Python library
• libcurl.so - URL transfer library

Checking Library Dependencies

See what libraries a program needs:

ldd /usr/bin/ls

Example output:

linux-vdso.so.1 (0x00007ffd9c3f5000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8e5c2a0000)
libc.so.6 => /lib64/libc.so.6 (0x00007f8e5c000000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f8e5bf60000)
/lib64/ld-linux-x86-64.so.2 (0x00007f8e5c300000)

This shows ls uses the SELinux library, C library, PCRE library, and the dynamic linker.

/usr/share - Architecture-Independent Data

/usr/share contains files that don't depend on your CPU architecture—they're the same on 32-bit, 64-bit, ARM, etc.

What's Inside

ls /usr/share | head -20

Major subdirectories:

Man Pages Location

Remember from Post 11 when we used man commands? They live here:

ls /usr/share/man

Output:

man1  man2  man3  man4  man5  man6  man7  man8  man9

Each manN directory contains man pages for that section!

Documentation We've Used

ls /usr/share/doc | head -10

This is the same directory we explored in Post 14 for package documentation!

/usr/local - Locally Installed Software

/usr/local is a special hierarchy for software you install manually (not through package managers).

Purpose

Structure

/usr/local mirrors /usr:

ls -l /usr/local

You'll find:

• /usr/local/bin - Your manually installed programs
• /usr/local/sbin - Your manually installed admin tools
• /usr/local/lib - Libraries for your local software
• /usr/local/share - Data for your local software
• /usr/local/include - Development headers
• /usr/local/src - Source code you've downloaded

When to Use /usr/local

Use /usr/local when:

• Compiling software from source
• Installing software not available in repositories
• Testing newer versions than in official repos
• Installing custom scripts system-wide

Example workflow:

# Download source code
cd /tmp
wget https://example.com/software-1.0.tar.gz
tar xzf software-1.0.tar.gz
cd software-1.0

# Compile and install to /usr/local
./configure --prefix=/usr/local
make
sudo make install

# Program now in /usr/local/bin
which software
# Output: /usr/local/bin/software

/usr/include - Development Headers

If you're compiling software, you'll encounter /usr/include.

What Are Header Files?

Header files (.h files) contain declarations for C/C++ programming:

ls /usr/include | head -20

Examples:

• stdio.h - Standard input/output
• stdlib.h - Standard library
• string.h - String manipulation
• math.h - Mathematical functions

When You Need These

You need header files when:

• Compiling C/C++ programs
• Building software from source
• Developing applications

/usr/libexec - Internal Binaries

/usr/libexec contains programs meant to be called by other programs, not directly by users.

ls /usr/libexec | head -10

Examples:

• Helper programs for applications
• Backend daemons
• Plugin executables

Key characteristic: You typically don't run these directly; they're called by other software.

Why is /usr Read-Only?

In many production systems, /usr is mounted read-only for security and stability.

Benefits of Read-Only /usr

Checking if /usr is Read-Only

mount | grep '/usr'

If read-only, you'll see:

/dev/sda2 on /usr type ext4 (ro,relatime)
#                                ^^
#                                read-only flag

Writable (most desktop systems):

/dev/sda2 on /usr type ext4 (rw,relatime)
#                                ^^
#                                read-write flag

How Package Managers Use /usr

When you install software with dnf or apt, here's what happens:

Installation Process

Tracking Installed Files

See where a package installed its files:

rpm -ql vim-enhanced | head -20

dpkg -L vim | head -20

Example output:

/usr/bin/vim
/usr/share/applications/vim.desktop
/usr/share/doc/vim-enhanced
/usr/share/man/man1/vim.1.gz
/usr/share/vim

The $PATH Variable and /usr

When you type a command, how does Linux find it? Through the $PATH environment variable!

Viewing Your PATH

echo $PATH

Typical output:

/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/username/.local/bin

Path Search Order

Linux searches directories in order (left to right):

1. /usr/local/bin - Check for local installations first
2. /usr/bin - Then system binaries
3. /usr/local/sbin - Then local admin tools
4. /usr/sbin - Then system admin tools
5. ~/.local/bin - Finally user's personal binaries

/usr Directory Size

Let's see how big /usr typically is:

du -sh /usr

Typical sizes:

• Minimal server: 2-5 GB
• Desktop system: 8-15 GB
• Workstation with development tools: 15-30+ GB

Check subdirectory sizes:

sudo du -sh /usr/* | sort -rh | head -10

Typically largest:

• /usr/lib64 or /usr/lib - Libraries
• /usr/share - Documentation, icons, data
• /usr/bin - Thousands of programs

🧪 Practice Labs

Time to explore /usr hands-on! Remember: only solutions are collapsible.

Warm-Up Labs (Beginner)

Lab 1: Explore /usr Structure (Beginner)

Task: Navigate through /usr and identify all major subdirectories.

Steps:

1. Navigate to /usr
2. List all subdirectories with details
3. Identify the largest directories
4. Count how many items are in /usr

# Navigate to /usr
cd /usr

# List with details
ls -lh

# Check sizes of subdirectories
du -sh *

# Sort by size
du -sh * | sort -rh

# Count items
ls | wc -l

# Get total size
du -sh /usr

Lab 2: Count Your Commands (Beginner)

Task: Discover how many commands are available in /usr/bin.

Steps:

1. Count files in /usr/bin
2. Find a specific command (like python3)
3. Verify /bin links to /usr/bin
4. Understand the consolidation

# Count commands in /usr/bin
ls /usr/bin | wc -l

# Find Python
ls -l /usr/bin/python*

# Check if /bin is a symlink
ls -l /bin

# Verify they're the same
readlink /bin
# Output: usr/bin

# List the same file both ways
ls -l /bin/ls
ls -l /usr/bin/ls
# Same file!

Lab 3: Find Where Commands Live (Beginner)

Task: Use which and type to locate various commands.

Steps:

1. Find where ls lives
2. Find where vim lives
3. Find where fdisk lives (system admin tool)
4. Understand the pattern

# Find user commands
which ls
# Output: /usr/bin/ls

which grep
# Output: /usr/bin/grep

which python3
# Output: /usr/bin/python3

# Find system admin commands
which fdisk
# Output: /usr/sbin/fdisk

which useradd
# Output: /usr/sbin/useradd

# Use type for more info
type ls
# Output: ls is /usr/bin/ls

type cd
# Output: cd is a shell builtin

Lab 4: Explore Man Pages Location (Beginner)

Task: Understand where man pages are stored.

Steps:

1. Navigate to /usr/share/man
2. List the sections
3. Find a specific man page file
4. Connect this to Post 11 knowledge

# Navigate to man pages
cd /usr/share/man

# List sections
ls
# Output: man1 man2 man3 ... man9

# See how many man pages in section 1
ls man1 | wc -l

# Find ls man page
ls -l man1/ls.1.gz

# View it
man ls
# This reads from /usr/share/man/man1/ls.1.gz!

# Count total man pages
find /usr/share/man -name "*.gz" | wc -l

Lab 5: Check Library Dependencies (Beginner)

Task: Use ldd to see what libraries a program needs.

Steps:

1. Check ls dependencies
2. Check vim dependencies
3. Identify common libraries
4. Understand why libraries matter

# Check ls dependencies
ldd /usr/bin/ls

# Check vim dependencies
ldd /usr/bin/vim

# Check Python dependencies
ldd /usr/bin/python3

# Find a specific library file
ls -lh /usr/lib64/libc.so.6

# All programs use libc!
# It's the fundamental C library

# Check library path
ldconfig -p | grep libc.so.6

Core Practice Labs (Intermediate)

Lab 6: Explore /usr/share/doc (Intermediate)

Task: Revisit /usr/share/doc from Post 14 with new context.

Steps:

1. Navigate to /usr/share/doc
2. Pick a package and explore its documentation
3. Find README files
4. Connect this knowledge to Post 14

# Navigate to documentation
cd /usr/share/doc

# List packages with docs
ls | head -20

# Explore a package (e.g., sudo)
cd sudo
ls

# Read the README
cat README.md

# Or with less
less README.md

# Find all README files in /usr/share/doc
find /usr/share/doc -name "README*" | head -20

# Check size of documentation
du -sh /usr/share/doc

Lab 7: Investigate /usr/local (Intermediate)

Task: Understand the structure and purpose of /usr/local.

Steps:

1. Explore /usr/local structure
2. Check if you have any local software
3. Compare to /usr structure
4. Understand when to use /usr/local

# Explore /usr/local
ls -l /usr/local

# Check for locally installed programs
ls /usr/local/bin

# Check your PATH includes /usr/local
echo $PATH | grep "/usr/local"

# See if /usr/local comes before /usr
echo $PATH
# Should see /usr/local/bin before /usr/bin

# Create a test script in /usr/local/bin
sudo nano /usr/local/bin/mytest
# Add: #!/bin/bash
#      echo "Local script!"

# Make executable
sudo chmod +x /usr/local/bin/mytest

# Run it
mytest
# Works because /usr/local/bin is in PATH!

# Clean up
sudo rm /usr/local/bin/mytest

Lab 8: Track Package Files (Intermediate)

Task: See exactly where a package installed its files.

Steps:

1. Pick a package you have installed
2. List all its files
3. Identify patterns in file locations
4. Understand package organization

# For RPM-based (CentOS, RHEL, Fedora)
rpm -ql vim-enhanced | head -30

# For Debian-based (Ubuntu, Debian)
dpkg -L vim | head -30

# See the pattern:
# - Binaries in /usr/bin
# - Libraries in /usr/lib64
# - Docs in /usr/share/doc
# - Man pages in /usr/share/man

# Check another package
rpm -ql bash | grep "^/usr" | head -20

# Count files
rpm -ql bash | wc -l

# Find config files (usually in /etc)
rpm -ql bash | grep "^/etc"

Lab 9: Compare /usr/bin vs /usr/sbin (Intermediate)

Task: Understand the difference between user and system binaries.

Steps:

1. List commands in /usr/bin
2. List commands in /usr/sbin
3. Try running sbin commands without sudo
4. Understand when sudo is needed

# Count user commands
ls /usr/bin | wc -l

# Count system commands
ls /usr/sbin | wc -l

# Find disk-related commands in sbin
ls /usr/sbin | grep -E 'fdisk|mkfs|parted'

# Try fdisk without sudo (will complain)
fdisk -l
# Error: must be superuser

# Now with sudo
sudo fdisk -l
# Works!

# Check your PATH
echo $PATH
# Notice /usr/bin comes before /usr/sbin

# See if regular user can even find sbin commands
which fdisk
# Output: /usr/sbin/fdisk

# But try a user command
which ls
# Output: /usr/bin/ls

Lab 10: Explore Symbolic Links in /usr (Intermediate)

Task: Find and understand symlinks within /usr.

Steps:

1. Find symlinks in /usr/bin
2. See where they point
3. Understand why symlinks exist
4. Check Python symlinks as example

# Find symlinks in /usr/bin
find /usr/bin -type l | head -20

# Check specific symlinks
ls -l /usr/bin/python*

# Example output:
# python3 -> python3.11
# python3.11 (actual file)

# This means typing 'python3' runs 'python3.11'

# Check where they point
readlink /usr/bin/python3

# Find all symlinks and their targets
find /usr/bin -type l -ls | head -10

# Count symlinks vs regular files
find /usr/bin -type l | wc -l   # Symlinks
find /usr/bin -type f | wc -l   # Regular files

Lab 11: Check /usr Disk Usage (Intermediate)

Task: Analyze disk space usage in /usr.

Steps:

1. Check total /usr size
2. Find largest subdirectories
3. Identify what uses the most space
4. Understand growth patterns

# Check total size
du -sh /usr

# Check subdirectories
sudo du -sh /usr/* | sort -rh | head -10

# Typically largest:
# /usr/lib64 - libraries
# /usr/share - documentation and data
# /usr/bin - thousands of programs

# Check specific large directories
du -sh /usr/share/doc
du -sh /usr/share/icons
du -sh /usr/share/locale

# Find largest files in /usr
sudo find /usr -type f -size +50M -exec ls -lh {} \; | head -10

# Compare to other directories
du -sh /usr /var /home /opt 2>/dev/null

Lab 12: Understand Read-Only /usr Concept (Intermediate)

Task: Check if /usr is mounted read-only and understand implications.

Steps:

1. Check current mount status of /usr
2. Understand read-only vs read-write
3. Try to create a file (will likely fail or succeed depending on setup)
4. Understand why read-only is used

# Check how /usr is mounted
mount | grep '/usr'

# Check for 'ro' (read-only) or 'rw' (read-write)
# Example read-write: /dev/sda2 on /usr type ext4 (rw,relatime)
# Example read-only: /dev/sda2 on /usr type ext4 (ro,relatime)

# Try to create a file in /usr/bin (will likely fail)
sudo touch /usr/bin/testfile
# If /usr is rw: succeeds
# If /usr is ro: fails with "Read-only file system"

# Clean up if successful
sudo rm /usr/bin/testfile

# Check mount options more clearly
findmnt /usr

# See all filesystems and their options
findmnt -l

Lab 13: Explore Include Headers (Intermediate)

Task: Understand development header files in /usr/include.

Steps:

1. Navigate to /usr/include
2. View a standard header file
3. Understand when these are needed
4. Check if you have development packages

# Navigate to include directory
cd /usr/include

# List headers
ls | head -20

# View a standard header (stdio.h)
cat stdio.h | head -30

# See what's defined
grep "^#define" stdio.h | head -10

# Check if development tools installed
which gcc
# If found: development tools present
# If not found: need to install build tools

# Check for Python development headers
ls -l /usr/include/python*

# Find all header files
find /usr/include -name "*.h" | wc -l

Challenge Labs (Advanced)

Lab 14: Create Custom /usr/local Software (Advanced)

Task: Simulate installing custom software to /usr/local.

Steps:

1. Create a simple script
2. Install it to /usr/local/bin
3. Make it executable
4. Test that it works system-wide

# Create a custom command
sudo nano /usr/local/bin/sysinfo

# Add this content:
#!/bin/bash
echo "=== System Information ==="
echo "Hostname: $(hostname)"
echo "Kernel: $(uname -r)"
echo "Uptime: $(uptime -p)"
echo "Memory: $(free -h | grep Mem | awk '{print $3 "/" $2}')"
echo "Disk: $(df -h / | tail -1 | awk '{print $3 "/" $2 " (" $5 ")"}')"

# Save and exit

# Make executable
sudo chmod +x /usr/local/bin/sysinfo

# Test it
sysinfo

# Works! Because /usr/local/bin is in PATH

# Check where it is
which sysinfo
# Output: /usr/local/bin/sysinfo

# Remove when done
sudo rm /usr/local/bin/sysinfo

Lab 15: Analyze Library Dependencies Tree (Advanced)

Task: Deep dive into library dependencies and understand the dependency chain.

Steps:

1. Pick a complex program
2. View its dependencies
3. Check dependencies of dependencies
4. Understand the dependency tree

# Check a complex program like Python
ldd /usr/bin/python3

# Save to file for analysis
ldd /usr/bin/python3 > /tmp/python-deps.txt
cat /tmp/python-deps.txt

# Check a library's dependencies
ldd /usr/lib64/libpython3.11.so.1.0 | head -20

# Find where a library is
ldconfig -p | grep libssl

# Check what depends on a specific library
# Find all binaries that use libssl
for bin in /usr/bin/*; do
    ldd "$bin" 2>/dev/null | grep -q libssl && echo "$bin uses libssl"
done | head -10

# Count dependencies
ldd /usr/bin/python3 | wc -l

# Compare to simpler program
ldd /usr/bin/ls | wc -l

Lab 16: Map Your Entire /usr Structure (Advanced)

Task: Create a complete map of your /usr directory.

Steps:

1. Use tree to visualize /usr
2. Document major sections
3. Calculate sizes
4. Create a reference document

# Install tree if needed
sudo dnf install tree   # CentOS/RHEL
sudo apt install tree   # Ubuntu/Debian

# Create 2-level tree
tree -L 2 -d /usr > ~/usr-structure.txt

# View it
cat ~/usr-structure.txt

# Create detailed report
cat > ~/usr-analysis.txt << 'EOF'
/usr Directory Analysis
=======================

EOF

# Add sizes
echo "Directory Sizes:" >> ~/usr-analysis.txt
sudo du -sh /usr/* | sort -rh >> ~/usr-analysis.txt

echo -e "\nBinary Counts:" >> ~/usr-analysis.txt
echo "User commands: $(ls /usr/bin | wc -l)" >> ~/usr-analysis.txt
echo "System commands: $(ls /usr/sbin | wc -l)" >> ~/usr-analysis.txt

echo -e "\nLibrary Info:" >> ~/usr-analysis.txt
echo "Libraries: $(find /usr/lib64 -name "*.so*" 2>/dev/null | wc -l)" >> ~/usr-analysis.txt

echo -e "\nDocumentation:" >> ~/usr-analysis.txt
echo "Man pages: $(find /usr/share/man -name "*.gz" 2>/dev/null | wc -l)" >> ~/usr-analysis.txt
echo "Doc packages: $(ls /usr/share/doc | wc -l)" >> ~/usr-analysis.txt

cat ~/usr-analysis.txt

Lab 17: Simulate Package Installation Paths (Advanced)

Task: Understand exactly where package managers install different file types.

Steps:

1. Pick a package
2. Categorize its files by type
3. See the installation pattern
4. Create a summary

# Pick a package (example: vim)
PACKAGE="vim-enhanced"  # CentOS/RHEL
# or PACKAGE="vim" for Ubuntu/Debian

# List all files
rpm -ql $PACKAGE > /tmp/package-files.txt
# or: dpkg -L vim > /tmp/package-files.txt

# Categorize files
echo "=== Binaries ==="
grep "^/usr/bin/" /tmp/package-files.txt

echo -e "\n=== Libraries ==="
grep "^/usr/lib" /tmp/package-files.txt | head -5

echo -e "\n=== Man Pages ==="
grep "^/usr/share/man/" /tmp/package-files.txt

echo -e "\n=== Documentation ==="
grep "^/usr/share/doc/" /tmp/package-files.txt

echo -e "\n=== Application Data ==="
grep "^/usr/share/vim" /tmp/package-files.txt | head -10

# Count by directory
echo -e "\n=== File Distribution ==="
cut -d/ -f1-3 /tmp/package-files.txt | sort | uniq -c | sort -rn

# Clean up
rm /tmp/package-files.txt

Lab 18: Find Orphaned Files in /usr/local (Advanced)

Task: Check for manually installed software and document it.

Steps:

1. Search /usr/local for installed software
2. Identify what's there
3. Create an inventory
4. Understand why tracking is important

# Find all files in /usr/local
find /usr/local -type f > /tmp/local-files.txt

# Count files
wc -l /tmp/local-files.txt

# Check binaries
echo "=== Locally Installed Binaries ==="
ls -lh /usr/local/bin

# Check libraries
echo -e "\n=== Local Libraries ==="
find /usr/local/lib -name "*.so*" 2>/dev/null

# Create inventory
cat > ~/usr-local-inventory.txt << 'EOF'
/usr/local Software Inventory
==============================

EOF

echo "Binaries:" >> ~/usr-local-inventory.txt
ls /usr/local/bin 2>/dev/null >> ~/usr-local-inventory.txt || echo "None" >> ~/usr-local-inventory.txt

echo -e "\nLibraries:" >> ~/usr-local-inventory.txt
find /usr/local/lib -name "*.so*" 2>/dev/null >> ~/usr-local-inventory.txt || echo "None" >> ~/usr-local-inventory.txt

echo -e "\nSource Code:" >> ~/usr-local-inventory.txt
ls /usr/local/src 2>/dev/null >> ~/usr-local-inventory.txt || echo "None" >> ~/usr-local-inventory.txt

cat ~/usr-local-inventory.txt

# Clean up
rm /tmp/local-files.txt

Lab 19: Understand Shared Library Loading (Advanced)

Task: Learn how Linux finds and loads shared libraries.

Steps:

1. Check library search paths
2. Understand LD_LIBRARY_PATH
3. View library cache
4. Debug library issues

# View library search paths
ldconfig -v 2>/dev/null | head -30

# Check library cache
ldconfig -p | head -20

# Find specific library
ldconfig -p | grep libssl

# Check configured library paths
cat /etc/ld.so.conf

# See included configs
ls /etc/ld.so.conf.d/

# Check if program can find its libraries
ldd /usr/bin/ls

# Simulate library path issue
# (Just demonstration, don't actually break things)
echo "Current LD_LIBRARY_PATH: $LD_LIBRARY_PATH"

# Where libraries are searched (in order):
# 1. LD_LIBRARY_PATH (if set)
# 2. /etc/ld.so.cache (generated by ldconfig)
# 3. /lib64 and /usr/lib64
# 4. /lib and /usr/lib

# Rebuild library cache (requires sudo)
sudo ldconfig

# Verify
ldconfig -p | wc -l
echo "Total cached libraries"

Lab 20: Create Complete /usr Reference Guide (Advanced)

Task: Build a comprehensive cheat sheet for the /usr directory.

Steps:

1. Document all major subdirectories
2. Add examples and use cases
3. Include commands for exploration
4. Create a permanent reference

cat > ~/usr-directory-guide.txt << 'EOF'
====================================================
/usr DIRECTORY - COMPLETE REFERENCE GUIDE
====================================================

STRUCTURE
/usr             Unix System Resources

KEY SUBDIRECTORIES
/usr/bin         User commands and applications (1000+ programs)
                 Examples: ls, grep, vim, python3, gcc
                 Command: ls /usr/bin | wc -l

/usr/sbin        System administration binaries
                 Examples: fdisk, useradd, systemctl, ifconfig
                 Requires: Usually need sudo to run
                 Command: ls /usr/sbin | wc -l

/usr/lib         32-bit libraries (on multilib systems)
/usr/lib64       64-bit libraries (primary on 64-bit systems)
                 Contains: Shared code libraries (.so files)
                 Check deps: ldd /usr/bin/ls
                 Command: ls /usr/lib64 | grep "^lib.*\.so" | head

/usr/share       Architecture-independent shared data
  /usr/share/man       Manual pages (man command)
  /usr/share/doc       Package documentation
  /usr/share/icons     Desktop icons
  /usr/share/fonts     System fonts
  /usr/share/locale    Translations
  Command: du -sh /usr/share/*

/usr/local       Locally installed software (not from package manager)
  /usr/local/bin       Local binaries
  /usr/local/lib       Local libraries
  /usr/local/share     Local data
  Use for: Software compiled from source
  Command: ls /usr/local/bin

/usr/include     C/C++ header files for development
                 Needed for: Compiling programs
                 Command: ls /usr/include/*.h | head

/usr/libexec     Internal helper programs (called by other programs)
                 Not directly run by users

/usr/src         Kernel source code (if installed)

SYMLINKS (Modern Systems)
/bin -> usr/bin
/sbin -> usr/sbin
/lib -> usr/lib
/lib64 -> usr/lib64
Reason: Consolidation for simplicity

CHARACTERISTICS
- Typically read-only or read-mostly
- Largest directory on system (5-30+ GB)
- Managed by package manager
- Shareable across network
- Critical for system operation

COMMON TASKS
Count commands:    ls /usr/bin | wc -l
Find program:      which vim
Check libs:        ldd /usr/bin/python3
Read man pages:    man ls (from /usr/share/man)
Package docs:      ls /usr/share/doc
Check size:        du -sh /usr
Library cache:     ldconfig -p
Install to local:  ./configure --prefix=/usr/local

PACKAGE MANAGER BEHAVIOR
When you run: sudo dnf install vim
Files go to:
- Binary:     /usr/bin/vim
- Libraries:  /usr/lib64/libvim*.so
- Man pages:  /usr/share/man/man1/vim.1.gz
- Docs:       /usr/share/doc/vim
- Data:       /usr/share/vim

BEST PRACTICES
✓ Never manually edit files in /usr (except /usr/local)
✓ Use package manager for software installation
✓ Use /usr/local for custom installations
✓ Keep /usr read-only in production
✓ Document anything in /usr/local
✓ Regularly check /usr disk usage

PATH SEARCH ORDER
When you type a command, Linux searches:
1. /usr/local/bin   (custom software)
2. /usr/bin         (system software)
3. /usr/local/sbin  (custom admin tools)
4. /usr/sbin        (system admin tools)

====================================================
EOF

cat ~/usr-directory-guide.txt
echo -e "\nGuide saved to: ~/usr-directory-guide.txt"

📚 Best Practices

For System Administration

1. Never manually modify /usr (except /usr/local)

   • Always use package manager for /usr changes
   • Manual changes in /usr/bin or /usr/lib64 will be overwritten

2. Use /usr/local for custom software

   ./configure --prefix=/usr/local
   make
   sudo make install
   

   Copy

3. Keep /usr read-only in production

   • Enhances security
   • Prevents accidental changes

4. Document /usr/local installations

   • Package managers don't track /usr/local
   • Keep a list of what you've installed manually

For Development

1. Install development packages

   # CentOS/RHEL
   sudo dnf groupinstall "Development Tools"
   
   # Ubuntu/Debian
   sudo apt install build-essential
   

   Copy

2. Check library dependencies before compiling

   ldd /path/to/program
   

   Copy

3. Use pkg-config to find libraries

   pkg-config --libs --cflags library-name
   

   Copy

🚨 Common Pitfalls

Pitfall 1: Confusing /usr with /home

Problem: Thinking /usr is for user files.

# ❌ Wrong: Don't store personal files in /usr
# /usr is for system software, not user data

# ✅ Right: User files go in /home
~/Documents/myfile.txt

Pitfall 2: Installing Software to Wrong Location

Problem: Compiling software to /usr instead of /usr/local.

# ❌ Wrong: Conflicts with package manager
./configure --prefix=/usr

# ✅ Right: Use /usr/local
./configure --prefix=/usr/local

Pitfall 3: Assuming /bin is Separate from /usr/bin

Problem: Not realizing they're the same on modern systems.

# Check first!
readlink /bin
# Output: usr/bin (they're the same!)

Pitfall 4: Forgetting Library Dependencies

Problem: Copying a binary to another system without libraries.

# ❌ Wrong: Just copying binary
scp /usr/bin/myprogram otherserver:/usr/local/bin/

# ✅ Right: Check dependencies first
ldd /usr/bin/myprogram
# Then copy all required libraries too

📝 Command Cheat Sheet

Exploring /usr

ls -lh /usr                    # List /usr contents
du -sh /usr                    # Check /usr size
du -sh /usr/* | sort -rh       # Find largest subdirectories
find /usr -name "filename"     # Find files in /usr

Finding Commands

which command                  # Find command location
type command                   # Show command type
whereis command                # Find binary, source, and man page

Working with Libraries

ldd /usr/bin/program           # Check library dependencies
ldconfig -p                    # List library cache
ldconfig -p | grep libname     # Find specific library
sudo ldconfig                  # Rebuild library cache

Package Management

rpm -ql package                # List package files (RPM)
dpkg -L package                # List package files (Debian)
rpm -qf /usr/bin/file          # Find which package owns file

Documentation

man command                    # Read man page
ls /usr/share/man/man1         # List section 1 man pages
ls /usr/share/doc              # List documentation

🎯 Key Takeaways

🚀 What's Next?

Excellent work! You've mastered the /usr directory and understand where most of your Linux software lives. You now know:

• ✅ The structure and purpose of /usr
• ✅ Where commands, libraries, and documentation are stored
• ✅ The difference between /usr and /usr/local
• ✅ How package managers organize files
• ✅ Why /usr is typically read-only

In the next post (Part 19), we'll explore /var and /etc—two critical directories for system configuration and variable data. We'll cover logs in /var/log, configuration files in /etc, and understand why these directories are so important for system administration.

Coming up:

• Post 19: Understanding /var and /etc Explained
• Post 20: Write Permissions and Access Control
• Post 21: Using Wildcards for File Management

Series Navigation:

• ← Previous: Part 17 - Understanding Linux Filesystem Hierarchy Overview
• → Next: Part 19 - Understanding /var and /etc Explained (Coming Next!)

Part of the LFCS Certification Preparation Series - Phase 1 of 9