Study Note: Unix-Like Commands

Utility

Archiving, Splitting, and Encrypting Large Files for DVD Storage

When archiving large folders for storage across multiple DVDs, encryption can be added to enhance security. The following details the steps for archiving, splitting, and encrypting, with a focus on flexible and secure methods.

(A) General Archiving and Splitting Approach

To efficiently manage large folders for single-layer DVD storage (approx. 4.5GB), the tar and split commands can be combined:

tar -zcvf - folder_name | split -b 4500M - archive_name.tar.gz.

• tar -zcvf - folder_name: Compresses the specified folder into a .tar.gz archive, outputting it to standard output.
• split -b 4500M - archive_name.tar.gz.: Reads the output from the tar command and splits the archive into 4.5GB chunks. The files are named sequentially, e.g., archive_name.tar.gz.aa, archive_name.tar.gz.ab, etc.

To restore the split archive, the following steps can be used to concatenate the parts and extract the archive:

cat archive_name.tar.gz.* > full_archive.tar.gz
tar -zxvf full_archive.tar.gz

Alternatively, both steps may be combined into one:

cat archive_name.tar.gz.* | tar -zxvf -

type archive_name.tar.gz.* | tar -zxvf - (Windows)        

(B) Advanced Splitting for Specific DVD Sizes

The split command allows for adjustments based on DVD size:

• For Single-Layer DVDs (4.7GB):

FOLDER="./folder_name" && tar -zcvf - "$FOLDER" | split -b 4480M - "${FOLDER}.tar.gz."

• For Dual-Layer DVDs (8.5GB):

FOLDER="./folder_name" && tar -zcvf - "$FOLDER" | split -b 8150M - "${FOLDER}.tar.gz."

In each case, the archive is split into chunks that can fit onto DVDs, with the files being named sequentially.

(C) Encrypting the Archive

Encryption can be added using external tools like GPG or OpenSSL since neither tar nor split directly support password protection.

C-1) Using GPG for Encryption

GPG provides AES-256 symmetric encryption for securing the tarball with a password.

Encrypting and Splitting the Archive:

tar -zcvf - folder_name | gpg --symmetric --cipher-algo AES256 -o archive_name.tar.gz.gpg
split -b 4500M archive_name.tar.gz.gpg archive_name_split.

• gpg --symmetric --cipher-algo AES256: Encrypts the tarball using AES-256 encryption.
• split -b 4500M archive_name.tar.gz.gpg archive_name_split.: Splits the encrypted tarball into 4.5GB parts.

Restoring the Archive:

cat archive_name_split.* > full_archive.tar.gz.gpg
gpg -o full_archive.tar.gz -d full_archive.tar.gz.gpg
tar -zxvf full_archive.tar.gz

This restores the archive by combining the parts, decrypting, and extracting the tarball.

C-2) Using OpenSSL for Encryption

OpenSSL is another option for adding password-based encryption to the tar archive.

Encrypting and Splitting the Archive:

tar -zcvf - folder_name | openssl enc -aes-256-cbc -e -k 'password' -out archive_name.tar.gz.enc
split -b 4500M archive_name.tar.gz.enc archive_name_split.

• openssl enc -aes-256-cbc -e -k 'password': Encrypts the tarball using AES-256 encryption and a specified password.

Restoring the Archive:

cat archive_name_split.* > full_archive.tar.gz.enc
openssl enc -aes-256-cbc -d -k 'password' -in full_archive.tar.gz.enc -out full_archive.tar.gz
tar -zxvf full_archive.tar.gz

This process ensures that large folders can be efficiently archived, split, and encrypted for secure storage across multiple DVDs.

Image Processing with sips (Scriptable Image Processing System)

(A) Converting PNG to JPEG/JPG

To convert a PNG file to a JPEG file, the following command can be used:

sips -s format jpeg input.png --out output.jpg

• sips: The command-line utility for image processing on macOS.
• -s format jpeg: Sets the output format to JPEG.
• input.png: Refers to the input PNG file.
• --out output.jpg: Specifies the name of the output file, which will be in JPEG format.

Note: The extensions .jpg and .jpeg are interchangeable. The sips command processes both formats the same way.

(B) Converting JPEG/JPG to PNG

To convert a JPEG (or JPG) file to a PNG file, the following command is used:

sips -s format png input.jpg --out output.png

• -s format png: Sets the output format to PNG.
• input.jpg: Refers to the input JPEG or JPG file.
• --out output.png: Specifies the name of the output file, which will be in PNG format.

Resize JPGs on macOS (sips & ImageMagick)

On macOS, JPG files in the current directory can be reduced from approximately 3 MB to around 250 KB either by using the built-in sips tool or by employing ImageMagick if it is installed via Homebrew. Two approaches are outlined below:

I. Using sips (built-in macOS tool)

The sips command allows adjustment of image quality and compression. Example:

mkdir resized
for f in *.jpg; do
    sips -s formatOptions 50 "$f" --out "resized/$f"
done

Explanation:

• -s formatOptions 50 sets JPEG quality on a scale from 0 to 100.
• Lower values yield smaller file sizes. A value between 40 and 60 may help approach the target of 250 KB.
• Output files are stored in the resized folder.

The sips utility does not provide a direct method to specify the target file size. Adjustments to formatOptions are necessary until the desired average file size is achieved.

II. Using ImageMagick (greater control)

ImageMagick offers more precise control of file size. It may be installed via Homebrew as follows:

brew install imagemagick

Once installed, the following command sequence can be applied:

mkdir resized
for f in *.jpg; do
    magick "$f" -define jpeg:extent=250kb "resized/$f"
done

Explanation:

• -define jpeg:extent=250kb compresses the output automatically to fit under approximately 250 KB.
• This method simplifies the process by directly targeting the desired file size.
• Optimized images are produced while maintaining reasonable visual quality.

Written on September 1, 2025

Convert All HEIC Files to PNG on macOS (sips) (Written October 10, 2025)

macOS includes a built-in command-line utility called sips (Scriptable Image Processing System), which can convert HEIC images to PNG in batch mode. The following example scans the current directory for all files with the extensions .heic or .HEIC and converts them to PNG format with the same base filenames:

for f in *.(heic|HEIC); do                                 
  base="${f%.*}"
  sips -s format png "$f" --out "${base}.png"
done

Explanation:

• for f in *.(heic|HEIC); do ... done: Iterates through all files ending with .heic or .HEIC in the current directory.
• base="${f%.*}": Removes the file extension from the filename and stores the base name in the variable base.
• sips -s format png "$f" --out "${base}.png": Converts the original HEIC file into a PNG file with the same base name.

Example:
If a file named photo1.HEIC exists, this script will produce photo1.png in the same directory.

Tip: If you want to place all converted files into a separate folder (e.g., converted), create the directory first and adjust the output path like so:

mkdir converted
for f in *.(heic|HEIC); do
  base="${f%.*}"
  sips -s format png "$f" --out "converted/${base}.png"
done

This approach keeps the original HEIC files intact while storing the converted PNG images in a dedicated folder.

Written on October 10, 2025

Making passport-size photos on macOS without size or ratio distortion (Written October 12, 2025)

I. Purpose and scope

This document provides a systematic, reproducible method to prepare and print passport-size photos on macOS while preserving the original aspect ratio and physical dimensions. The guidance covers two reliable workflows: a professional 300 dpi flow and a macOS-friendly 144 dpi flow that avoids unintended enlargement during printing. Practical shell commands and a complete, ready-to-run script are included.

II. Prerequisites

III. Dimensions, dpi, and pixel math

Most ID photos are 3.5 cm × 4.5 cm. The correct pixel size depends on the dpi used for printing. When macOS or printer drivers ignore the intended dpi or default to a different value, physical output scales incorrectly. Correctly setting dpi metadata avoids this behavior.

The pixel sizes above are approximate due to rounding. Setting correct dpi metadata is critical so that macOS interprets the physical size correctly at print time.

IV. Workflow A (professional): 3.5 × 4.5 cm at 300 dpi

This flow yields a 413 × 531 px image tagged at 300 dpi, suitable for professional printing environments.

1. Resize the long edge to 531 px while preserving aspect ratio:

   sips -Z 531 input.jpg --out passport_300dpi.jpg

2. Set dpi metadata to 300:

   sips --setProperty dpiWidth 300 --setProperty dpiHeight 300 passport_300dpi.jpg

3. (Optional) Center-crop to exactly 413 × 531 px if needed:

   sips -c 531 413 passport_300dpi.jpg --out passport_300dpi_exact.jpg

4. Create a 6-up A4 PDF:

   montage passport_300dpi.jpg passport_300dpi.jpg passport_300dpi.jpg \
   passport_300dpi.jpg passport_300dpi.jpg passport_300dpi.jpg \
   -geometry 413x531+50+50 -tile 2x3 -background white -density 300 A4_passport_300dpi.pdf

V. Workflow B (macOS-friendly): 3.5 × 4.5 cm at 144 dpi

This flow has proven robust on macOS desktops, preventing oversized prints by aligning image pixels and dpi metadata with common macOS interpretations.

1. Resize the long edge to 255 px while preserving aspect ratio:

   sips -Z 255 input.jpg --out passport_144dpi.jpg

2. Set dpi metadata to 144:

   sips --setProperty dpiWidth 144 --setProperty dpiHeight 144 passport_144dpi.jpg

3. Create a 6-up A4 PDF at 144 dpi:

   montage passport_144dpi.jpg passport_144dpi.jpg passport_144dpi.jpg \
   passport_144dpi.jpg passport_144dpi.jpg passport_144dpi.jpg \
   -geometry 198x255+40+40 -tile 2x3 -background white -density 144 A4_passport_144dpi.pdf

When unexpected enlargement occurs on macOS, the 144 dpi flow is typically the most reliable solution.

Written on October 12, 2025

Batch rotation of HEIC images and PDF compilation on macOS command line (Written December 29, 2025)

I. Objective

A reproducible command-line workflow for (i) rotating all .HEIC files in a directory by 90° counterclockwise, then (ii) generating a single multi-page PDF named bible.pdf in alphabetical order.

II. Starting conditions and directory structure

• Working directory contains multiple files matching *.HEIC.
• Output artifacts:
  • Rotated HEIC images placed in rotated/
  • Intermediate PNG images placed in rotated/_tmp_png/
  • Final PDF created as rotated/bible.pdf

III. Command history summary

IV. Rewritten Q&A: end-to-end workflow

Rotation of all HEIC files by 90° counterclockwise

1. Question. How to rotate all *.HEIC images in the current directory by 90° counterclockwise using macOS command line?

   Answer. Use sips (built-in) for batch rotation. For safe output (no overwrite), create an output directory and write rotated copies there.

   mkdir rotated
   for f in *.HEIC; do
     sips -r -90 "$f" --out "rotated/$f"
   done

   Notes:

   • -r -90 applies 90° counterclockwise rotation.
   • Output files retain original names and are placed under rotated/.

PDF compilation after alphabetical sorting

1. Question. After rotation, how to combine all HEIC files into bible.pdf after alphabetical sorting?

   Answer. Convert HEIC to PNG in sorted order, then merge PNGs into a multi-page PDF. This avoids HEIC-to-PDF edge cases and ensures stable ordering.

   cd rotated
   mkdir _tmp_png
   for f in $(ls *.HEIC | sort); do
     sips -s format png "$f" --out "_tmp_png/${f%.HEIC}.png" >/dev/null
   done

   Notes:

   • ls *.HEIC | sort enforces alphabetical processing.
   • ${f%.HEIC}.png maps each HEIC filename to a PNG filename.
   • /dev/null suppresses conversion logs.

2. Question. How to merge the sorted PNG files into a single PDF?

   Answer. Use img2pdf for deterministic, multi-page PDF generation. If not installed, install it with Homebrew.

   cd _tmp_png
   brew install img2pdf
   img2pdf $(ls *.png | sort) -o ../bible.pdf
   cd ..

3. Question. Why did img2pdf initially fail with zsh: command not found: img2pdf?

   Answer. The utility was not installed in the environment. Installing it via brew install img2pdf resolved the missing command.

4. Question. Why did sips -s format pdf ... --out ../bible.pdf fail with an error indicating no destination directory?

   Answer. For multiple input files, sips expects --out to be a destination directory, not a single output file. sips is suitable for one-image-to-one-PDF conversions but is not suitable for reliably merging multiple images into one PDF via a single output file path.

   # Example of the failing pattern (multiple inputs -> single file via sips)
   sips -s format pdf $(ls *.png | sort) --out ../bible.pdf

V. Final recommended playbook

Minimal, reproducible command sequence

1. Rotate HEIC images into a separate directory.

   mkdir rotated
   for f in *.HEIC; do
     sips -r -90 "$f" --out "rotated/$f"
   done

2. Convert rotated HEIC to PNG in sorted order.

   cd rotated
   mkdir _tmp_png
   for f in $(ls *.HEIC | sort); do
     sips -s format png "$f" --out "_tmp_png/${f%.HEIC}.png" >/dev/null
   done

3. Install and use img2pdf to create a single multi-page PDF.

   cd _tmp_png
   brew install img2pdf
   img2pdf $(ls *.png | sort) -o ../bible.pdf
   cd ..

VI. Practical checks and housekeeping

Validation commands

1. Confirm PDF presence and size.

   ls -lh bible.pdf

2. Open the PDF for visual verification.

   open bible.pdf

Optional cleanup

1. Remove intermediate PNG files after confirming output.

   rm -rf _tmp_png

Written on December 29, 2025

Editing Videos

Trimming Videos with FFmpeg or Graphical Editors on Windows and macOS, and Resolving Homebrew PATH Issues

This guide provides a structured approach to trimming a specific section from a video using either FFmpeg or a graphical editor on Windows and macOS. It explains how to install and use FFmpeg, explores alternative editing methods, and offers troubleshooting steps to resolve PATH issues when installing FFmpeg via Homebrew on macOS. Every step and consideration is presented to ensure a smooth and professional workflow.

Trimming Videos with FFmpeg

1. Installing FFmpeg

   FFmpeg is a free, open-source tool that supports a wide range of audio and video operations. It is available on both Windows and macOS.

   Platform	Installation Steps
   Windows	Download the latest release from ffmpeg.org. Extract or install the package. (Optional) Add the FFmpeg bin folder to the system’s PATH for easier command-line usage.
   macOS	Use Homebrew if installed, by running: brew install ffmpeg Or manually download from ffmpeg.org. Ensure that the FFmpeg directory is included in the system’s PATH.

   Note: If Homebrew is used on Apple Silicon (M1/M2) Macs, binaries often reside in /opt/homebrew/bin. On Intel Macs, they often reside in /usr/local/bin.

2. Basic Command to Trim a Video

   Once FFmpeg is installed, the following command trims a segment from abc.mp4—starting at 00:00:49 and ending at 00:04:41—and saves the trimmed content into abc_edited.mp4:

   ffmpeg -i abc.mp4 -ss 00:00:49 -to 00:04:41 -c copy abc_edited.mp4

   • -i abc.mp4: Specifies the input file.
   • -ss 00:00:49: Sets the start time at 49 seconds.
   • -to 00:04:41: Sets the end time at 4 minutes, 41 seconds.
   • -c copy: Performs a “stream copy,” preserving the original quality without re-encoding.
   • abc_edited.mp4: Specifies the output filename.

3. Additional FFmpeg Usage Notes

   Feature	Stream Copy	Re-encoding
   Quality	Original (No quality loss)	May degrade slightly depending on settings
   Speed	Very fast (no compression needed)	Slower (requires processing and compression)
   Editing	Limited to cutting/trimming	Flexible (supports format conversion, resizing, etc.)
   Command Example	-c copy	-c:v libx264 -c:a aac (or other codecs)

   • Re-encoding:
     • Removing -c copy and specifying codecs (e.g., -c:v libx264 -c:a aac) will force FFmpeg to re-encode:

       ffmpeg -i abc.mp4 -ss 00:00:49 -to 00:04:41 -c:v libx264 -c:a aac abc_edited.mp4

     • Re-encoding may reduce quality and increase processing time, but it allows changing codecs or formats.
   • Keyframe Alignment:
     • Using -ss before -i can cause FFmpeg to seek to the nearest keyframe, which occasionally introduces slight timing differences. If necessary, experiment with placing -ss either before or after -i:

       ffmpeg -ss 00:00:49 -i abc.mp4 -to 00:04:41 -c copy abc_edited.mp4

     • In most scenarios, -ss after -i is sufficient with -c copy.

Trimming Videos Using Graphical Editors

Although FFmpeg is command-line based, some may prefer graphical methods. These editors typically re-encode video, which can take longer and potentially reduce quality, but they offer an intuitive visual interface.

1. Windows (Built-In Photos/Video Editor on Windows 10/11)

   1. Right-click abc.mp4 and select Open with → Photos (or Video Editor).
   2. Choose the Trim option (in Photos) or create a New video project (in Video Editor).
   3. Drag the slider handles to select content from 00:49 to 04:41.
   4. Save the trimmed section by choosing Save a copy (in Photos) or Finish video (in Video Editor).
   5. Export the result as abc_edited.mp4.

2. macOS (Using iMovie)

   1. Launch iMovie and import abc.mp4 into a new or existing project timeline.
   2. Position the playhead at 00:49 and use Command + B (or the Split Clip command) to split the clip.
   3. Repeat at 00:04:41 to isolate the desired segment.
   4. Delete content before 00:49 and after 04:41.
   5. Go to File → Share → File to export the trimmed result as abc_edited.mp4.

Troubleshooting FFmpeg PATH Issues on macOS with Homebrew

Occasionally, macOS users who install FFmpeg via Homebrew experience “command not found” errors. This typically indicates that the shell cannot locate the installed FFmpeg binary, often due to PATH misconfiguration.

1. Verifying FFmpeg Installation

   brew list ffmpeg

   or

   brew info ffmpeg

   These commands display details about the FFmpeg package. If no information appears, consider reinstalling:

   brew reinstall ffmpeg

2. Locating the FFmpeg Binary

   Homebrew generally installs software in one of the following directories:

   • Apple Silicon (M1/M2): /opt/homebrew/bin
   • Intel Macs: /usr/local/bin

   To confirm the exact location of FFmpeg, run:

   find "$(brew --prefix)" -name ffmpeg -type f

   This command returns the full path to the installed ffmpeg binary (for example, /opt/homebrew/bin/ffmpeg).

3. Checking the PATH Environment Variable

   To see if the correct installation directory is in the PATH, run:

   echo $PATH

   If /opt/homebrew/bin (Apple Silicon) or /usr/local/bin (Intel) is absent, the shell will not be able to locate FFmpeg.

4. Updating the PATH

   Apple Silicon (M1/M2) Macs
   If /opt/homebrew/bin is missing, add the following line to the shell configuration file (e.g., ~/.zshrc), then reload:

   export PATH="/opt/homebrew/bin:$PATH"
   source ~/.zshrc

   Intel Macs
   If /usr/local/bin is missing (uncommon, but possible), add the following line to the shell configuration file (e.g., ~/.zshrc or ~/.bash_profile):

   export PATH="/usr/local/bin:$PATH"
   source ~/.zshrc   # or source ~/.bash_profile if using bash

   Running ffmpeg -version afterward verifies a successful configuration.

Written on February 12, 2025

Trimming Videos on macOS with FFmpeg

FFmpeg is a powerful, open‐source multimedia framework capable of handling a wide range of video and audio operations. On macOS, it provides an efficient way to trim, concatenate, and re‐encode video clips via command‐line instructions. This guide focuses on installing and configuring FFmpeg on macOS, trimming videos (both single and multiple segments), and verifying the tool’s installation path.

Installing and Configuring FFmpeg on macOS

1. Homebrew Installation (Recommended)

   1. Install Homebrew if it is not already present. Instructions are available at https://brew.sh/.
   2. Install FFmpeg using Homebrew:

      brew install ffmpeg

   3. Default Homebrew Paths
      • Apple Silicon (M1/M2): /opt/homebrew/bin
      • Intel-based Macs: /usr/local/bin

2. Manual Download (Alternative)

   1. Download a macOS build of FFmpeg from https://ffmpeg.org/download.html.
   2. Unzip or install the downloaded package according to the official instructions.
   3. Optionally, move the ffmpeg binary to a convenient directory, such as ~/ffmpeg.

3. Adding FFmpeg to the PATH

   1. Edit the relevant shell configuration file (e.g., ~/.zshrc or ~/.bash_profile).
   2. Append one of the following lines, depending on system architecture:
      • Apple Silicon (M1/M2):

        echo 'export PATH="/opt/homebrew/bin:$PATH"' >> ~/.zshrc
        source ~/.zshrc

      • Intel Macs:

        echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bash_profile
        source ~/.bash_profile

   3. Confirm the installation and path configuration:

      ffmpeg -version

      A successful output indicates FFmpeg is correctly installed and accessible.

4. Locating the FFmpeg Installation

   In some cases, it may be necessary to confirm the exact path where FFmpeg has been installed (for example, when configuring external tools or diagnosing “command not found” errors). The following command uses Homebrew’s prefix to locate the ffmpeg binary:

   find "$(brew --prefix)" -name ffmpeg -type f

   • brew --prefix returns the base directory where Homebrew is installed. On Apple Silicon systems, this is commonly /opt/homebrew; on Intel-based Macs, /usr/local.
   • Substitution with $(...) instructs the shell to execute brew --prefix and insert that output into the find command.
   • find "$(brew --prefix)" -name ffmpeg -type f searches all subdirectories under Homebrew’s prefix for any file named ffmpeg, restricting results to regular files (-type f).
   • Outcome: Provides the precise file path to the FFmpeg binary, enabling verification or troubleshooting of installation and PATH issues.

Trimming a Single Segment with FFmpeg

Trimming one continuous portion of a video is simple using -ss (start time), -to (end time), and -c copy (stream copy). Stream copy avoids re-encoding, preserving original quality and saving time.

ffmpeg -i abc.mp4 -ss 00:00:49 -to 00:04:41 -c copy abc_edited.mp4

• -i abc.mp4: Specifies the input file.
• -ss 00:00:49: Sets the start time to 49 seconds.
• -to 00:04:41: Sets the end time to 4 minutes, 41 seconds.
• -c copy: Copies audio and video without re-encoding.
• abc_edited.mp4: The name of the output file.

Trimming Multiple Segments from One File

When multiple non-contiguous sections of a video need to be combined into a single output, there are two primary approaches:

• Concat Demuxer (two-step process, no re-encoding).
• Filter Complex (single command, re-encodes for advanced edits).

1. Approach A: Concat Demuxer (Two-Step Process)

   Step A: Extract Each Desired Segment

   Assume three segments are required from abc.mp4:

   • 2:03 to 3:12
   • 3:40 to 4:03
   • 5:02 to 5:55

   # Segment 1: 2:03–3:12
   ffmpeg -i abc.mp4 -ss 00:02:03 -to 00:03:12 -c copy part1.mp4
   
   # Segment 2: 3:40–4:03
   ffmpeg -i abc.mp4 -ss 00:03:40 -to 00:04:03 -c copy part2.mp4
   
   # Segment 3: 5:02–5:55
   ffmpeg -i abc.mp4 -ss 00:05:02 -to 00:05:55 -c copy part3.mp4

   Step B: Concatenate the Segments

   1. Create a text file (e.g., mylist.txt) with each extracted segment in order:

      file 'part1.mp4'
      file 'part2.mp4'
      file 'part3.mp4'

   2. Run FFmpeg with the concat demuxer:

      ffmpeg -f concat -safe 0 -i mylist.txt -c copy abc_edited.mp4

      • -f concat: Uses the concat demuxer.
      • -safe 0: Allows absolute or relative paths in mylist.txt.
      • -c copy: Maintains source quality by copying streams without re-encoding.

   Note: This two-step method is fast and lossless but requires creating multiple intermediate files.

2. Approach B: Filter Complex (Single Command)

   For a one-step method or when advanced processing (like overlays, resizing, or format changes) is needed, FFmpeg’s filter_complex can be used. This process involves re-encoding:

   ffmpeg -i abc.mp4 \
     -filter_complex "
       [0:v]trim=start=123:end=192,setpts=PTS-STARTPTS[v0];
       [0:a]atrim=start=123:end=192,asetpts=PTS-STARTPTS[a0];
       [0:v]trim=start=220:end=243,setpts=PTS-STARTPTS[v1];
       [0:a]atrim=start=220:end=243,asetpts=PTS-STARTPTS[a1];
       [0:v]trim=start=302:end=355,setpts=PTS-STARTPTS[v2];
       [0:a]atrim=start=302:end=355,asetpts=PTS-STARTPTS[a2];
       [v0][a0][v1][a1][v2][a2]concat=n=3:v=1:a=1[v][a]
     " \
     -map "[v]" -map "[a]" \
     -c:v libx264 -c:a aac -crf 18 -preset veryfast abc_edited.mp4

   • trim / atrim: Select specified time ranges for video/audio.
   • setpts / asetpts: Reset timestamps for seamless concatenation.
   • concat=n=3: Concatenates three segments.
   • -c:v libx264 -c:a aac: Encodes video with H.264 and audio with AAC.
   • -crf 18 -preset veryfast: Manages output quality and encoding speed.

   Note: Re-encoding can reduce quality unless CRF or bitrate settings are high, and it generally takes longer than stream copy.

3. Comparison of the Two Methods

   Criteria	Concat Demuxer	Filter Complex
   Workflow	Two-step (extract → concatenate)	Single command
   Re-encoding	No (lossless)	Yes (may affect quality unless configured carefully)
   Speed	Faster (stream copy only)	Slower (due to re-encoding)
   Flexibility	Limited to trimming and joining	Supports resizing, overlays, format changes, etc.

Additional FFmpeg Notes

• Keyframe Alignment: Placing -ss before -i can sometimes lead to frame-inaccurate trims in stream copy mode. If exact frame accuracy is critical, consider placing -ss after -i or re-encode for more precise cuts.
• Audio/Video Synchronization: Always verify final outputs to ensure audio and video stay in sync, especially when using advanced filters or re-encoding.

Written on February 21, 2025

macOS ffmpeg workflow notes (rotation and concatenation for MOV files) (Written January 1, 2026)

Purpose: Provide a compact, reliable set of commands for two recurring tasks.

Assumption: ffmpeg is installed and available in the terminal environment.

I. Rotate a MOV file 90° counterclockwise (method proven working)

Question

How to rotate a MOV video 90° counterclockwise on macOS, preserving high visual fidelity and avoiding playback/compatibility issues?

Answer

Command (single input)

ffmpeg -i INPUT.MOV -vf "transpose=2" -c:v prores_ks -profile:v 3 -c:a copy OUTPUT_ROTATED.mov

Step-by-step procedure

1. Set the input filename to the source MOV (example: INPUT.MOV).
2. Run the command above to apply a 90° counterclockwise rotation via transpose=2.
3. Verify playback and orientation in a target player/editor.

Notes

• transpose=2: 90° counterclockwise rotation.
• prores_ks -profile:v 3: Produces an editing-friendly, high-quality output (file size may increase significantly).
• -c:a copy: Preserves the original audio without re-encoding (when compatible).

II. Concatenate multiple MOV files sequentially (method proven working)

Question

How to concatenate several MOV clips sequentially into a single MOV output, avoiding timestamp errors and maintaining stable playback?

Answer

Command (three inputs)

ffmpeg \
-i CLIP_01.MOV -i CLIP_02.MOV -i CLIP_03.MOV \
-filter_complex "[0:v][0:a][1:v][1:a][2:v][2:a]concat=n=3:v=1:a=1[v][a]" \
-map "[v]" -map "[a]" \
-c:v prores_ks -profile:v 3 \
-c:a pcm_s16le \
-movflags +faststart \
OUTPUT_CONCAT.mov

Step-by-step procedure

1. Rename or reference the intended clips as CLIP_01.MOV, CLIP_02.MOV, CLIP_03.MOV (any filenames are acceptable; placeholders are shown).
2. Run the command above to concatenate in the exact input order.
3. Confirm duration and clip boundaries in a player/editor.

Why this method is preferred in practice

Stream-copy concatenation (-c copy) can emit warnings such as Non-monotonic DTS and may produce incorrect timestamps. The filter-based concat method generates a clean, continuous timeline and has proven stable.

Notes

• concat=n=3:v=1:a=1: Concatenates 3 segments, producing 1 continuous video stream and 1 continuous audio stream.
• -c:a pcm_s16le: Uncompressed PCM audio minimizes timestamp edge cases and improves editability (file size increases).
• -movflags +faststart: Improves responsiveness for preview/streaming workflows by placing key metadata earlier in the file.

III. Practical filename patterns (copy/paste ready)

Question

Which filename conventions support repeated use without rewriting commands?

Answer

Rotation naming

1. Input: PROJECT_SCENE_TAKE.mov
2. Output: PROJECT_SCENE_TAKE_rotCCW90_prores.mov

Concatenation naming

1. Inputs: SEG_01.mov, SEG_02.mov, SEG_03.mov
2. Output: SEG_01-03_concat_prores.mov

Written on January 1, 2026

macOS ffmpeg workflow notes: controlling file size while keeping reliable rotation and concatenation (Written January 1, 2026) (Written January 4, 2026)

I. Summary of the root cause

ProRes 422 HQ (prores_ks -profile:v 3) is an editing mezzanine codec with a far higher bitrate than typical phone/camera “delivery” MOV files. As a result, 20 MB source clips commonly expand to hundreds of MB or multiple GB after rotation/concatenation when ProRes HQ is enforced.

II. Recommended decision table (quality vs size vs reliability)

III. Rotation: keep size reasonable (two reliable options)

Option A — Rotation + re-encode to H.264 (recommended for small output)

ffmpeg -i INPUT.MOV \
  -vf "transpose=2" \
  -c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
  -c:a aac -b:a 160k \
  -movflags +faststart \
  OUTPUT_ROTATED.mp4

1. CRF controls quality/size: smaller number = higher quality/larger size. Typical range: 18–23.
2. yuv420p improves broad compatibility.

Option B — Rotation while staying in ProRes, but smaller than HQ

ffmpeg -i INPUT.MOV \
  -vf "transpose=2" \
  -c:v prores_ks -profile:v 1 \
  -c:a pcm_s16le \
  -movflags +faststart \
  OUTPUT_ROTATED_PRORES_LT.mov

1. -profile:v 1 = ProRes LT (materially smaller than HQ).
2. For even smaller: -profile:v 0 (Proxy).

IV. Concatenation: avoid large files and timestamp issues

Best-case (no re-encode): concat demuxer + stream copy

This is the smallest and fastest approach, but requires matching codecs/parameters across clips.

Step 1 — Create list.txt

printf "file '%s'\n" CLIP_01.MOV CLIP_02.MOV CLIP_03.MOV > list.txt

Step 2 — Concatenate

ffmpeg -f concat -safe 0 -i list.txt \
  -c copy \
  -movflags +faststart \
  OUTPUT_CONCAT.mov

If this fails (errors about non-monotonous DTS, differing streams, or missing audio), use re-encode below.

Reliable (re-encode): concat filter + H.264 to keep size small

ffmpeg \
  -i CLIP_01.MOV -i CLIP_02.MOV -i CLIP_03.MOV \
  -filter_complex "[0:v][0:a][1:v][1:a][2:v][2:a]concat=n=3:v=1:a=1[v][a]" \
  -map "[v]" -map "[a]" \
  -c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
  -c:a aac -b:a 160k \
  -movflags +faststart \
  OUTPUT_CONCAT.mp4

1. Keeps the “stable playback and seeking” benefits of re-encoding while preventing ProRes-size explosion.
2. If the source has variable frame rate and seeking remains imperfect, adding a constant FPS can help.
3. Append to video options: -r 30 (or -r 60) as appropriate.

V. Practical size intuition chart (rough)

Small  ┃ H.265 (CRF ~22–26)
       ┃ H.264 (CRF ~18–23)
       ┃ ProRes Proxy (profile 0)
       ┃ ProRes LT    (profile 1)
Large  ┃ ProRes HQ    (profile 3)

VI. Suggested default settings (repeatable)

• Delivery / sharing / archiving with small size: libx264 -crf 20 -preset slow, AAC 160k
• Editing-friendly without extreme size: prores_ks -profile:v 1 (LT)
• Only when clip parameters match: concat demuxer + -c copy

Written on January 4, 2026

macOS ffmpeg workflow notes: side-by-side and vertical stacking for two video files (Written January 4, 2026) (Written January 4, 2026)

I. Purpose

Provide a compact, repeatable workflow to combine two video files into a single output in either side-by-side (left–right) or vertical stack (top–bottom) layouts, including reliable handling for unequal durations.

II. Assumptions

• ffmpeg is installed and available in the terminal.
• Two input files are available (example names): IMG_9438.MOV and ai_yolov8.mp4.
• Outputs are created as H.264 MP4 for broad compatibility and reasonable file size.

III. Quick selection table

IV. Common principles (applies to all commands below)

Each input is normalized to a shared height (720) while preserving aspect ratio via scale=-1:720. Pixel aspect ratio is normalized using setsar=1 to avoid unintended stretching.

• Layout operator: hstack for left–right, vstack for top–bottom
• Duration mismatch handling: stop at the shorter clip (shortest=1) or freeze the shorter clip (tpad stop_mode=clone)
• Output encoding: libx264 -crf 20 -preset slow for good quality and manageable size
• Audio policy: default commands disable audio (-an) for clarity in visual comparisons

V. Side-by-side (left–right) workflows

Side-by-side, stop at shorter clip (recommended default)

ffmpeg -i IMG_9438.MOV -i ai_yolov8.mp4 \
-filter_complex "[0:v]scale=-1:720,setsar=1[v0];[1:v]scale=-1:720,setsar=1[v1];[v0][v1]hstack=inputs=2:shortest=1[v]" \
-map "[v]" \
-c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
-an \
-movflags +faststart \
side_by_side.mp4

1. Confirm inputs: IMG_9438.MOV (left) and ai_yolov8.mp4 (right).
2. Run the command to generate side_by_side.mp4.
3. Verify that playback stops when the shorter input ends.

Side-by-side, match longer clip (freeze shorter clip on last frame)

ffmpeg -i IMG_9438.MOV -i ai_yolov8.mp4 \
-filter_complex "[0:v]scale=-1:720,setsar=1,tpad=stop_mode=clone:stop_duration=3600[v0];[1:v]scale=-1:720,setsar=1,tpad=stop_mode=clone:stop_duration=3600[v1];[v0][v1]hstack=inputs=2[v]" \
-map "[v]" \
-c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
-an \
-movflags +faststart \
side_by_side_full_length.mp4

1. Run the command to generate side_by_side_full_length.mp4.
2. Confirm that the shorter input freezes on the last frame while the longer input continues.
3. If needed, reduce stop_duration from 3600 to a smaller number (in seconds).

VI. Vertical stack (top–bottom) workflows

Vertical stack, stop at shorter clip (recommended default)

ffmpeg -i IMG_9438.MOV -i ai_yolov8.mp4 \
-filter_complex "[0:v]scale=-1:720,setsar=1[v0];[1:v]scale=-1:720,setsar=1[v1];[v0][v1]vstack=inputs=2:shortest=1[v]" \
-map "[v]" \
-c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
-an \
-movflags +faststart \
vertical_stack.mp4

1. Confirm inputs: IMG_9438.MOV (top) and ai_yolov8.mp4 (bottom).
2. Run the command to generate vertical_stack.mp4.
3. Verify that playback stops when the shorter input ends.

Vertical stack, match longer clip (freeze shorter clip on last frame)

ffmpeg -i IMG_9438.MOV -i ai_yolov8.mp4 \
-filter_complex "[0:v]scale=-1:720,setsar=1,tpad=stop_mode=clone:stop_duration=3600[v0];[1:v]scale=-1:720,setsar=1,tpad=stop_mode=clone:stop_duration=3600[v1];[v0][v1]vstack=inputs=2[v]" \
-map "[v]" \
-c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
-an \
-movflags +faststart \
vertical_stack_full_length.mp4

1. Run the command to generate vertical_stack_full_length.mp4.
2. Confirm that the shorter input freezes on the last frame while the longer input continues.
3. If needed, reduce stop_duration from 3600 to a smaller number (in seconds).

VII. Optional audio handling

The default commands above disable audio (-an). To keep audio from a single input, remove -an and add one of the mappings below.

Keep audio from the first input (top or left)

-map 0:a -c:a aac -b:a 160k

Keep audio from the second input (bottom or right)

-map 1:a -c:a aac -b:a 160k

1. Insert the chosen audio mapping before the output filename.
2. If no audio exists in the chosen input, ffmpeg may error; in that case, use -an instead.

VIII. Troubleshooting checklist

Unexpected layout (horizontal vs vertical)

1. Confirm hstack is used for side-by-side, and vstack is used for vertical stacking.
2. Confirm the final stack line contains [v0][v1] followed by the intended stack operator.

Black bars or mismatched geometry

1. Confirm both inputs are scaled to the same height (example: scale=-1:720).
2. If a different target size is preferred, replace 720 with 1080 (or another height) consistently for both inputs.

Timing or sync feels inconsistent

1. If one input is variable frame rate, consider normalizing frame rate by adding -r 30 (or -r 60) to output options.
2. For strict comparison, ensure both inputs start at the intended timepoints before stacking.

IX. Vertical stacking with longer-duration alignment and per-panel text labels

1. Aligning total duration to the longer clip

When two inputs have unequal durations, a naive use of vstack will stop the output as soon as one input ends. To ensure the final stacked video runs for the full duration of the longer clip, the shorter clip must be extended by freezing its final frame.

This is achieved using the tpad filter with stop_mode=clone, which repeats the last decoded frame for a specified duration. To avoid unintended infinite or excessively long outputs, the padding duration should be chosen deliberately.

• Key idea: pad only the shorter clip, and cap the output to the longer duration
• Mechanism: tpad=stop_mode=clone freezes the last frame
• Best practice: explicitly limit output duration using -t

Vertical stack, match longer clip safely (recommended pattern)

ffmpeg -t 26.62 -i IMG_9438.MOV -t 26.62 -i ai_yolov8.mp4 \
-filter_complex "
  [0:v]
    scale=trunc(iw*720/ih/2)*2:720,
    setsar=1,
    tpad=stop_mode=clone:stop_duration=7.8
  [v0];
  [1:v]
    scale=trunc(iw*720/ih/2)*2:720,
    setsar=1
  [v1];
  [v0][v1]
    vstack=inputs=2
  [v]
" \
-map "[v]" \
-c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
-an -movflags +faststart \
vertical_stack_full_length.mp4

In this example, the shorter clip is padded by approximately the duration difference (26.62 − 18.83 ≈ 7.8 seconds), and the output is explicitly capped at the longer clip’s duration.

This approach avoids the common pitfall of using arbitrarily large padding values (e.g. 3600 seconds), which can cause unexpectedly long or seemingly “never-ending” encodes.

2. Inserting text labels into each stacked panel

Text labels are inserted before stacking, directly onto each individual video stream. This ensures that each label remains confined to its own panel (top or bottom), rather than spanning the combined output.

The drawtext filter is used with explicit centering logic based on rendered text width.

• Horizontal centering: x=(w-text_w)/2
• Vertical offset: fixed pixel offset from top (example: y=40)
• Readability: semi-transparent background box

Vertical stack with labels and longer-duration alignment

ffmpeg -t 26.62 -i IMG_9438.MOV -t 26.62 -i ai_yolov8.mp4 \
-filter_complex "
  [0:v]
    scale=trunc(iw*720/ih/2)*2:720,
    setsar=1,
    drawtext=
      text='PyScript version':
      x=(w-text_w)/2:
      y=40:
      fontsize=36:
      fontcolor=white:
      box=1:
      boxcolor=black@0.45:
      boxborderw=12,
    tpad=stop_mode=clone:stop_duration=7.8
  [v0];
  [1:v]
    scale=trunc(iw*720/ih/2)*2:720,
    setsar=1,
    drawtext=
      text='JavaScript version':
      x=(w-text_w)/2:
      y=40:
      fontsize=36:
      fontcolor=white:
      box=1:
      boxcolor=black@0.45:
      boxborderw=12
  [v1];
  [v0][v1]
    vstack=inputs=2
  [v]
" \
-map "[v]" \
-c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
-an -movflags +faststart \
vertical_stack_labeled_full_length.mp4

Implementation notes

• Text is rendered per-input, not on the final stacked frame, preserving visual separation.
• Width is explicitly forced to an even number using trunc(iw*720/ih/2)*2 to satisfy H.264 yuv420p requirements.
• Font size, box opacity, and vertical offset may be adjusted to suit content density.

Conceptual summary

• Duration alignment is achieved by padding the shorter stream, not by letting the stack operator decide.
• Text overlays are safest and most predictable when applied before stacking.
• Explicit duration control prevents runaway encodes and improves reproducibility.

X. Two-by-two grid: three videos plus one static image (video+code panel)

1. Purpose

A common refinement after experimenting with multi-row stacks is to eliminate unused (empty) space and present content as a compact 2×2 grid.

• Row 1: Video #1 (left) + Video #2 (right)
• Row 2: Video #3 (left) + Static image (right, e.g., VS Code source capture)

2. Key constraint: exact geometry is mandatory

Both hstack and vstack require strict dimensional compatibility:

• hstack: input heights must match
• vstack: input widths must match

For a 2×2 grid, the safest pattern is to force every tile to a fixed geometry (example: 1280×720 per tile), producing a combined frame of 2560×1440.

3. Recommended pattern: “fit + pad” for the static image tile

Static screenshots rarely match the target aspect ratio. A predictable approach is: scale to fit within the tile, then pad to the exact target size. This guarantees the final vstack does not fail due to width drift (e.g., 2528 vs 2560).

• Fit: scale=1280:720:force_original_aspect_ratio=decrease
• Pad: pad=1280:720:(ow-iw)/2:(oh-ih)/2

Two-by-two grid with per-panel labels and a static code screenshot

ffmpeg \
  -i IMG_9574.MOV \
  -i IMG_9577.MOV \
  -i IMG_9581.MOV \
  -loop 1 -i third.png \
  -filter_complex "
    [0:v]
      scale=trunc(iw*720/ih/2)*2:720,
      setsar=1,
      drawtext=
        text='Traditional motor w/ remote control':
        x=(w-text_w)/2:
        y=40:
        fontsize=36:
        fontcolor=white:
        box=1:
        boxcolor=black@0.45:
        boxborderw=12
    [r1l];

    [1:v]
      scale=trunc(iw*720/ih/2)*2:720,
      setsar=1,
      drawtext=
        text='LEGO Mindstorms w/ manual motor control':
        x=(w-text_w)/2:
        y=40:
        fontsize=36:
        fontcolor=white:
        box=1:
        boxcolor=black@0.45:
        boxborderw=12
    [r1r];

    [r1l][r1r]
      hstack=inputs=2
    [row1];

    [2:v]
      scale=trunc(iw*720/ih/2)*2:720,
      setsar=1,
      drawtext=
        text='MicroPython script automated motor control':
        x=(w-text_w)/2:
        y=40:
        fontsize=36:
        fontcolor=white:
        box=1:
        boxcolor=black@0.45:
        boxborderw=12
    [r2l];

    [3:v]
      scale=1280:720:force_original_aspect_ratio=decrease,
      pad=1280:720:(ow-iw)/2:(oh-ih)/2:color=black,
      setsar=1,
      drawtext=
        text='Source code (VS Code capture)':
        x=(w-text_w)/2:
        y=40:
        fontsize=36:
        fontcolor=white:
        box=1:
        boxcolor=black@0.45:
        boxborderw=12
    [r2r];

    [r2l][r2r]
      hstack=inputs=2
    [row2];

    [row1][row2]
      vstack=inputs=2
    [v]
  " \
  -map "[v]" \
  -t 35.37 \
  -c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
  -an -movflags +faststart \
  grid_2x2_videos_plus_code.png.mp4

Implementation notes

• Static image persistence: -loop 1 -i third.png keeps the screenshot available throughout the encode.
• Exact row width guarantee: the screenshot tile is forced to 1280×720 via scale+pad, ensuring row2 becomes exactly 2560 pixels wide.
• Duration control: -t 35.37 caps the output to the intended master duration (often the longest video).
• Optional full-duration visibility for shorter clips: apply tpad=stop_mode=clone to the shorter video streams if freezing the last frame is preferred over early termination.

Conceptual summary

• A clean 2×2 grid removes unused space by pairing panels horizontally, then stacking rows vertically.
• A reliable stack requires exact panel geometry; screenshots should be fit-and-padded to the target tile size.
• Labels remain most predictable when applied before stacking on each individual tile stream.

    ffmpeg \
    -i IMG_9574.MOV \
    -i IMG_9577.MOV \
    -i IMG_9581.MOV \
    -loop 1 -i third.png \
    -filter_complex "
    [0:v]
    scale=trunc(iw*720/ih/2)*2:720,
    setsar=1,
    drawtext=
    text='Traditional motor w/ remote control':
    x=(w-text_w)/2:
    y=40:
    fontsize=36:
    fontcolor=white:
    box=1:
    boxcolor=black@0.45:
    boxborderw=12
    [r1l];

    [1:v]
    scale=trunc(iw*720/ih/2)*2:720,
    setsar=1,
    drawtext=
    text='LEGO Mindstorms w/ manual motor control':
    x=(w-text_w)/2:
    y=40:
    fontsize=36:
    fontcolor=white:
    box=1:
    boxcolor=black@0.45:
    boxborderw=12
    [r1r];

    [r1l][r1r]
    hstack=inputs=2
    [row1];

    [2:v]
    scale=trunc(iw*720/ih/2)*2:720,
    setsar=1,
    drawtext=
    text='MicroPython script automated motor control':
    x=(w-text_w)/2:
    y=40:
    fontsize=36:
    fontcolor=white:
    box=1:
    boxcolor=black@0.45:
    boxborderw=12
    [r2l];

    [3:v]
    scale=1280:720:force_original_aspect_ratio=decrease,
    pad=1280:720:(ow-iw)/2:(oh-ih)/2:color=black,
    setsar=1,
    drawtext=
    text='Source code (VS Code capture)':
    x=(w-text_w)/2:
    y=40:
    fontsize=36:
    fontcolor=white:
    box=1:
    boxcolor=black@0.45:
    boxborderw=12
    [r2r];

    [r2l][r2r]
    hstack=inputs=2
    [row2];

    [row1][row2]
    vstack=inputs=2
    [v]
    " \
    -map "[v]" \
    -map "2:a?" \
    -t 35.37 \
    -c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
    -c:a aac -b:a 192k \
    -movflags +faststart \
    grid_2x2_videos_plus_code_with_audio.mp4

4. Audio-selection rationale

Inputs are indexed in declaration order. In this command:

• 0 → IMG_9574.MOV
• 1 → IMG_9577.MOV
• 2 → IMG_9581.MOV (audio source)
• 3 → static image (no audio)

The mapping -map "2:a?" selects audio from the third video input. The trailing ? makes the mapping optional, preventing failure if the audio stream is absent.

Quotation marks are required on macOS to prevent zsh from interpreting ? as a filename wildcard.

5. Implementation notes

• Audio is selected independently of the filter graph, preserving clarity and predictability.
• Widths are forced to even values to satisfy H.264 yuv420p requirements.
• Static images are fit and padded to exact tile dimensions to guarantee stack compatibility.
• Duration is explicitly capped to avoid unintended overrun.

Conceptual summary

• Multi-panel layouts benefit from explicit, single-source audio selection.
• Audio mapping via indexed -map is deterministic and reproducible.
• Shell-safe quoting is essential for portable macOS workflows.

XI. Two-by-two grid with explicit audio selection from a single video source

1. Purpose

In multi-panel compositions that combine several videos and static images into a 2×2 grid, visual information is distributed across panels, but audio should typically originate from one authoritative video source.

This section presents a complete, standalone workflow that:

• Builds a 2×2 grid from three video inputs and one static image
• Applies per-panel text labels prior to stacking
• Explicitly selects audio from a single designated video input
• Avoids shell-related pitfalls on macOS (zsh)

2. Input layout

• Row 1: Video A (left) + Video B (right)
• Row 2: Video C (left) + Static image (right)

Audio is taken exclusively from Video C.

3. Complete command (video grid + labeled panels + audio from one source)

ffmpeg \
  -i IMG_9574.MOV \
  -i IMG_9577.MOV \
  -i IMG_9581.MOV \
  -loop 1 -i third.png \
  -filter_complex "
    [0:v]
      scale=trunc(iw*720/ih/2)*2:720,
      setsar=1,
      drawtext=
        text='Traditional motor w/ remote control':
        x=(w-text_w)/2:
        y=40:
        fontsize=36:
        fontcolor=white:
        box=1:
        boxcolor=black@0.45:
        boxborderw=12
    [r1l];

    [1:v]
      scale=trunc(iw*720/ih/2)*2:720,
      setsar=1,
      drawtext=
        text='LEGO Mindstorms w/ manual motor control':
        x=(w-text_w)/2:
        y=40:
        fontsize=36:
        fontcolor=white:
        box=1:
        boxcolor=black@0.45:
        boxborderw=12
    [r1r];

    [r1l][r1r]
      hstack=inputs=2
    [row1];

    [2:v]
      scale=trunc(iw*720/ih/2)*2:720,
      setsar=1,
      drawtext=
        text='MicroPython script automated motor control':
        x=(w-text_w)/2:
        y=40:
        fontsize=36:
        fontcolor=white:
        box=1:
        boxcolor=black@0.45:
        boxborderw=12
    [r2l];

    [3:v]
      scale=1280:720:force_original_aspect_ratio=decrease,
      pad=1280:720:(ow-iw)/2:(oh-ih)/2:color=black,
      setsar=1,
      drawtext=
        text='Source code (VS Code capture)':
        x=(w-text_w)/2:
        y=40:
        fontsize=36:
        fontcolor=white:
        box=1:
        boxcolor=black@0.45:
        boxborderw=12
    [r2r];

    [r2l][r2r]
      hstack=inputs=2
    [row2];

    [row1][row2]
      vstack=inputs=2
    [v]
  " \
  -map "[v]" \
  -map "2:a?" \
  -t 35.37 \
  -c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
  -c:a aac -b:a 192k \
  -movflags +faststart \
  grid_2x2_videos_plus_code_with_audio.mp4

4. Audio-selection rationale

Inputs are indexed in declaration order. In this command:

• 0 → IMG_9574.MOV
• 1 → IMG_9577.MOV
• 2 → IMG_9581.MOV (audio source)
• 3 → static image (no audio)

The mapping -map "2:a?" selects audio from the third video input. The trailing ? makes the mapping optional, preventing failure if the audio stream is absent.

Quotation marks are required on macOS to prevent zsh from interpreting ? as a filename wildcard.

5. Implementation notes

• Audio is selected independently of the filter graph, preserving clarity and predictability.
• Widths are forced to even values to satisfy H.264 yuv420p requirements.
• Static images are fit and padded to exact tile dimensions to guarantee stack compatibility.
• Duration is explicitly capped to avoid unintended overrun.

Conceptual summary

• Multi-panel layouts benefit from explicit, single-source audio selection.
• Audio mapping via indexed -map is deterministic and reproducible.
• Shell-safe quoting is essential for portable macOS workflows.

XII. Echocardiography 2×2 grid, then timed PNG slides (no audio)

1. Purpose

This workflow composes four echocardiography clips into a labeled 2×2 grid, then appends a short PNG slide sequence (1.png → 4.png, 2 seconds each) at the end. Audio is intentionally disabled (-an), which is appropriate for typical echo captures that carry no meaningful audio stream.

2. Panel layout and labeling

• Top-left: PLAX.mp4 → label PLAX
• Top-right: PLAX_c.mp4 → label PLAX (Color Doppler: blood flow across MV & AV)
• Bottom-left: PSAX.mp4 → label PSAX
• Bottom-right: PSAX_c.mp4 → label PSAX (Color Doppler: blood flow across Pulmonary valve)

3. What the command does (high-level pipeline)

1. Normalizes frame rate to a stable target (fps=30) to reduce irregular timing effects across clips that may have different original frame rates.
2. Normalizes geometry by scaling each input to a common panel height (720) while preserving aspect ratio, with even widths enforced for H.264 yuv420p compatibility.
3. Applies per-panel labels using drawtext prior to stacking, keeping text confined to each tile.
4. Builds a 2×2 grid using xstack, producing a combined frame of approximately 2560×1440.
5. Appends slides by turning each PNG into a 2-second video segment, scaling and padding each to exactly match the grid resolution, and concatenating grid + 1.png + 2.png + 3.png + 4.png into a single output.

4. Complete command (ready to reuse)

ffmpeg -y \
  -i PLAX.mp4 \
  -i PLAX_c.mp4 \
  -i PSAX.mp4 \
  -i PSAX_c.mp4 \
  -loop 1 -t 2 -i 1.png \
  -loop 1 -t 2 -i 2.png \
  -loop 1 -t 2 -i 3.png \
  -loop 1 -t 2 -i 4.png \
  -filter_complex "
    [0:v]
      fps=30,
      scale=trunc(iw*720/ih/2)*2:720,
      setsar=1,
      drawtext=
        text='PLAX':
        x=(w-text_w)/2:
        y=40:
        fontsize=40:
        fontcolor=white:
        box=1:
        boxcolor=black@0.50:
        boxborderw=14
    [r1l];

    [1:v]
      fps=30,
      scale=trunc(iw*720/ih/2)*2:720,
      setsar=1,
      drawtext=
        text='PLAX (Color Doppler\: blood flow across MV & AV)':
        x=(w-text_w)/2:
        y=40:
        fontsize=40:
        fontcolor=white:
        box=1:
        boxcolor=black@0.50:
        boxborderw=14
    [r1r];

    [2:v]
      fps=30,
      scale=trunc(iw*720/ih/2)*2:720,
      setsar=1,
      drawtext=
        text='PSAX':
        x=(w-text_w)/2:
        y=40:
        fontsize=40:
        fontcolor=white:
        box=1:
        boxcolor=black@0.50:
        boxborderw=14
    [r2l];

    [3:v]
      fps=30,
      scale=trunc(iw*720/ih/2)*2:720,
      setsar=1,
      drawtext=
        text='PSAX (Color Doppler\: blood flow across Pulmonary valve)':
        x=(w-text_w)/2:
        y=40:
        fontsize=40:
        fontcolor=white:
        box=1:
        boxcolor=black@0.50:
        boxborderw=14
    [r2r];

    [r1l][r1r][r2l][r2r]
      xstack=inputs=4:layout=0_0|w0_0|0_h0|w0_h0:fill=black
    [grid];

    [4:v]
      fps=30,
      scale=2560:1440:force_original_aspect_ratio=decrease,
      pad=2560:1440:(ow-iw)/2:(oh-ih)/2:color=black,
      setsar=1
    [p1];

    [5:v]
      fps=30,
      scale=2560:1440:force_original_aspect_ratio=decrease,
      pad=2560:1440:(ow-iw)/2:(oh-ih)/2:color=black,
      setsar=1
    [p2];

    [6:v]
      fps=30,
      scale=2560:1440:force_original_aspect_ratio=decrease,
      pad=2560:1440:(ow-iw)/2:(oh-ih)/2:color=black,
      setsar=1
    [p3];

    [7:v]
      fps=30,
      scale=2560:1440:force_original_aspect_ratio=decrease,
      pad=2560:1440:(ow-iw)/2:(oh-ih)/2:color=black,
      setsar=1
    [p4];

    [grid][p1][p2][p3][p4]
      concat=n=5:v=1:a=0
    [v]
  " \
  -map "[v]" \
  -c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \
  -an \
  -movflags +faststart \
  echo_grid_2x2_then_4pngs.mp4

5. Usage checklist (minimal)

1. Place PLAX.mp4, PLAX_c.mp4, PSAX.mp4, PSAX_c.mp4 in the working directory.
2. Place 1.png, 2.png, 3.png, 4.png in the same directory.
3. Run the command to produce echo_grid_2x2_then_4pngs.mp4.
4. If the echo clips end too quickly relative to the slides, consider appending additional slides or extending slide durations (increase -t 2).

6. Practical tuning points

• Label appearance: adjust fontsize, boxcolor, and boxborderw while keeping the overlay applied before xstack.
• Panel size: change the per-panel target height from 720 to 540 or 1080, and update the slide scaling target accordingly (grid becomes ~2×width by 2×height).
• Slide duration: modify each PNG input duration by changing -t 2 per PNG to another value.

Conceptual summary

• A stable 2×2 echocardiography comparison grid is created via per-tile normalization, labeling, then xstack.
• A timed slide tail is appended by converting static PNGs into short video segments, normalized to the grid resolution, then concatenated.
• Disabling audio (-an) simplifies output when source clips are silent and prevents unexpected stream mapping issues.

Written on January 4, 2026

FFmpeg macOS workflow A: bitrate-based downsizing, then fast concatenation (Written February 9, 2026)

I. Purpose and intent

This workflow documents a pragmatic procedure to reduce file size for multiple GoPro-class MP4 sources and then concatenate them into a single MP4 efficiently. The approach emphasizes operational clarity and reproducibility, so that the procedure may be recalled and repeated without ambiguity at a later time.

II. Preconditions and operating assumptions

• Environment: macOS Terminal with FFmpeg available.
• Input files: GH010880.MP4, GH020880.MP4, GH030880.MP4, GH040880.MP4 located in the same working directory.
• Design constraint: All intermediate files must share identical encoding parameters to allow concatenation without re-encoding.

III. Method overview

IV. Single macOS command sequence

The following block is intended to be executed as a single shell workflow. Line breaks are preserved for natural reading and later reuse.

# Step 1 — Re-encode all sources to a smaller, uniform format
for f in GH010880.MP4 GH020880.MP4 GH030880.MP4 GH040880.MP4; do
  base="${f%.MP4}"

  ffmpeg -y -i "$f" \
    -c:v libx264 -preset slow -b:v 900k \
    -c:a aac -b:a 128k \
    -movflags +faststart \
    "${base}_small.mp4"
done

# Step 2 — Create concat list
printf "file 'GH010880_small.mp4'\nfile 'GH020880_small.mp4'\nfile 'GH030880_small.mp4'\nfile 'GH040880_small.mp4'\n" > list.txt

# Step 3 — Concatenate without re-encoding
ffmpeg -f concat -safe 0 -i list.txt -c copy GH_combined.mp4

V. Practical tuning notes

• Stronger compression: lower the video bitrate (example: -b:v 700k).
• Higher visual fidelity: raise the bitrate (example: -b:v 1200k).
• Observed limitation: At low bitrates, fine texture and fast motion may degrade despite unchanged resolution.

FFmpeg macOS workflow B: CRF high-quality downsizing, then fast concatenation

I. Purpose and intent

This workflow records a quality-oriented approach that preserves the original spatial resolution while reducing file size through constant-quality encoding. The intent is to maintain visual coherence and fine detail, particularly for action-camera footage, while still enabling fast concatenation.

II. Preconditions and operating assumptions

• Environment: macOS Terminal with FFmpeg available.
• Input files: GH010880.MP4, GH020880.MP4, GH030880.MP4, GH040880.MP4.
• Resolution handling: No scaling is applied; original frame dimensions are retained.

III. Method overview

IV. Single macOS command sequence

CRF is set to 18 as a high-quality baseline. Line breaks are preserved to maintain readability and recall.

# Step 1 — High-quality re-encode (constant quality, full resolution)
for f in GH010880.MP4 GH020880.MP4 GH030880.MP4 GH040880.MP4; do
  base="${f%.MP4}"

  ffmpeg -y -i "$f" \
    -c:v libx264 -preset slow -crf 18 \
    -pix_fmt yuv420p \
    -c:a aac -b:a 160k \
    -movflags +faststart \
    "${base}_hq.mp4"
done

# Step 2 — Concatenate without re-encoding
printf "file 'GH010880_hq.mp4'\nfile 'GH020880_hq.mp4'\nfile 'GH030880_hq.mp4'\nfile 'GH040880_hq.mp4'\n" > list.txt

ffmpeg -f concat -safe 0 -i list.txt -c copy GH_combined_hq.mp4

V. CRF selection guide

VI. Output artifacts

• Intermediate files: *_small.mp4 or *_hq.mp4
• Final output: GH_combined.mp4 or GH_combined_hq.mp4
• Auxiliary file: list.txt

Written on February 9, 2026

Emacs

During undergraduate studies in Computer Science, Emacs was recommended and has been used for over two decades. Familiarity with its shortcuts has facilitated work in C kernel programming and debugging. This document serves both as a guide for readers to grasp the benefits of Emacs and as a resource for personal learning, combining well-known features with newly explored aspects intended for future use.

(D) Browsing with Emacs: Benefits and How to Navigate

EWW (Emacs Web Wowser) is a built-in web browser in Emacs that allows browsing the web within a text-based environment. Although minimal compared to graphical browsers, EWW provides an efficient means to navigate the web while fully leveraging the keyboard-driven workflow appreciated by many Emacs users.

1': Ctrl+v (Page Down), Meta+v (Page Up), Arrow Keys / Ctrl+n (Down) / Ctrl+p (Up)

2': ESC+x eww-view-source

3': Add (setq eww-search-prefix "https://www.google.com/search?q=") to configuration

Benefits of Using EWW:

• Lightweight and Efficient: Being text-based, EWW consumes fewer system resources compared to graphical browsers like Firefox or Chrome. It is ideal for reading articles, documentation, or other text-heavy content without distractions from ads or media files.
• Integrated Workflow: EWW seamlessly fits into the Emacs environment, allowing the use of familiar keybindings, split windows, and buffer management techniques while browsing the web. For instance, a webpage can be opened in one buffer while taking notes in another within the same workspace.
• Distraction-Free Browsing: EWW's text-centric design minimizes distractions from images, ads, or multimedia, fostering a more focused browsing experience. This is particularly useful for technical reading, documentation, or studying code examples.
• Scripting and Automation: Since EWW operates within Emacs, the pages viewed can be parsed, analyzed, or manipulated programmatically using Emacs Lisp. This capability opens possibilities for web scraping, automated tasks, and custom workflows not easily achievable with traditional browsers.

While EWW does not replace full-featured browsers like Firefox for multimedia-heavy browsing or complex web applications, it offers an efficient, minimalistic browsing experience for those who prefer staying within the Emacs ecosystem and rely on text-based content.

(E) Enhancing File Management with Dired Mode

Dired (Directory Editor) mode in Emacs provides a powerful and interactive method for managing files. It facilitates browsing and manipulating files and directories within the editor, thereby streamlining file system operations.

Dired mode transforms Emacs into a comprehensive file management system, providing the necessary tools to handle complex file operations without leaving the editor environment.

(F) Emacs Lisp: Extending Emacs Functionality

Emacs Lisp (Elisp) is the programming language embedded within Emacs, allowing for extensive customization and extension of the editor's capabilities. Emacs Lisp enables the writing of scripts, defining new commands, and creating custom workflows tailored to individual needs.

Custom Key Bindings:

Emacs Lisp can be used to remap existing key bindings or create new ones, enhancing the efficiency of the keyboard-driven workflow. For example, binding a frequently used command to a simpler key combination can streamline operations.

;; Example: Bind F5 to save all buffers
(global-set-key (kbd "<f5>") 'save-some-buffers)  

Automating Tasks:

Repetitive tasks can be automated using Emacs Lisp, reducing the need for manual intervention and minimizing the potential for errors. Automating file operations, text transformations, or buffer management are common applications.

;; Example: Automatically delete trailing whitespace on save
(add-hook 'before-save-hook 'delete-trailing-whitespace)  

Defining New Commands:

Users can define new interactive commands to perform specialized functions, enhancing the editor's functionality to suit specific workflows or projects.

;; Example: Define a command to insert the current date
(defun insert-current-date ()
   "Insert the current date at point."
   (interactive)
   (insert (format-time-string "%Y-%m-%d")))

(global-set-key (kbd "C-c d") 'insert-current-date)  

Creating Custom Modes:

Emacs Lisp allows for the creation of new major or minor modes, providing tailored environments for different programming languages, file types, or project requirements.

;; Example: Define a simple minor mode
(define-minor-mode my-custom-mode
   "A simple custom minor mode."
   :lighter " MyMode"
   :keymap (let ((map (make-sparse-keymap)))
      (define-key map (kbd "C-c m") 'insert-current-date)
       map))

(add-hook 'text-mode-hook 'my-custom-mode)  

Examples of Emacs Lisp Enhancements:

• Enhanced Navigation: Creating functions to jump to specific sections within a file or to navigate project directories more efficiently.
• Custom UI Elements: Adding new UI components such as toolbars, menus, or status indicators to provide additional information or functionality within the editor.
• Integration with External Tools: Writing Emacs Lisp scripts to interface with external applications or services, enabling seamless integration and data exchange.

Emacs Lisp empowers users to transform Emacs into a highly personalized and powerful development environment. By leveraging Emacs Lisp, users can tailor Emacs to meet their unique requirements, enhancing productivity and fostering an efficient workflow.

Advanced string replacement techniques in Emacs 🔍

I. Foundation — built‑in replacement commands

Emacs offers several native commands for interactive or automatic string substitution. The macOS convention is used throughout (⌥ = Meta (M), ⌘ = Super (s)).

• M % (query-replace) — step‑by‑step confirmation for each match.
• M ⇧ % (query-replace-regexp) — regular‑expression variant with identical prompts.
• M‑x replace-string — unattended, buffer‑wide substitution of literal text.

1. Command comparison ⚙️

   Command	Scope & confirmation	Pattern type	Typical keystroke
   query‑replace	Interactive, buffer or region	Literal	M %
   query‑replace‑regexp	Interactive, buffer or region	Emacs Lisp regexp	M ⇧ %
   replace‑string	Automatic, buffer or region	Literal	M‑x replace-string

II. Advanced scenarios

The following examples illustrate practical refactoring patterns and the reasoning behind each step.

1. Global refactor with captured groups ✨

   M ⇧ %  ^\(defun\s-+\)old_\(.*\)$ RET \1new_\2 RET !

   What happens:

   • ^\(defun\s-+\) captures the function keyword plus its required space into Group 1.
   • old_\(.*\)$ captures the remainder of the symbol (e.g. old_process) into Group 2.
   • The replacement string \1new_\2 rebuilds each definition as (defun new_process …), preserving the original suffix.
   • ! at the prompt answers “yes to all,” executing replacements across the entire buffer without further queries.

   This technique is ideal for systematic API renaming after a naming‑policy change.

2. Selective region replacement

   1. Activate a region—perhaps an individual function—using C space to mark the start and motion keys (e.g. M >) to mark the end.
   2. Invoke M % (or M ⇧ % for regex) and supply search/replacement terms.
   3. Only occurrences inside the highlighted region are offered for confirmation, providing a safety net against global side effects.

   Regional replacement is particularly useful when refactoring temporary variables inside a long file while leaving other sections untouched.

3. Backward traversal 🔄

   C r M %

   The prefix C r calls query-replace in reverse, scanning from point toward the beginning of the buffer (BOB). Reverse traversal prevents accidental double replacements when iterating through matches already passed during forward edits.

4. Embedded newline and tab literals

   M ⇧ %  ,\s-*\\n RET ,\n\t RET !

   Purpose: re‑formatting comma‑separated JSON arrays so that each element begins on a new, indented line.

   • \s-* matches any horizontal whitespace.
   • The search pattern ends with a literal escaped newline \\n, ensuring the match includes the line break itself.
   • The replacement inserts a true newline (\n) followed by a tab (\t) before the next array element.

   The command may be combined with narrowing (C‑x n n) to focus on a JSON block without disturbing surrounding code.

III. Complementary replacement methods

• Multiple‑cursors (📌 mc/edit-lines) — simultaneous edits on several lines, excellent for column‑wise renaming.
• Iedit (🖋️ iedit-mode) — highlights and live‑updates every occurrence of the symbol at point.
• Occur edit mode — M‑x occur followed by C‑c C‑o turns the *Occur* buffer writable; committed changes propagate back.
• Dired batch replacement — within Dired, Q invokes dired-do-query-replace-regexp across marked files.
• Keyboard macros — record with F3/F4, then replay on each match found via next-error or grep.

IV. Best‑practice checklist ✅

• Create checkpoints — save buffers and commit to version control before large substitutions.
• Restrict scope — narrow (C‑x n n) or operate within occur/grep buffers to avoid unintended files.
• Preview wisely — . shows context for the current match; ^ toggles case‑sensitive mode on the fly.
• Understand escapes — replacement escapes (\n \t \1) follow Emacs Lisp conventions, not POSIX syntax.

Written on May 11, 2025

Zsh

Apple’s Transition from Bash to Zsh

In 2019, Apple officially adopted Zsh (Z Shell) as the default shell, starting with macOS Catalina (10.15). This transition marked a significant change from the previously utilized Bash, which had been the default since the inception of macOS. The switch was largely driven by licensing issues and the enhanced features offered by Zsh, making it a more appealing choice for modern developers and power users.

Licensing Issues

Apple's decision to shift from Bash to Zsh was influenced substantially by licensing concerns. Until version 3.2, Bash was licensed under the GNU General Public License v2 (GPLv2), which posed fewer restrictions on redistribution and modification. Apple continued using this version for many years.

However, with the release of Bash 4.0, the license changed to GPLv3, which introduced stricter conditions. Under GPLv3:

• Organizations are required to share any modifications made to GPLv3-licensed software under the same license, potentially exposing proprietary changes.
• These restrictions were generally less favorable for companies inclined to maintain closed-source software, leading Apple to avoid updating to later versions of Bash.

By transitioning to Zsh, which is licensed under an MIT-like license, Apple was able to circumvent these issues. This permissive license allowed Apple to include Zsh without the obligation to disclose proprietary modifications, aligning more effectively with Apple’s distribution model.

Advantages of Zsh Over Bash

Apart from addressing licensing concerns, Zsh provided various technical advantages that improved the user experience and rendered it a more suitable choice for Apple’s ecosystem.

1. Permissive Licensing

The MIT-like license associated with Zsh afforded Apple greater flexibility. Unlike GPLv3, it does not impose the requirement to share modifications, permitting Apple to distribute Zsh freely without concerns over proprietary rights.

2. Enhanced Features for Power Users

Zsh offers a range of features that enhance productivity and streamline shell interactions, which are particularly beneficial for developers:

• Advanced Auto-Completion: Zsh includes sophisticated auto-completion capabilities with context-aware suggestions and partial completion, facilitating faster and more accurate command input.
• Improved Globbing and Pattern Matching: Zsh provides robust globbing features, making complex file operations more manageable.
• Customization Options and Plugins: Frameworks such as Oh My Zsh and Prezto offer extensive customization opportunities, from themes to plugins, which help tailor the terminal environment to meet specific preferences.

3. User-Configurable Options and Prompt Customization

Zsh supports a broad spectrum of configuration options, enabling users to personalize nearly every aspect of the shell. This includes the capability to create dynamic prompts that display real-time information, contributing to a more informative and engaging terminal experience.

Community and Ecosystem Support

Zsh’s popularity among developers and system administrators has fostered a vibrant community that actively provides resources, such as:

• Plugins and Themes: Numerous plugins and themes are available, enhancing functionality, ease of use, and aesthetic appeal.
• Comprehensive Documentation: The community has generated extensive documentation and tutorials, simplifying the process of learning and customizing Zsh.

In adopting Zsh as the default shell, Apple aligned with the preferences of a considerable portion of its developer user base. Many developers had already embraced Zsh for its advanced features, and the switch made macOS more intuitive and appealing to this audience.

Security and Maintenance Benefits

Shifting to Zsh also facilitated a departure from the aging Bash 3.2, bringing several advantages in terms of security and maintainability:

• Regular Updates: Zsh receives updates more frequently than Bash 3.2, ensuring access to the latest features and security patches.
• Reduced Legacy Dependencies: By leaving Bash 3.2 behind, Apple mitigated the security risks associated with maintaining outdated software components.

Using Wildcards with scp in Zsh

Understanding Zsh’s Globbing Behavior

1. Local vs. Remote Expansion

When utilizing wildcards with scp, it is important to recognize that Zsh may attempt to expand these wildcards locally before executing the command. For example, a command intended to copy all .txt files from a remote server might resemble:

scp user@remote:/path/to/files/*.txt /local/destination/

Zsh might expand *.txt based on the local file system, potentially leading to unintended behavior. This happens because Zsh’s default behavior involves expanding wildcards during the globbing phase, which occurs before the command is executed. If matching files exist in the specified local path, Zsh replaces the wildcard with these files.

2. Why Escaping Wildcards Works

To ensure the wildcard is interpreted on the remote server rather than locally, escaping the wildcard with \* is necessary:

scp user@remote:/path/to/files/\*.txt /local/destination/

Escaping the asterisk directs Zsh to pass the wildcard to scp without local expansion, allowing the remote shell to interpret *.txt and carry out the intended file selection.

Alternative Methods to Control Expansion

Several techniques can prevent Zsh from performing local expansion on wildcards meant for remote servers:

• Quoting the Remote Path: Using single or double quotes around the path prevents local expansion by Zsh.

  scp 'user@remote:/path/to/files/*.txt' /local/destination/

• Using noglob: Zsh’s noglob directive disables wildcard expansion for the specified command.

  noglob scp user@remote:/path/to/files/*.txt /local/destination/

• Using rsync for Complex Transfers: For advanced file transfers, especially those involving recursion and selective inclusion/exclusion, rsync offers better control over wildcard patterns.

  rsync -av --include='*.txt' --exclude='*' user@remote:/path/to/files/ /local/destination/

Testing and Troubleshooting with scp

For verification and troubleshooting, the manner in which Zsh interprets a command can be checked by prepending it with echo:

echo scp user@remote:/path/to/files/\*.txt /local/destination/

Alternatively, using the -v option with scp yields verbose output, aiding in the diagnosis of file transfer issues:

scp -v 'user@remote:/path/to/files/*.txt' /local/destination/

Best Practices for Wildcards in Zsh

• Always Escape Remote Wildcards: Use \*, quotes, or noglob to ensure wildcards are processed on the remote server.
• Utilize Quotes for Complex Patterns: For patterns containing multiple wildcards or special characters, quoting ensures the entire pattern is treated as a single argument.
• Review Zsh’s Globbing Options: The setopt and unsetopt commands in Zsh allow for adjustments to wildcard handling. Reviewing these settings can assist in tailoring Zsh’s behavior to specific needs.

Setting Up Environment Variables and Frequently Used Commands in Linux and macOS (zsh or bash)

Configuring environment variables and adding aliases or functions for frequently used commands can greatly enhance efficiency in the command-line environment. This guide provides detailed instructions on how to set environment variables temporarily and permanently, both for individual users and system-wide, as well as how to add aliases and functions in zsh or bash shells, applicable to both Linux and macOS systems.

(A) Setting Up Environment Variables Temporarily

To set an environment variable for the current terminal session, use the export command. This change will only persist for the duration of the session and will be cleared once the terminal is closed.

Example: To temporarily set the PYTHONPATH environment variable:

# Temporarily set PYTHONPATH
export PYTHONPATH="/path/to/python/libs"

This sets the PYTHONPATH variable to include the specified directory for the current session.

(B) Setting Up Environment Variables Permanently

To make environment variables persist across sessions, they must be added to the shell's configuration file. For zsh users, this is typically ~/.zshrc; for bash users, it is ~/.bashrc.

Step-by-Step Instructions:

1. Open the Configuration File
   • For zsh users:

     # Open .zshrc with a text editor
     emacs ~/.zshrc

   • For bash users:

     # Open .bashrc with a text editor
     emacs ~/.bashrc

2. Add Environment Variables

   For example, to set the PYTHONPATH environment variable permanently:

   # Set PYTHONPATH permanently
   export PYTHONPATH="/path/to/python/libs"

   If using pyenv, it may be necessary to add:

   # Set up pyenv
   export PYENV_ROOT="$HOME/.pyenv"
   export PATH="$PYENV_ROOT/bin:$PATH"

3. Save and Exit

   Save the file and exit the text editor.

4. Apply the Changes Immediately
   • For zsh users:

     # Apply changes
     source ~/.zshrc

   • For bash users:

     # Apply changes
     source ~/.bashrc

Note: On macOS, the default shell is zsh (since macOS Catalina). The same steps apply for setting environment variables in zsh on macOS.

(C) Setting Environment Variables System-Wide

For environment variables that should be available to all users on the system, add them to system-wide configuration files. On Linux, these files are /etc/environment, /etc/profile, or /etc/bash.bashrc. On macOS, system-wide configurations for zsh can be added to /etc/zshenv or /etc/zshrc.

Editing System-Wide Configuration Files

1. Open the Appropriate File with Administrative Privileges
   • On Linux:
     • To edit /etc/environment:

       sudo emacs /etc/environment

     • To edit /etc/profile:

       sudo emacs /etc/profile

   • On macOS (zsh):

     sudo emacs /etc/zshrc

2. Add Environment Variables

   For example, to set PYTHONPATH globally:

   # Set PYTHONPATH globally
   export PYTHONPATH="/usr/local/lib/python3.9/site-packages"

3. Save and Apply Changes

   Save the file and exit the text editor.

   To apply the changes, log out and log back in, or source the configuration file. Note that changes to some system-wide files may require a system reboot or re-login to take effect.

(D) Adding Frequently Used Commands with Aliases and Functions

Aliases and functions allow for efficient command reuse and can be added to the shell's configuration files.

D-1) Adding Aliases

Aliases are shortcuts for commands. To add aliases:

1. Open the Configuration File
   • For zsh users:

     # Open .zshrc. If absent, use .zprofile
     emacs ~/.zshrc

   • For bash users:

     # Open .bashrc
     emacs ~/.bashrc

2. Add Alias Definitions

   # Alias to compress and split a folder
   alias compress_folder='FOLDER="folder_name" && tar -zcvf - "$FOLDER" | split -b 4480M - "${FOLDER}.tar.gz."'
   
   # Alias to search a specific folder for a pattern
   alias grep_designated_folder='find /path/to/designated/folder -type f -print0 | xargs -0 grep -i "###" 1> tmp1 2> tmp2'

   • compress_folder: Compresses and splits a folder into chunks.
   • grep_designated_folder: Searches a specific folder for a pattern.
3. Save and Apply Changes

   # Apply alias changes for zsh
   source ~/.zshrc
   
   # Apply alias changes for zsh when using .zprofile
   source ~/.zprofile
   
   # Apply alias changes for bash
   source ~/.bashrc

D-2) Adding Functions

Functions provide more flexibility with parameters than aliases. To add functions:

Add Function Definitions

# Function to compress a folder with a given name
compress_folder() {
   FOLDER="$1"
   tar -zcvf - "$FOLDER" | split -b 4480M - "${FOLDER}.tar.gz."
}

# Function to search a specified folder for a given pattern
grep_designated_folder() {
   find "$1" -type f -print0 | xargs -0 grep -i "$2" 1> tmp1 2> tmp2
}

• compress_folder: Accepts a folder name as an argument and compresses it.
• grep_designated_folder: Searches a specified folder for a given pattern.

D-3) My Own .bashrc or .zprofile

alias gonginx="cd /opt/homebrew/etc/nginx/"
alias gohttp="cd /opt/homebrew/var/www/"
alias mm="make clean && make"
alias kprint="sudo journalctl -k -f"
    
cd /opt/homebrew/var/www    

function tar_backup_prototype() {
    cd /opt/homebrew/var || return
    filename="WEB$(date +"%Y%m%d")"
    tar -zcvf "${filename}.tar.gz" www/
    cd /opt/homebrew/var/www || return    
}

function tar_backup() {
    cd /opt/homebrew/var || return

    # Initial base filename
    base_filename="WEB$(date +"%Y%m%d")"
    filename="${base_filename}.tar.gz"

    # Check if the filename already exists, and append a counter if necessary
    counter=1
    while [ -e "$filename" ]; do
        filename="${base_filename}_${counter}.tar.gz"
        counter=$((counter + 1))
    done

    # Create the tar archive with the unique filename
    tar -zcvf "$filename" www/

    # Return to the specified directory
    cd /opt/homebrew/var/www || return
}
    

function tar_web() {
    # Check if a filename argument is provided
    if [ -z "$1" ]; then
        echo "Usage: tar_web "
        return 1  # Exit the function with a non-zero status
    fi

    # Navigate to the specified directory or exit if it fails
    cd /opt/homebrew/var || return

    # Create the tar.gz archive with the provided filename
    tar -zcvf "${1}.tar.gz" www/

    # Navigate back to the www directory or exit if it fails
    cd /opt/homebrew/var/www || return
}

function scp_backup_today() {
    scp "ngene.org:/opt/homebrew/var/WEB$(date +"%Y%m%d")*.tar.gz" ~/Desktop/
}

function scp_backup_all() {
    scp "ngene.org:/opt/homebrew/var/*.tar.gz" ~/Desktop/
}       

scp2web() {
  local filename="$1"
  scp "${filename}"* ngene.org:/opt/homebrew/var/www/
}

scp2src() {
  local filename="$1"
  scp "${filename}"* ngene.org:/opt/homebrew/var/www/src/
}

scp2src_wild() {
  if [[ $# -eq 0 ]]; then
    echo "Usage: scp2src_wild PATTERN..."
    return 1
  fi

  local files=()

  # Collect matches for all patterns given
  for pattern in "$@"; do
    matches=($pattern)   # zsh expands wildcards here
    if [[ ${#matches[@]} -eq 0 ]]; then
      echo "No files found matching: $pattern"
    else
      files+=("${matches[@]}")
    fi
  done

  if [[ ${#files[@]} -eq 0 ]]; then
    echo "No files to upload."
    return 1
  fi

  echo "Uploading files: ${files[*]}"
  scp "${files[@]}" ngene.org:/opt/homebrew/var/www/src/
}

eval "$(/opt/homebrew/bin/brew shellenv)"
export PATH="/opt/homebrew/sbin:$PATH"
    
alias zprofile_change='emacs ~/.zprofile'
alias zprofile_apply='source ~/.zprofile'

# Function to search for a specific term within files under a specified directory, case-insensitive.
function file_grep() {
    # Check if both search term and search path are provided
    if [[ -z "$1" || -z "$2" ]]; then
        echo "Usage: file_grep  "
        echo "Example: file_grep \"nginx\" /opt/homebrew"
        return 1
    fi

    # Assign arguments to variables for clarity
    search_term="$1"
    search_path="$2"

    # Execute the search command
    sudo find "$search_path" -type f -print0 | xargs -0 grep -i "$search_term"
}

# Function to search files by regex in a specified directory (case-insensitive)
function find_re() {
    # Display usage instructions if arguments are missing
    if [[ -z "$1" || -z "$2" ]]; then
        echo "Usage: find_re  "
        echo "Example: find_re /opt/homebrew '.*frank.*'"
        return 1
    fi

    # Assign arguments to variables for readability
    search_path="$1"
    regex_pattern="$2"

    # Execute the find command with case-insensitive regex
    find "$search_path" -type f -iregex "$regex_pattern"
}

# Function to search for files with a case-insensitive substring match in the filename
function find_str() {
    # Display usage instructions if arguments are missing
    if [[ -z "$1" || -z "$2" ]]; then
        echo "Usage: find_str  "
        echo "Example: find_str /opt/homebrew '###'"
        return 1
    fi

    # Assign arguments to variables for clarity
    search_path="$1"
    search_text="$2"

    # Execute the find command with case-insensitive name matching
    find "$search_path" -type f -iname "*$search_text*"
}

##################################
alias emacs="emacs -nw"

(E) Using Environment Variables, Aliases, and Functions

Once the environment variables, aliases, or functions are set in the shell's configuration file, they become available in every new terminal session.

Using the compress_folder Function: To compress and split a folder named folder_name, run:

# Compress and split a folder
compress_folder folder_name

Automatic Environment Variables: The PYTHONPATH variable will be automatically set upon opening a new terminal, allowing Python to locate additional libraries specified in the path.

Efficient Linux at the command line by Daniel J. Barrett (Written November 12, 2025)

1. Basic Text Processing Commands

   Linux provides several commands to inspect and manipulate text data. Some of the most frequently used are wc, head, cut, grep, sort, and uniq. These tools can be combined in pipelines to filter and process textual information efficiently.

   1. wc – Word, Line, and Byte Count

      The wc command ("word count") counts lines, words, and bytes (characters) in files or input. It is often used to get a quick size estimate of a file or output.

      • Count lines: wc -l filename (use -l to count lines only).
      • Count words: wc -w filename (use -w to count words).
      • Count characters: wc -c filename (use -c for byte count).

      For example, wc -l /var/log/syslog will output the number of lines in the system log file.

   2. head – View Beginning of Files

      The head command displays the first lines of a file. By default, it shows the first 10 lines. It is useful for previewing large files or outputs.

      • Show first N lines: head -n 20 filename will display the first 20 lines.
      • Combine with other commands: You can pipe output into head to preview results, e.g. ps aux | head to see the top of a process list.

      $ head -n 3 sample.txt
      Line 1 of the file
      Line 2 of the file
      Line 3 of the file

   3. cut – Extract Columns of Text

      cut is used to extract columns or fields from lines of text, especially when data is structured with a delimiter (such as CSV or log files). It can cut by bytes, characters, or fields separated by a delimiter.

      • Cut by delimiter: cut -d ':' -f1,3 file.txt splits each line by ':' and extracts the 1st and 3rd fields.
      • Cut by character position: cut -c 1-10 file.txt extracts the first 10 characters of each line.

      $ echo "user:password:uid:gid" | cut -d ':' -f1,4
      user:gid

   4. grep – Search Text with Patterns

      grep finds lines in text that match a given pattern. It is a powerful search tool supporting plain text searches as well as regular expressions. By default, grep prints the matching lines.

      • Case-insensitive search: grep -i "error" logfile.txt matches “error” in any case (Error, ERROR, etc.).
      • Line numbers: grep -n "main" source.c shows matching lines with line numbers.
      • Invert match: grep -v "^#" filters out lines starting with # (common to skip comments).

      $ grep -n "root" /etc/passwd
      1:root:x:0:0:root:/root:/bin/bash

   5. sort – Sort Text Lines

      The sort command sorts lines of text alphabetically or numerically. It’s often used to organize data or prepare for other operations (like finding duplicates).

      • Numeric sort: sort -n numbers.txt treats the content as numbers (so 2 comes before 10).
      • Reverse sort: sort -r outputs results in descending order.
      • Sort by column: sort -k 2 file.txt sorts by the 2nd field (default delimiter is whitespace).

   6. uniq – Uniquify and Count Duplicates

      uniq filters out adjacent duplicate lines in a sorted file (or stream). It is commonly used after sort to count or remove duplicates.

      • Basic usage: sort file.txt | uniq will output each line once (all duplicates collapsed).
      • Count occurrences: uniq -c prefixes each unique line with the number of occurrences. (Use sort first, since uniq only collapses consecutive duplicates.)
      • Only duplicates: uniq -d shows only lines that were duplicated.

      $ sort items.txt | uniq -c
           3 apple
           1 banana
           2 orange

2. Finding Duplicate Files with md5sum and Pipelines

   Identifying duplicate files can be automated using cryptographic hashes. The md5sum tool computes MD5 hashes of files, which can be used as unique fingerprints. By hashing files and then sorting or counting identical hashes, duplicates can be found.

   Using md5sum with sorting and uniq: One approach is to generate an MD5 for each file and sort the results so that identical hashes line up consecutively, then use uniq to find duplicates. For example:

   $ md5sum *.jpg | sort | uniq -w32 -d
   

   In this command, md5sum *.jpg prints lines with the hash and filename for each .jpg file. We sort them (so identical hashes are adjacent) and then uniq -w32 -d checks for duplicates considering only the first 32 characters (the MD5 hash), printing lines that are duplicated (i.e., files with the same hash). This quickly lists files that have the same content.

   Using awk for detailed duplicate reports: For more detail, awk can aggregate file names by their hash:

   $ md5sum *.jpg | awk '{ hash=$1; file=$2; files[hash] = files[hash] file " "; count[hash]++; } END { for(h in files) if(count[h] > 1) { print count[h] " duplicates of hash " h ": " files[h]; } }' | sort -nr
   

   This one-liner computes all hashes, builds a list of files for each hash, and at the end prints out any hash that appeared more than once, along with the count and file names. The output is sorted numerically reverse (sort -nr) to show the largest groups of duplicates first.

   Using checksums is much faster than manually comparing files byte by byte, and it helps to detect duplicates even if file names differ.

3. Environment Variables and printenv

   Environment variables are dynamic values that affect the processes and behavior of the shell and system. They include settings like PATH (which lists directories to search for executables), HOME (the user’s home directory), and many others. They can be viewed and set in the shell.

   • Viewing all variables: The printenv command prints all environment variables (or a specific one if you provide a name). Similarly, env without arguments also displays the environment.
   • Viewing a specific variable: Use echo $VARIABLE or printenv VARIABLE. For example, echo $HOME prints your home directory path.
   • Setting a variable: Use the export command to set and export to the environment, e.g. export EDITOR=nano (so that $EDITOR is now “nano”).

   $ printenv | head -n 3
   HOME=/home/alice
   LANG=en_US.UTF-8
   PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

   Environment variables can influence program behavior. For example, a program might check $LANG to determine language, or $TZ for timezone. The env command can also run a command under a modified environment, e.g. env TZ=UTC date will run the date command with the timezone set to UTC (without changing the global environment).

4. Aliases and the .bashrc File

   Frequently used or long commands can be turned into shortcuts using aliases. An alias is like a custom command that expands into a longer command. Aliases are usually defined in the shell startup file (like ~/.bashrc for Bash) so they are available in every session.

   • Defining an alias: The syntax is alias name='command'. For example, alias ll='ls -l --color' creates an alias ll that lists directory contents in long format with color.
   • Making it permanent: Adding the alias definition to ~/.bashrc (or ~/.bash_profile on some systems) ensures it is set for each new shell. You can edit the file and append such lines.
   • Activating changes: After editing .bashrc, new terminals will pick up the changes. To apply them immediately in the current shell, use the source command (or . which is a synonym). For example, source ~/.bashrc reloads the configuration without logging out and back in.

   $ alias gs='git status'
   $ gs
   On branch main
   ...

   In the above example, an alias gs is set to run git status. After defining it (or after reloading your config), typing gs will execute git status.

   Aliases are a simple way to save keystrokes. For more complex logic (like multiple commands or conditional behavior), a shell function or script might be more appropriate.

5. Finding Files and Executables

   There are multiple commands to locate files and identify executables in your system. Depending on what you need (to find an executable’s path, or search for files by name or content), you might use different tools like which, type, find, grep, and xargs.

   1. Locating Commands: which vs type

      The which command searches your PATH for an executable file and prints the first match. For example, which python might output /usr/bin/python. It is an external command and only knows about executables in the PATH.

      Bash’s built-in type command is more powerful: it tells you how a given name would be interpreted by the shell. It can identify if a command is built-in, an alias, a function, or an external executable. For instance, if ls is an alias, type ls would show the alias definition, whereas which ls would just show the path to the ls executable.

      $ which ls
      /bin/ls
      $ type ls
      ls is aliased to `ls --color=auto`

      In the above example, which finds the ls binary’s location, while type reveals that ls is actually an alias (which eventually calls the binary with options). If you define custom aliases or functions, which won’t show them, but type will.

   2. Searching for Files by Name: find

      The find command performs a recursive search through directories. It is extremely flexible, with options to filter by name, type, size, modification time, and more. Basic usage requires a path to search in (like /etc or . for current directory) and a condition.

      • By name: find /etc -name "*.conf" finds all files under /etc ending with .conf.
      • By type: find . -type d -name "test*" finds directories (-type d) in current folder matching “test*”. Use -type f for files.
      • Depth control: find . -maxdepth 2 -name "*.txt" limits how deep to search in subdirectories.

      $ find /etc -type f -name "*.conf"
       /etc/nginx/nginx.conf
       /etc/ssh/sshd_config
       ...

      If file names contain spaces or special characters, find will handle them safely (printing each path on a separate line). You can combine multiple conditions as well, for example finding files modified in the last day and matching a name pattern.

   3. Running Commands on Found Files: -exec

      find not only locates files; it can also perform actions on them using the -exec option. This allows you to execute a command for each file found. In the -exec syntax, {} is replaced by the current file name, and it ends with either \; or +.

      • Listing found files: find /var/log -name "*.log" -exec ls -l {} \; will ls -l each log file found.
      • Safe placeholder: The {} is a placeholder for the file path. It’s often wrapped in quotes to handle spaces, as in "{}", though in many cases find handles it.
      • Dry-run with echo: Before running a potentially destructive command, you can substitute the command with echo to preview. For example, find ~/tmp -name "*~" -exec echo rm {} \; will output lines like rm /home/alice/tmp/file~ without actually deleting anything.

      $ find ~/tmp -type f -name "*~" -exec echo rm {} \;
      rm /home/alice/tmp/doc1.txt~
      rm /home/alice/tmp/image.png~

      After verifying the list, you could run the actual remove by replacing echo rm with rm in the command above. The -exec action executes once per file match. If you use -exec ... {} + (note the plus sign), find will try to pass multiple files at once to the command for efficiency.

   4. Finding Text in Files: grep and xargs

      To search for text inside files across directories, you have a couple of approaches:

      • grep -R (recursive grep): grep -R "search text" /path/to/dir will recursively search files under the given directory for the pattern, printing matches. This is a quick one-command solution for finding text in files.
      • Using find and xargs: Another approach is to use find to list files, and then pipe the list to xargs which runs grep on batches of those files. xargs takes input (like filenames) from standard input and appends them as arguments to the given command. For example:

      $ find . -name "*.py" -print0 | xargs -0 grep -H "TODO"
      ./app/main.py:    # TODO: improve error handling
      ./app/utils.py:    # TODO: optimize this function

      Here, find lists all .py files in the current directory and below. -print0 causes find to separate file names with a null character (instead of newline), and xargs -0 reads those safely (this handles file names with spaces). The -H option in grep forces it to always print the filename before matches. The result is all occurrences of "TODO" in Python files, with their file paths.

      The combination of find and xargs is powerful for cases where grep -R might not be available or if you want to perform other operations on the found files. Note that xargs will handle long lists by splitting into multiple command invocations if necessary. In most modern Linux systems, grep -R is the simplest method for searching within files.

6. Shell Command History and Reuse

   The Bash shell keeps a history of commands that have been entered. This history allows quick access to previously run commands, making it easy to repeat or modify them without retyping. There are also settings to control how many commands to remember and how they are stored.

   History Size and Configuration

   Bash by default remembers a certain number of commands (often 500). Two important environment variables control this:

   • HISTSIZE: the number of commands to keep in memory for the current session’s history.
   • HISTFILESIZE: the maximum number of lines to store in the ~/.bash_history file (which persists across sessions).

   By increasing these values in your ~/.bashrc, you can have a larger history. For example, adding HISTSIZE=10000 and HISTFILESIZE=20000 will allow up to 10k commands in the session history and 20k in the file.

   Another useful variable is HISTCONTROL, which can be set to ignoredups (to not record duplicate commands consecutively) or ignorespace (to not record lines starting with a space, allowing you to omit sensitive commands by prefixing a space).

   History Expansion Shortcuts

   Bash provides “history expansion” shortcuts using the ! (exclamation mark) for quickly referring to commands in the history:

   • !! – Repeat the last command. (This is equivalent to !-1.)
   • !n – Run command number n from your history list (as shown by history or history | tail). For example, !1203 runs the command with index 1203.
   • !-n – Run the command from n lines back. For example, !-3 executes the third from last command. Adding :p at the end (e.g. !-3:p) will print that command instead of executing it.
   • !string – Run the last command that begins with string. For instance, !grep runs the most recent command that started with “grep”.
   • !?string? – Run the last command that contains string anywhere in it. Example: !?ssh? would find the last command that had “ssh” in it (whether at the beginning or middle).
   • ^old^new – This is a quick substitution in the previous command. It finds the first occurrence of “old” in the last command and replaces it with “new”, then executes that command. For example, if you typed ls *.jp by mistake, entering ^jp^jpg will run ls *.jpg.

   These shortcuts can save a lot of time. For example, !! is often used after running a command that needed sudo: if you get a permission error, you can just do sudo !! to rerun the last command with sudo.

   You can search your history interactively as well by pressing Ctrl-R and typing a part of a command; Bash will autofill the most recent match, which you can then execute or edit.

7. Command Line Editing Mode (Emacs vs Vi)

   The Bash shell has two editing modes for the command line: Emacs mode and Vi mode. By default, Bash uses Emacs keybindings, which allow familiar cursor movement and editing shortcuts similar to the Emacs text editor (or common shortcuts in many apps).

   • Emacs mode: Enabled by default (or with set -o emacs). It allows quick editing via keys like Ctrl-A (move to start of line), Ctrl-E (move to end of line), Alt-B (backward one word), Alt-F (forward one word), Ctrl-K (cut to end of line), Ctrl-Y (yank/paste the cut text), and Ctrl-R (search history backward). These shortcuts make it efficient to fix or rerun commands.
   • Vi mode: Can be enabled with set -o vi for those who prefer Vi/Vim keybindings. In Vi mode, you press Esc to toggle between insert mode and command mode, and use h, j, k, l for navigation, b, w for word moves, etc., similar to editing text in Vi.

   These modes affect only how you edit the command line, not how commands execute. Emacs mode is generally easier for beginners (with arrow keys and Ctrl shortcuts working), whereas Vi mode is appreciated by those with Vi experience. You can choose the one that best fits your workflow and muscle memory.

8. Custom Shortcuts for Changing Directories

   Navigating deep directory structures can be time-consuming. Bash allows creation of shell functions to create smart shortcuts. One example is a function that quickly changes directory based on a keyword. Consider this function defined in your .bashrc:

   qcd() {
       case "$1" in
           work) cd /home/alice/projects/work ;;
           recipes) cd /home/alice/Documents/Recipes ;;
           *) echo "Unknown shortcut: $1" ;;
       esac
       pwd
   }
   

   This qcd function takes one argument and switches to a preset directory depending on the keyword. If qcd work is run, it jumps to /home/alice/projects/work; qcd recipes goes to /home/alice/Documents/Recipes. After changing directory, it prints the current directory (via pwd) as confirmation.

   To make it even more convenient, bash completion can be set up for this function. For example:

   complete -W "work recipes" qcd

   This line, when added after the function, tells Bash that valid completions for qcd are “work” and “recipes”. So you can type qcd w and press TAB to auto-complete “work”.

   Custom directory shortcuts like these can save a lot of typing. Another built-in tip for quick navigation is the CDPATH variable – you can list base directories in CDPATH, and then you can cd into subdirectories from anywhere. But defining explicit functions or aliases like qcd gives you more control over naming and behavior.

9. Toggling Between Directories with cd -

   Bash provides a built-in shorthand to jump to the previous working directory: cd -. This command toggles you back to the last directory you were in, which is very handy when switching between two locations frequently.

   $ pwd
   /home/alice/projects
   $ cd /var/www/html
   $ pwd
   /var/www/html
   $ cd -
   /home/alice/projects
   

   In this example, the user was in /home/alice/projects, then went to /var/www/html. Executing cd - returns them to /home/alice/projects, and also prints that path. Running cd - again would flip back to /var/www/html, and so on.

   This works because Bash remembers the previous directory in the variable OLDPWD, and cd - effectively does cd "$OLDPWD" and then prints it. It's a quick way to bounce between two directories without retyping the paths.

10. Other Handy Commands: date, seq, yes, and Advanced grep

    Beyond the basic text tools, here are a few more commands that prove useful in scripting and automation:

    date – Display or Format Dates

    The date command prints the current date and time, or can format dates in various ways. This is helpful for timestamps in scripts or logs.

    • Default output: Running date with no arguments shows the current date/time in a standard format.
    • Custom format: date "+%Y-%m-%d %H:%M:%S" outputs the date as YYYY-MM-DD HH:MM:SS. The + sign indicates a format string, where %Y is year, %m month, etc. You can compose many variations (e.g., %A for weekday name).
    • Using in scripts: You might use $(date +%s) to get the current Unix timestamp (seconds since 1970) or include a date in a filename (e.g., backup_$(date +%Y%m%d).tar.gz).

    seq – Generate Sequences of Numbers

    seq outputs a sequence of numbers, one per line (by default). It’s handy for generating loops or numbering things.

    • Basic sequence: seq 5 will output 1, 2, 3, 4, 5 on separate lines.
    • With step and range: seq 2 2 10 starts at 2, steps by 2, and goes up to 10 (printing 2, 4, 6, 8, 10).
    • Fixed width: seq -w 1 3 prints 01, 02, 03 – the -w option pads numbers with leading zeros to equal width.

    Sequences often complement other commands. For example, generating numbered filenames or running a loop in a shell script by iterating over $(seq N).

    yes – Automatic Confirmation or Repetition

    The yes command outputs a string repeatedly (by default “y” followed by a newline, if no string is provided). Its primary use is to feed a continuous stream of “yes” (or any text) into programs that prompt for input, effectively auto-confirming prompts.

    • Auto-confirm prompts: If a command asks “Are you sure? (y/n)”, you can do yes | command to send it a stream of "y" answers. For example, yes | apt-get install -y package is redundant (since -y already auto-confirms in apt-get), but demonstrates the concept.
    • Custom response: You can specify an alternate string: yes no | command would send "no" repeatedly. Use this carefully – it will continue until the receiving program terminates or the stream is closed.
    • Load testing or dummy data: yes can quickly generate a large volume of data. For instance, yes X | head -n 1000 > file.txt will produce a file with 1000 lines of "X". Remember to stop yes with Ctrl-C if you run it without a terminating pipe or limit, as it will otherwise run indefinitely.

    Advanced grep Techniques

    We introduced grep earlier for basic searches. Here are a few more useful options and patterns for powerful text searching:

    • Recursive search: As noted, grep -R (or grep -r) searches through directories recursively. Add -n to see line numbers and -H to always show filenames.
    • Context lines: grep -A 2 -B 2 "ERROR" logfile will show 2 lines After and 2 lines Before each "ERROR" match, providing context around matches (-C is shorthand for same number before & after).
    • Counting matches: grep -c "pattern" file returns just the number of matching lines (useful for quick stats).
    • Regex and extended patterns: grep -E "fail|error|critical" enables extended regular expressions (alternation in this case to match “fail” or “error” or “critical”). grep -P can even use Perl-compatible regex for advanced needs.
    • Only filenames: grep -l "text" *.md lists just the filenames of files that contain the text (useful to find which files have a certain string).

    With these options, grep becomes not only a finder of exact lines, but also a context-aware search tool and even a rudimentary file content scanner across directories.

11. Less-Known but Useful Text Utilities

    Beyond the commonly used commands, there are a number of other text processing utilities that can be very handy in specific situations:

    • tac – Concatenate and print files in reverse. (tac is essentially cat backwards.) It reads a file from bottom to top, outputting the last line first. For example, tac file.txt will print the file with lines in reverse order.
    • paste – Merge lines of files. It takes lines from multiple files and pastes them side by side separated by tabs (by default). For instance, if you have one file with names and another with scores, paste names.txt scores.txt will produce a two-column output combining corresponding lines.
    • diff – Compare files line by line. It outputs the differences between two files. By default, diff file1 file2 produces a set of instructions to change file1 into file2. Using -u gives unified diff format (popular for patches), and tools like colordiff or graphical diff viewers can make it easier to interpret. diff -r can compare directories recursively.
    • tr – Translate or delete characters. It reads from standard input and writes to standard output, replacing characters as specified. For example, tr 'A-Z' 'a-z' will uppercase input text (it replaces lowercase letters with their uppercase counterparts). You can also use tr -d to delete characters (e.g., tr -d '\r' to remove carriage returns from Windows text files).
    • rev – Reverse the characters in each line. It’s the character-wise counterpart to tac. Running rev on a line like “Hello” outputs “olleH”. This can be useful for certain text processing tasks or just for quirky uses (like reversing a string quickly in shell).

    Each of these commands solves a particular kind of problem. While they might not be used every day, knowing them means you won’t have to resort to more complex tools or code when a simple pipeline using these would do the job.

12. Text Processing Power Tools: awk and sed

    awk and sed are two classic Unix utilities for processing text streams. They are very powerful, but even basic uses of them can save time.

    sed – Stream Editor

    sed is a non-interactive editor that reads input (from files or standard input), transforms it, and outputs the result. It is commonly used for substitution (find and replace) or deleting lines.

    • Substitute text: sed 's/old/new/g' file.txt will replace all occurrences of “old” with “new” in each line of file.txt. (The g at the end means “global” replacement on the line; without it, only the first occurrence on each line is replaced.) This outputs the result to standard output; add -i for in-place editing of the file (be careful with in-place edits, consider backing up first).
    • Delete lines: sed '/^$/d' file.txt will delete all empty lines (lines that match the regex ^$ for start and end with nothing between). Or use sed '5,10d' to delete lines 5 through 10, for example.
    • Selective printing: By default sed prints all lines (after applying edits). You can suppress automatic printing with -n and use p to print only certain lines. For example, sed -n '1,3p' file.txt prints only lines 1 through 3 of the file.

    awk – Pattern Scanning and Processing

    awk is a programming language designed for text processing and typically used as a single-command tool. It operates on each line of input (or record) and can perform actions on lines that match patterns.

    • Field processing: awk splits each line into fields (by whitespace by default, or by a custom delimiter using -F). Fields are referenced by $1, $2, etc. For example, awk '{print $2, $1}' names.txt will print the second field, then the first field from each line of names.txt, effectively swapping two columns.
    • Conditionals: You can put a condition before the { ... } action block. For instance, awk '$3 > 100 {print $1,$2}' data.txt would print the first two fields of lines where the third field is greater than 100. If you have a CSV with columns, this is a quick way to filter by a numeric value in a certain column.
    • Aggregation: awk has the concept of BEGIN and END blocks for actions before reading input and after processing all input. It also retains variables. For example, awk '{sum += $1} END {print sum}' numbers.txt will calculate and print the sum of the first field of all lines (assuming a file of numbers).

    Both sed and awk can become very complex (each has entire books dedicated to them). But even using them for simple tasks as above can replace more cumbersome manual steps or multiple commands. They truly shine when you need to do text manipulation that goes beyond what grep or cut can do alone.

13. Combining Commands with &&, ||, and ;

    In bash (and many shells), you can string commands together on one line or in a script using control operators. The three most common operators are &&, ||, and the semicolon ;. They allow conditional or sequential execution of commands:

    • command1 && command2 – Executes command2 only if command1 succeeded (i.e., returned an exit status of 0). This is useful for chaining dependent commands. Example: mkdir newdir && cd newdir (change directory only if folder creation succeeded).
    • command1 || command2 – Executes command2 only if command1 failed (non-zero exit status). Example: grep "pattern" file || echo "Pattern not found". Here, if the grep does not find the pattern (and thus exits with status 1), the echo will run; otherwise, if grep succeeds, the echo is skipped.
    • command1; command2 – Executes command1 and then command2 regardless of success or failure. Commands separated by ; always run sequentially, effectively ignoring exit status. Example: echo "Starting update"; sudo apt update (the message will echo, then apt update runs after).

    Using && and || can make shell scripts and one-liners more robust by handling success/failure paths succinctly. They function like logical AND and OR for command execution.

14. Command Substitution: Using $(...) to Embed Commands

    Command substitution allows the output of one command to be inserted into the arguments of another command. In Bash, this is done with the syntax $( ... ). (Older shell scripts may use backticks `...`, but $(...) is preferred for its nesting capabilities and clarity.)

    For example, suppose you have a bunch of .txt files and you want to move those that contain the word "Kansas" to a folder named "kansas". You can use grep to find which files contain the text, and then mv to move them:

    $ mv $(grep -l "Artist: Kansas" *.txt) kansas/

    Here’s what happens:

    • grep -l "Artist: Kansas" *.txt lists all .txt filenames in the current directory that contain the string "Artist: Kansas". The -l option makes grep output only the filenames of matching files, not the matching lines.
    • The $( ... ) around that command captures its output. So the shell replaces $(grep -l ...) with the list of files that grep found.
    • The mv command then sees something like mv file1.txt file2.txt file3.txt kansas/, which will move all those files into the kansas/ directory (which should already exist as a directory).

    Command substitution is extremely useful for feeding the result of one operation into another. Another quick example: echo "Today is $(date +%A)" will print something like "Today is Wednesday" by substituting the output of the date +%A command.

15. Process Substitution: Treating a Command’s Output Like a File

    Process substitution is a feature of Bash (and some other shells) that allows the output of a command to be referenced as if it were a file. The syntax uses <(command) for substituting a command’s output and >(command) (less common) for feeding input from a file into a command's input. It's often used with commands that normally compare or merge files.

    A classic example is using diff to compare outputs of two commands, rather than files. Normally, diff compares two files line by line. With process substitution, you can do:

    $ diff <(ls *.jpg | sort) <(seq -f "%g.jpg" 1 1000)

    In this command:

    • ls *.jpg | sort lists all .jpg files in the current directory and sorts them (maybe numerically or alphabetically). This is one “virtual file”.
    • seq -f "%g.jpg" 1 1000 generates a sequence of numbers from 1 to 1000 and formats them as “n.jpg” (using the -f format option). This produces lines like 1.jpg, 2.jpg, ... 1000.jpg. This is the second “virtual file”.
    • diff then compares the output of those two commands as if they were files. Essentially, it’s comparing the list of existing image files to the list of expected image filenames from 1.jpg to 1000.jpg. The diff output will show which files are missing or extra.

    The <( ) syntax causes the shell to run the commands inside and provide a temporary file name (like /dev/fd/63) that diff can read from. This trick can be used for many situations where a command insists on a file, but you want to supply dynamic content. Note that process substitution is not POSIX standard and is specific to Bash, Zsh, and a few other shells.

16. Executing Commands from a Pipe or File

    Normally, we run commands directly in the shell or by executing a script file. However, you can also send a command (or series of commands) into a shell via a pipeline or redirect a file into the shell. This is useful for automation or dynamically generating commands.

    For example, consider:

    $ echo "ls -l /home" | bash

    This will take the string "ls -l /home" and pipe it into the bash program. Bash will execute the commands it receives from its standard input. In this case, it will run ls -l /home and output the directory listing. Essentially, you are feeding a command to a new bash process via a pipe.

    Similarly, you could have a file commands.txt that contains a list of commands, and run bash < commands.txt to have bash execute all the commands from that file in sequence.

    Be cautious when using this technique, especially if the input is coming from an untrusted source, because you are effectively executing whatever commands are provided. However, it can be very powerful when used to automate tasks. One use case is generating a sequence of commands via script and piping them into sh or bash to execute, instead of writing them to a temporary file.

17. Running Commands in the Background

    On a Linux terminal, appending an ampersand & to a command will run it in the background. This means you get the prompt back immediately and can continue doing other work while the command executes. Background jobs are useful for long-running processes.

    For example:

    $ long_running_script.sh &
    [1] 4821

    When you run this, you get a job number (in brackets) and a process ID. Here, [1] is the job number, and 4821 is the PID of the background process. The shell prompt returns right away.

    You can manage and monitor background jobs with a few built-in commands:

    • jobs – Lists current background jobs with their job number and status (Running, Stopped, etc.).
    • fg %1 – Bring job number 1 back to the foreground. (Using %jobnumber references a job; fg without arguments brings the most recent job to foreground.)
    • bg %1 – If a job is stopped or in foreground, bg will resume it in the background.
    • kill – Send a signal to a process by PID or job ID. For example, kill 4821 will send SIGTERM to the process with PID 4821. You can also use kill %1 to kill job number 1. Use kill -9 (SIGKILL) if a process doesn’t terminate with a gentle kill.

    To check how far a background process has progressed, it depends on the process itself. Many command-line programs will output their progress periodically or upon completion. If the output is not redirected, you will see it appear in the terminal even while you have the prompt (the text may intermix with your typing). You can redirect output to a file (using > output.txt 2>&1) and then inspect that file or use tail -f output.txt to watch it live.

    Bash will notify you with a message when a background job finishes or stops (next time you press Enter). For instance, you might see [1]+ Done long_running_script.sh when job 1 completes. This is a handy reminder of background job completion.

    If you need a command to keep running even after you log out, consider using nohup (no hang-up) or running the process inside a terminal multiplexer like screen or tmux. nohup command & will make the process immune to hangup signals (so it won't terminate when your session ends), and it writes output to nohup.out by default.

18. Using Subshells in Pipelines (Tar Archiving Examples)

    A subshell is a separate shell process launched by your current shell. Parentheses ( ... ) around commands cause them to execute in a subshell. One common use of subshells is to localize a change in directory or environment that shouldn’t affect the main shell. This is frequently seen in advanced shell one-liners, especially involving tar for archiving files.

    Consider these use cases:

    • Extracting an archive to a target directory: cat archive.tar.gz | (mkdir -p /tmp/other && cd /tmp/other && tar xzvf -) – This command creates a directory /tmp/other (if it doesn’t exist), then changes into it within a subshell, and extracts the archive from standard input (the - tells tar to read from stdin). The current shell’s directory is not affected by the cd because that happens in the subshell. After extraction, you’ll find the files in /tmp/other.
    • Copying a directory tree locally: tar czf - dir1 | (cd /tmp/dir2 && tar xzf -) – Here tar czf - dir1 creates a tarball of dir1 and sends it to stdout (the - as the output file means stdout). That output is piped into a subshell where we cd /tmp/dir2 and then tar xzf - to extract from stdin into /tmp/dir2. This effectively copies dir1 into /tmp/dir2 in one line without creating an intermediate file on disk.
    • Copying to a remote host: tar czf - dir1 | ssh user@remote "(cd /path/on/remote && tar xzf -)" – This variation uses ssh to run a subshell on the remote host. The local tar writes to stdout, which ssh sends to the remote shell’s stdin for the tar extraction. The remote subshell ensures the extraction happens in the desired directory on the remote machine. This is a convenient way to transfer directories between machines without intermediate files.

    These examples demonstrate the elegance of combining subshells with pipes. The subshell (enclosed in parentheses) can change directories or other settings temporarily. The && ensures that each step (like making the directory or changing directory) must succeed before proceeding to the next, adding safety. Using - with tar for stdin/stdout, along with ssh for remote execution, can accomplish in one line what might otherwise take several steps.

19. Subshells vs. Replacing the Shell (Process Replacement)

    When you run a command like bash from your current shell, you start a new subshell (a child process). Your prompt might change to indicate a new shell, and any changes or commands in that subshell won’t affect the parent shell once you exit it. For example:

    $ echo $PS1
    \u@\h:\w\$ 
    $ bash   # start a new Bash subshell
    $ PS1="Doomed> "   # change the prompt in subshell
    Doomed> echo "Hello"
    Hello
    Doomed> exit   # exit the subshell
    $ 
    

    In this snippet, the user’s normal prompt \u@\h:\w\$ (username@host:cwd$) is shown. After running bash, a new shell starts (which inherited the same prompt format by default). The user then changes PS1 (the prompt variable) to "Doomed> " inside the subshell, and we can see the prompt change. The echo command runs in the subshell and prints "Hello". After exiting the subshell, the original prompt and shell are restored.

    Sometimes you might want to not just start a subshell, but actually replace the current shell process with a new program. This can be done with the exec command. exec takes a command and runs it in place of the current shell, meaning the current shell will not continue after exec. For instance, if you run exec bash in your shell, you don't get a new nested shell – your current shell becomes the new bash process. When you exit that bash, the session ends (since there was no parent shell to return to).

    The exec command is often used in scripts when you want to hand over control to another program, or to replace a shell with a different one without spawning extra processes. It's also used to redirect file descriptors at a shell level (for advanced scripting scenarios).

    In summary, parentheses ( ... ) create a subshell for grouped commands, launching a new shell like bash gives you a child shell that you can exit, and exec replaces the current shell process entirely with a new program. Each has its use: subshells for isolation, new shells for temporary environments, and exec for full replacement.

20. Useful One-Liner Examples

    Throughout this guide, we've seen many one-liner commands – powerful combinations of tools strung together to perform complex tasks in one go. Mastering one-liners can significantly boost efficiency on the command line. Here are a few additional examples of handy bash one-liners:

    • Find the ten largest files in a directory tree: du -ah . | sort -h -r | head -n 10. This uses du (disk usage) to list sizes of all files and directories under the current directory (-a for all files, -h for human-readable sizes), then sorts them by size (-h sorts by human-readable sizes, -r for reverse order so largest first), and shows the top 10.
    • Recursively search for a string in all files: grep -R "needle" .. This will print all matches of "needle" in the current directory and subdirectories. Add -n for line numbers or -l to list only filenames. (Alternatively, rg "needle" if using the ripgrep tool for even faster recursive search.)
    • Count occurrences of a word in a file: grep -o "apple" file.txt | wc -l. grep -o will output each occurrence of "apple" on a new line, and wc -l counts the lines, giving the number of occurrences. This is a quick way to count matches (otherwise grep -c counts matching lines, which may be fewer if multiple matches per line).
    • Batch rename file extensions: for f in *.jpeg; do mv "$f" "${f%.jpeg}.jpg"; done. This loop goes through each .jpeg file in the directory and renames it to have a .jpg extension. The expression ${f%.jpeg}.jpg takes the filename stored in $f and replaces the .jpeg suffix with nothing (effectively dropping it) then appends .jpg.
    • Monitor a command's output in real time: watch -n 5 'df -h'. The watch command runs the given command at regular intervals (every 5 seconds here) and updates the display. In this example, it repeatedly shows disk usage (df -h) every 5 seconds. This isn't exactly a pipeline one-liner, but it's extremely useful for watching system status changes.

    By combining basic commands, redirections, and a bit of shell syntax, you can accomplish a lot without writing a full script. As you discover more commands and get comfortable with these patterns, you'll find yourself creating one-liners to solve problems on the fly. It's always a good practice to break down what a complex one-liner is doing (as we did above) to ensure you understand it and it's doing exactly what you intend.

Quick Reference Cheat Sheet

File and Text Processing

• wc -l file.txt – Print the number of lines in file.txt. (Use -w for words, -c for bytes.)
• head -n 20 file.txt – Show the first 20 lines of a file. (tail -n 20 shows the last 20 lines.)
• cut -d ',' -f1-3 data.csv – Extract the 1st through 3rd fields from a CSV (comma-separated) file.
• sort -r -n numbers.txt – Sort the lines in numeric order (-n) and reverse (-r) to get highest to lowest.
• uniq -c sorted.txt – Collapse duplicate lines in a sorted file, prefixing each unique line with its count (-c).
• tac file.txt – Output the file with lines in reverse order (last line first).
• paste file1.txt file2.txt – Combine two files line-by-line (joining lines with a tab by default).
• tr '[:upper:]' '[:lower:]' – Translate uppercase to lowercase from standard input (outputs to stdout). Also use tr -d to delete characters (e.g., tr -d '\n' to remove newlines).
• rev file.txt – Reverse each line of the file (characters in line reversed).
• diff -u old.txt new.txt – Compare two files and produce a unified diff (prefixed with +/– to show changes).

Finding and Searching

• which command – Show the path of the executable that would run for "command". (E.g., which python)
• type command – Show how "command" is interpreted (alias, function, built-in, or file path).
• find /path -type f -name "*.log" – Recursively find files under /path that match the name pattern *.log. Use -type d for directories.
• find . -size +100M -exec ls -lh {} \; – Find files larger than 100MB in current directory and list them (using ls -lh). The \; ends the -exec clause.
• find . -name "*.tmp" -exec rm {} \; – Find and remove (rm) all files ending with .tmp in current directory tree. (Use -exec echo rm ... first to verify.)
• grep -R "foo" /path – Recursively search for "foo" in all files under /path. Add -n to show line numbers, -i for ignore case.
• grep -v "^#" – From input, output all lines that do not start with # (useful for ignoring comment lines).
• xargs – Construct argument lists and execute utility. Often used with find: e.g., find . -type f -print0 | xargs -0 grep "hello" to search "hello" in all files found (null-separated safe).

Shell and Environment

• printenv – Print all environment variables (or printenv VAR for a specific variable).
• export VAR=value – Set an environment variable in the current shell (and sub-processes). E.g., export EDITOR=vim.
• alias ll='ls -la' – Create an alias ll for a command (ls -la in this case). Put aliases in ~/.bashrc to make them permanent.
• source ~/.bashrc (or . ~/.bashrc) – Apply changes from the bashrc file to the current session (reload the config).
• HISTSIZE, HISTFILESIZE – Variables controlling how many commands to remember in history (in-memory and in the ~/.bash_history file respectively).
• HISTCONTROL=ignoredups:ignorespace – Setting to ignore consecutive duplicate commands and those starting with space in history.
• Ctrl-R – (interactive) Reverse search through command history. Start typing to find a past command.
• set -o vi / set -o emacs – Switch shell editing mode to Vi or Emacs style. (Emacs is default; Vi gives Vim-like command line editing.)
• cd - – Switch to the previous directory (toggle back and forth between two directories).
• pushd dir / popd – Navigate directories using a stack. pushd changes directory like cd but pushes the old directory onto a stack; popd goes back by popping from that stack. Useful for quickly jumping around and back.

Command Combinations and Scripting

• command1 && command2 – Run command2 only if command1 succeeds.
• command1 || command2 – Run command2 only if command1 fails.
• command1; command2 – Run command1 and then command2 regardless of success/failure.
• $(command) – Command substitution: use the output of command in place. E.g., echo "Files: $(ls | wc -l)".
• diff <(cmd1) <(cmd2) – Process substitution: treat outputs of cmd1 and cmd2 as files to diff (or similarly, feed into any command that expects file arguments).
• echo "command" | bash – Pipe a dynamically generated command or script into a new bash shell to execute it.
• cmd & – Append & to run cmd in background. Use jobs to list jobs, fg/%n to bring to foreground, kill to terminate if needed.
• nohup cmd & – Run cmd immune to hangup (logout), in background. Output is redirected to nohup.out by default.
• (cd /tmp && do_something) – Perform do_something in a subshell where current directory is /tmp. The enclosing shell’s directory isn’t affected.
• exec program – Replace the current shell with program (the shell process ends, and program runs in its place).
• for i in {1..5}; do ...; done – Loop from 1 to 5 (inclusive). Inside the loop, use $i for the current number. Useful for simple iterations in shell.

Written on November 12, 2025

Efficient macOS Command-Line Techniques (Written November 12, 2025)

1. Basic Text Processing Commands

   macOS (like other Unix systems) provides several commands to inspect and manipulate text data. Some of the most frequently used are wc, head, cut, grep, sort, and uniq. These tools can be combined in pipelines to filter and process textual information efficiently.

   1. wc – Word, Line, and Byte Count

      The wc command (“word count”) counts lines, words, and bytes (characters) in files or input. It is often used to get a quick size estimate of a file or output.

      • Count lines: wc -l filename (use -l to count lines only).
      • Count words: wc -w filename (use -w to count words).
      • Count characters: wc -c filename (use -c for byte count).

      For example, wc -l /var/log/system.log will output the number of lines in the system log file.

   2. head – View Beginning of Files

      The head command displays the first lines of a file. By default, it shows the first 10 lines. It is useful for previewing large files or outputs.

      • Show first N lines: head -n 20 filename will display the first 20 lines.
      • Combine with other commands: It is common to pipe output into head to preview results. For example, ps aux | head will show the top of a process list.

      $ head -n 3 sample.txt
      Line 1 of the file
      Line 2 of the file
      Line 3 of the file

   3. cut – Extract Columns of Text

      cut is used to extract columns or fields from lines of text, especially when data is structured with a delimiter (such as CSV or log files). It can cut by bytes, characters, or fields separated by a delimiter.

      • Cut by delimiter: cut -d ':' -f1,3 file.txt splits each line by : and extracts the 1st and 3rd fields.
      • Cut by character position: cut -c 1-10 file.txt extracts the first 10 characters of each line.

      $ echo "user:password:uid:gid" | cut -d ':' -f1,4
      user:gid

   4. grep – Search Text with Patterns

      grep finds lines in text that match a given pattern. It is a powerful search tool supporting plain text searches as well as regular expressions. By default, grep prints the matching lines.

      • Case-insensitive search: grep -i "error" logfile.txt matches “error” in any case (error, ERROR, etc.).
      • Line numbers: grep -n "main" source.c shows matching lines with line numbers.
      • Invert match: grep -v "^#" filters out lines that start with # (commonly used to skip comments).

      $ grep -n "root" /etc/passwd
      1:root:x:0:0:root:/root:/bin/bash

   5. sort – Sort Text Lines

      The sort command sorts lines of text alphabetically or numerically. It’s often used to organize data or prepare for other operations (like finding duplicates).

      • Numeric sort: sort -n numbers.txt treats the content as numbers (so 2 comes before 10).
      • Reverse sort: sort -r outputs results in descending order.
      • Sort by column: sort -k 2 file.txt sorts by the 2nd field (default delimiter is whitespace).

   6. uniq – Uniquify and Count Duplicates

      uniq filters out adjacent duplicate lines in a sorted file (or stream). It is commonly used after sort to count or remove duplicates.

      • Basic usage: sort file.txt | uniq will output each line once (all duplicates collapsed).
      • Count occurrences: uniq -c prefixes each unique line with the number of occurrences. (Use sort first, since uniq only collapses consecutive duplicates.)
      • Only duplicates: uniq -d shows only lines that were duplicated.

      $ sort items.txt | uniq -c
           3 apple
           1 banana
           2 orange

2. Finding Duplicate Files with md5sum and Pipelines

   Identifying duplicate files can be automated using cryptographic hashes. The md5sum tool computes MD5 hashes of files, which can serve as unique fingerprints. By hashing files and then sorting or counting identical hashes, duplicates can be detected.

   Using md5sum with sorting and uniq: One approach is to generate an MD5 for each file and sort the results so that identical hashes line up consecutively, then use uniq to find duplicates. For example:

   $ md5sum *.jpg | sort | uniq -w32 -d
   

   In this command, md5sum *.jpg prints lines with the hash and filename for each .jpg file. The output is piped into sort (so identical hashes are adjacent), and then uniq -w32 -d prints only the hashes (and associated line) that are duplicated (considering only the first 32 characters, which constitute the MD5 hash). This quickly lists files that have the same content.

   Using awk for detailed duplicate reports: For more detail, an awk one-liner can aggregate file names by their hash:

   $ md5sum *.jpg | awk '{ hash=$1; file=$2; files[hash] = files[hash] file " "; count[hash]++; } END { for(h in files) if(count[h] > 1) { print count[h] " duplicates of hash " h ": " files[h]; } }' | sort -nr
   

   This command computes all hashes, builds a list of files for each hash, and at the end prints out any hash that appeared more than once, along with the count and file names. The output is sorted in reverse numeric order (sort -nr) to show the largest groups of duplicates first.

   Using checksums is much faster than manually comparing files byte by byte, and it helps detect duplicates even if file names differ.

3. Environment Variables and printenv

   Environment variables are dynamic values that affect the behavior of the shell and system. They include settings like PATH (which lists directories to search for executables), HOME (the user’s home directory), and many others. These variables can be viewed and set in the shell.

   • Viewing all variables: The printenv command prints all environment variables (or a specific one if a name is provided). Similarly, running env with no arguments also displays the environment.
   • Viewing a specific variable: Use echo $VARIABLE or printenv VARIABLE. For example, echo $HOME prints the home directory path.
   • Setting a variable: Use the export command to set a variable and mark it for export to child processes, e.g., export EDITOR=nano (this makes $EDITOR equal “nano” for that session and any processes started from it).

   $ printenv | head -n 3
   HOME=/Users/alice
   LANG=en_US.UTF-8
   PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

   Environment variables can influence program behavior. For example, a program might check $LANG to determine the language, or $TZ for the timezone. The env command can also run a command under a modified environment (for instance, env TZ=UTC date will run the date command with the timezone temporarily set to UTC, without changing the global environment).

   Note for macOS: The default shell on modern macOS is Zsh (replacing Bash). Configuration that would be placed in ~/.bashrc on Linux systems should be added to Zsh’s startup files on macOS. For login sessions (such as opening a new Terminal window), Zsh reads ~/.zprofile, and for interactive shells it reads ~/.zshrc. Therefore, to persistently set environment variables or define aliases on macOS, add them to ~/.zprofile (or ~/.zshrc) and reload the configuration.

   For example, to ensure Homebrew’s installation is properly integrated into the environment, one can include the following lines in ~/.zprofile:

   # Homebrew environment setup (macOS)
   eval "$(/opt/homebrew/bin/brew shellenv)"      # Set PATH and other Homebrew variables
   export PATH="/opt/homebrew/sbin:$PATH"         # Include Homebrew sbin in PATH
   

   Explanation: The first line runs the Homebrew brew shellenv command, which outputs the necessary export statements to configure Homebrew in the shell; using eval applies those exports. The second line explicitly prepends Homebrew’s sbin directory to the PATH, ensuring any executables installed there (for example, some server binaries like Nginx) can be found. After adding these lines, the updated environment will be applied in new terminals, or immediately in the current session with source ~/.zprofile.

4. Aliases and Shell Startup Files

   Frequently used or long commands can be turned into shortcuts using aliases. An alias is like a custom command that expands into a longer command. Aliases are usually defined in a shell startup file (for example, ~/.bashrc on Linux or ~/.zshrc on macOS) so they are available in every session.

   • Defining an alias: The syntax is alias name='command'. For example, alias ll='ls -l --color' creates an alias ll that lists directory contents in long format (with output colored, if supported).
   • Making it permanent: Adding the alias definition to the shell’s profile file (for example, ~/.bashrc or ~/.bash_profile on Linux, and ~/.zshrc or ~/.zprofile on macOS) ensures it is configured for each new shell session.
   • Activating changes: After updating the configuration file, new terminal sessions will pick up the changes. To apply them immediately in the current shell, use the source command. For example, source ~/.bashrc (or ~/.zshrc) reloads the configuration without the need to log out and back in.

   $ alias gs='git status'
   $ gs
   On branch main
   ...

   In the above example, an alias gs is set to run git status. After defining it (or after reloading the configuration), typing gs executes git status.

   Aliases are a simple way to save keystrokes. For more complex logic (such as running multiple commands or conditional behavior), a shell function or script might be more appropriate.

5. Finding Files and Executables

   There are multiple commands to locate files and identify executables on the system. Depending on the need (finding an executable’s path, or searching for files by name or content), different tools can be used, such as which, type, find, grep, and xargs.

   1. Locating Commands: which vs type

      The which command searches the directories listed in $PATH for an executable file and prints the first match. For example, which python might output /usr/bin/python. It is an external command and only knows about executables in the PATH.

      The shell’s built-in type command is more powerful: it indicates how a given name would be interpreted by the shell. It can identify if a command is built-in, an alias, a function, or an external executable. For instance, if ls is an alias, type ls would show the alias definition, whereas which ls would just show the path to the ls executable.

      $ which ls
      /bin/ls
      $ type ls
      ls is aliased to `ls --color=auto`

      In the above example, which finds the ls binary’s location, while type reveals that ls is actually an alias (which eventually calls the binary with options). If custom aliases or functions are defined, which will not show them, but type will.

   2. Searching for Files by Name: find

      The find command performs a recursive search through directories. It is extremely flexible, with options to filter by name, type, size, modification time, and more. Basic usage requires a path to search in (such as /etc or . for the current directory) and a search condition.

      • By name: find /etc -name "*.conf" finds all files under /etc ending with .conf.
      • By type: find . -type d -name "test*" finds directories (-type d) in the current folder whose names start with “test” (use -type f for files).
      • Depth control: find . -maxdepth 2 -name "*.txt" limits how deep to search in subdirectories (in this case, 2 levels deep).

      $ find /etc -type f -name "*.conf"
      /etc/nginx/nginx.conf
      /etc/ssh/sshd_config
      ...

      If file names contain spaces or special characters, find will handle them safely (printing each path on a separate line). Multiple conditions can be combined as well (for example, finding files modified in the last day that also match a name pattern).

   3. Running Commands on Found Files: -exec

      find can not only locate files, but also perform actions on them using the -exec option. This option executes a specified command on each found file. In the -exec syntax, {} is a placeholder for the current file name, and the command is terminated with either \; or +.

      • Listing found files: find /var/log -name "*.log" -exec ls -l {} \; will run ls -l on each log file found.
      • Safe placeholder: {} is the placeholder for the file path. It’s often wrapped in quotes (like "{}") to handle file names with spaces, although find handles many such cases automatically.
      • Dry-run with echo: Before running a potentially destructive command, substituting the command with echo is prudent. For example, find ~/tmp -name "*~" -exec echo rm {} \; will output lines like rm /Users/alice/tmp/file~ for each match, without actually deleting anything.

      $ find ~/tmp -type f -name "*~" -exec echo rm {} \;
      rm /Users/alice/tmp/doc1.txt~
      rm /Users/alice/tmp/image.png~

      After verifying the list, the actual remove can be done by running the command without echo. The -exec action executes once per matching file. If -exec ... {} + (with a plus) is used, find will attempt to pass multiple files to a single command invocation for efficiency.

   4. Finding Text in Files: grep and xargs

      To search for text inside files across directories, a couple of approaches are available:

      • grep -R (recursive grep): grep -R "search text" /path/to/dir will recursively search through all files under the given directory for the pattern, printing any matches. This one-command solution is quick for finding text in files.
      • Using find and xargs: Another approach is to use find to list files and then pipe the list to xargs, which runs grep on batches of those files. xargs takes input (like filenames from find) and appends them as arguments to the specified command. For example:

      $ find . -name "*.py" -print0 | xargs -0 grep -H "TODO"
      ./app/main.py:    # TODO: improve error handling
      ./app/utils.py:    # TODO: optimize this function

      In this example, find lists all .py files in the current directory and below. The -print0 option separates file names with a null character (to handle spaces in names), and xargs -0 reads the input accordingly. The -H option in grep forces it to always print the filename with matches. The result is a list of all "TODO" occurrences in Python files, each prefixed by its file path.

      The combination of find and xargs is powerful when grep -R isn’t available or if additional filtering of files is needed before searching. Note that xargs will handle long lists by invoking the command multiple times as necessary. On modern systems, grep -R is usually the simplest method to search within files.

   5. Custom Search Functions

      Sometimes it’s handy to wrap common find/grep patterns in a shell function for quick reuse. Here are example functions that could be added to a shell configuration file (adjusting paths as needed):

      • file_grep <text> <path> – Recursively search for text in all files under path, case-insensitively. It uses sudo find <path> -type f -print0 | xargs -0 grep -i "<text>" to handle filenames safely (null-separated). Note: sudo is included to ensure access to all files under path; omit it if not needed. Add -n to the grep part for line numbers in results.
      • find_re <path> '<regex>' – Find files under path whose full file path matches the given regular expression (case-insensitive). It runs find <path> -type f -iregex "<regex>". For example, find_re /opt/homebrew '.*frank.*' would list files with “frank” anywhere in their path.
      • find_str <path> <substring> – Find files under path with substring in their filename (case-insensitive). It is a shortcut for using find -iname "*substring*". For instance, find_str /opt/homebrew conf might find files like nginx.conf or myconfig.txt (because “conf” appears in their names).

      These custom functions leverage the flexibility of find combined with grep, encapsulating common search tasks into single commands. They can be defined in ~/.zshrc or another shell startup file for convenience.

6. Shell Command History and Reuse

   The Zsh and Bash shells keep a history of commands that have been entered. This history allows quick access to previously run commands, making it easy to repeat or modify them without retyping. There are settings to control how many commands to remember and how the history is stored.

   History Size and Configuration

   By default, shells remember a certain number of commands (often 500). Two important environment variables control this memory:

   • HISTSIZE: The number of commands to keep in memory for the current session’s history.
   • HISTFILESIZE: The maximum number of lines to store in the history file (persisted in ~/.bash_history for Bash or ~/.zsh_history for Zsh).

   By increasing these values (for example, adding HISTSIZE=10000 and HISTFILESIZE=20000 in a startup file), the shell can retain a larger history (10,000 commands in memory and 20,000 in the file, in this example).

   Another useful variable is HISTCONTROL (in Bash) or HIST_IGNORE_DUPS / HIST_IGNORE_SPACE in Zsh, which can be set to avoid recording duplicate commands or commands prefixed with a space (often used to avoid logging sensitive commands).

   History Expansion Shortcuts

   Shells provide “history expansion” shortcuts using the ! character for quickly reusing commands from history:

   • !! – Repeat the last command. (Equivalent to !-1.)
   • !n – Run command number n from the history list (as shown by history). For example, !1203 would run the command with index 1203.
   • !-n – Run the command from n lines back. For example, !-3 executes the third from last command. Adding :p at the end (e.g., !-3:p) will print that command instead of executing it.
   • !string – Run the most recent command that begins with string. For instance, !grep runs the last command that started with “grep”.
   • !?string? – Run the most recent command that contains string anywhere in it. Example: !?ssh? would find the last command that had “ssh” in it (whether at the start or middle of the command).
   • ^old^new – A quick substitution in the previous command. This finds the first occurrence of “old” in the last command and replaces it with “new”, then executes the modified command. For example, if the previous command was ls *.jp (missing the ‘g’), typing ^jp^jpg will run ls *.jpg.

   These shortcuts can save time. For example, if a command fails due to insufficient permissions, using sudo !! will rerun the last command with elevated privileges (sudo).

   It is also possible to search the command history interactively by pressing Ctrl-R and typing a part of a command; the shell will auto-fill the most recent matching command, which can then be executed or edited.

7. Command Line Editing Mode (Emacs vs Vi)

   The shell has two primary editing modes for the command line: Emacs mode and Vi mode. By default, Zsh and Bash use Emacs keybindings, which allow familiar cursor movement and editing shortcuts (similar to common shortcuts in many applications, and to the Emacs text editor).

   • Emacs mode: Enabled by default (or with set -o emacs). It provides quick editing keys like Ctrl-A (move to start of line), Ctrl-E (move to end of line), Alt-B (move backward one word), Alt-F (move forward one word), Ctrl-K (cut text to end of line), Ctrl-Y (paste the cut text), and Ctrl-R (reverse search through history). These shortcuts make it efficient to fix typos or rerun commands.
   • Vi mode: Can be enabled with set -o vi for those who prefer Vi/Vim keybindings. In Vi mode, pressing Esc toggles from insert mode to command mode. Navigation then uses h, j, k, l (left, down, up, right), and word movements with b (backward) and w (forward), etc., similar to editing in Vi.

   These modes affect only how the command line is edited, not how commands execute. Emacs mode is generally easier for beginners (arrow keys and the Ctrl shortcuts are intuitive), whereas Vi mode is appreciated by those with Vi experience. Either mode can be used, depending on personal preference and familiarity.

8. Custom Shortcuts for Changing Directories

   Navigating deep directory structures can be time-consuming. Shells allow the creation of functions to act as smart shortcuts. One example is a function that quickly changes to a specific directory based on a keyword. Consider this function defined in a shell startup file:

   qcd() {
       case "$1" in
           web)    cd /opt/homebrew/var/www ;;    # jump to web root
           nginx)  cd /opt/homebrew/etc/nginx ;;  # jump to Nginx config
           *)      echo "qcd: unknown key '$1'"; return 1 ;;
       esac
       pwd  # print current directory
   }
   

   This qcd function takes one argument and switches to a preset directory depending on the keyword. For example, qcd web jumps to /opt/homebrew/var/www, and qcd nginx goes to /opt/homebrew/etc/nginx. After changing directory, it prints the current directory (using pwd) as confirmation.

   Custom directory shortcuts like this can save a lot of typing. Another built-in tip for quick navigation is the CDPATH environment variable – one can list base directories in CDPATH, and then cd will directly navigate into matching subdirectories from those bases. However, defining explicit functions or aliases (like qcd) provides more control over the shortcut names and target locations.

9. Toggling Between Directories with cd -

   The shell provides a quick way to jump to the previous working directory using cd -. This command switches back to the last directory you were in, which is very handy for toggling between two locations.

   $ pwd
   /Users/alice/projects
   $ cd /var/www/html
   $ pwd
   /var/www/html
   $ cd -
   /Users/alice/projects
   

   In this example, the user was in /Users/alice/projects, then changed to /var/www/html. Executing cd - returns to /Users/alice/projects and also prints that directory path. Running cd - again would flip back to /var/www/html, and so on.

   This works because the shell remembers the previous directory in the variable OLDPWD, and cd - is essentially a shorthand for cd "$OLDPWD" (with the added behavior of printing the directory after switching). It provides a quick way to bounce between two directories without retyping long paths.

10. Other Handy Commands: date, seq, yes, and Advanced grep

    Beyond the basic text tools, there are a few more commands that are useful in scripting and automation:

    date – Display or Format Dates

    The date command prints the current date and time, or formats the date/time in a specified way. This is helpful for timestamps in scripts or logs.

    • Default output: Running date with no arguments shows the current date/time in a standard format.
    • Custom format: date "+%Y-%m-%d %H:%M:%S" outputs the date as YYYY-MM-DD HH:MM:SS. The + indicates a format string, where %Y is the year, %m is the month, etc. Many format specifiers are available (for example, %A for the full weekday name).
    • In scripts: One can use $(date +%s) to get the current Unix timestamp (seconds since 1970-01-01), or embed a formatted date into a filename, e.g., backup_$(date +%Y%m%d).tar.gz.

    seq – Generate Sequences of Numbers

    seq outputs a sequence of numbers, one per line. It’s handy for generating loops or numbered lists.

    • Basic sequence: seq 5 produces 1, 2, 3, 4, 5 on separate lines.
    • With start, step, end: seq 2 2 10 starts at 2, steps by 2, and goes up to 10 (outputting 2, 4, 6, 8, 10).
    • Fixed width: seq -w 1 3 prints 01, 02, 03 – the -w option pads numbers with leading zeros to equalize width.

    Sequences often complement other commands. For example, generating numbered filenames in a script (by iterating over $(seq N)) or creating a simple loop in the shell.

    yes – Automatic Confirmation or Repetition

    The yes command outputs a string repeatedly (by default "y" followed by a newline, if no string is provided). Its primary use is to feed a continuous stream of a response into a program that asks for user input, effectively auto-confirming prompts.

    • Auto-confirm prompts: If a command asks “Are you sure? (y/n)”, yes | command can be used to automatically answer "y" to every prompt. For example, if a script or installer repeatedly prompts for confirmation, piping the output of yes will send "y" responses continuously. (Many package managers and installers have their own -y or “assume yes” option, which is often preferable to using yes, but yes demonstrates how to handle generic cases.)
    • Custom response: You can specify an alternate string: yes no | command would continuously output "no". Be cautious, as yes will not stop until the receiving program terminates or its input stream is closed.
    • Generate large input/data: yes can quickly produce a large volume of text. For instance, yes X | head -n 1000 > file.txt will create a file with 1000 lines of "X". Remember to stop yes with Ctrl-C if it’s running without a terminating condition, since it will otherwise run indefinitely.

    Advanced grep Techniques

    Earlier, this guide introduced grep for basic searches. Here are a few more useful options and patterns for powerful text searching:

    • Recursive search: As mentioned, grep -R (or grep -r) searches directories recursively. Use -n to include line numbers in output and -H to always show filenames, especially when searching a single file.
    • Context lines: grep -A 2 -B 2 "ERROR" logfile shows 2 lines After and 2 lines Before each matching line (in this case, “ERROR”), providing context around each match. (The -C option can be used as a shorthand when the number of lines before and after is the same.)
    • Counting matches: grep -c "pattern" file reports the number of matching lines (which can be useful for getting a quick count of occurrences).
    • Extended regex: grep -E "fail|error|critical" file.log enables extended regular expression mode, allowing the use of alternation (in this example, matching lines that contain “fail” or “error” or “critical”). For even more complex patterns, grep -P enables Perl-compatible regex (in GNU grep).
    • Only filenames: grep -l "text" *.md lists only the filenames of files that contain the search text (useful to identify which files have a certain string without printing the lines).

    With these options, grep becomes not just a tool to find exact lines, but also a context-aware search utility and a quick content scanner across many files.

11. Less-Known but Useful Text Utilities

    Apart from the commonly used commands, there are other text processing utilities that can be very handy in specific situations:

    • tac – Concatenate and print files in reverse. (tac is essentially cat backwards.) It reads a file from bottom to top, outputting the last line first. For example, tac file.txt will display the file with lines in reverse order. Note: macOS does not include tac by default, but an equivalent effect can be achieved with tail -r (or by installing GNU coreutils which provides tac).
    • paste – Merge lines of files. It reads lines from multiple files and pastes them side by side separated by tabs (by default). For instance, if one file contains names and another contains scores, paste names.txt scores.txt will produce a two-column output with names in the first column and corresponding scores in the second.
    • diff – Compare files line by line. It outputs the differences needed to change one file into the other. By default, diff file1 file2 produces a series of lines indicating changes. The -u option provides unified diff format (commonly used for patches), and using tools like colordiff or graphical diff viewers can make the output easier to interpret. Additionally, diff -r can compare directories recursively.
    • tr – Translate or delete characters. It reads from standard input and writes to standard output, performing character-by-character substitution or deletion. For example, tr 'A-Z' 'a-z' will translate lowercase letters to uppercase in the input text. tr -d can delete characters (e.g., tr -d '\r' removes carriage return characters, which is useful for converting Windows-formatted text files).
    • rev – Reverse the characters in each line. It’s the character-wise counterpart to tac. Running rev on a line like “Hello” outputs “olleH”. This is useful for certain text processing tricks or simply to reverse strings from the command line.

    Each of these commands addresses a particular type of task. While they might not be needed every day, knowing about them means having a simple solution at hand for specific problems instead of writing more complex code or using a heavier tool.

12. Text Processing Power Tools: awk and sed

    awk and sed are classic Unix utilities for processing text streams. They are extremely powerful, but even basic uses can save a lot of manual effort.

    sed – Stream Editor

    sed is a stream editor that reads input (from files or standard input), transforms it according to a set of commands, and outputs the result. It’s commonly used for substitution (find and replace) or for deleting lines.

    • Substitute text: sed 's/old/new/g' file.txt will replace all occurrences of “old” with “new” in each line of file.txt. (The g at the end stands for “global,” meaning replace all occurrences in the line. Without it, only the first occurrence on each line is replaced.) By default, sed prints the transformed text to standard output. Use -i (with caution) to edit the file in place.
    • Delete lines: sed '/^$/d' file.txt will delete all empty lines (lines matching the regex ^$, which means “start and end of line with nothing between”). Similarly, sed '5,10d' would delete lines 5 through 10 of the file.
    • Selective printing: Normally sed prints every line (after applying edits). Using -n suppresses automatic printing, and the p command can be used to print specific lines. For example, sed -n '1,3p' file.txt will output only lines 1 through 3 of the file.

    awk – Pattern Scanning and Processing Language

    awk is a small programming language specialized for text processing. An awk command typically consists of a pattern and an action, and it operates on each line of input (by default).

    • Field processing: awk splits each input line into fields (by whitespace by default, or by a delimiter given with -F). Fields are accessed by $1, $2, etc. For example, awk '{print $2, $1}' names.txt will print the second field followed by the first field of each line in names.txt, effectively swapping the first two columns.
    • Conditions: A pattern before the action block { ... } can filter which lines to act on. For instance, awk '$3 > 100 {print $1, $2}' data.txt will print the first two fields of lines where the third field is greater than 100. (If data.txt had multiple columns of numbers, this would extract those with a certain value in the third column.)
    • Aggregation: awk has BEGIN and END blocks for actions before processing input and after processing all input, respectively. It also supports variables and arithmetic. For example, awk '{sum += $1} END {print sum}' numbers.txt will calculate and print the sum of the first field of all lines in numbers.txt.

    Both sed and awk can accomplish complex text manipulations that might otherwise require writing a separate program. They have a learning curve, but mastering them (even at a basic level) greatly enhances your command-line text processing capabilities.

13. Combining Commands with &&, ||, and ;

    In shell scripting and on the command line, multiple commands can be strung together on one line using control operators. The three most common operators are &&, ||, and the semicolon ;. They allow conditional or sequential execution of commands:

    • command1 && command2 – Executes command2 only if command1 succeeded (i.e., returned an exit status of 0). This is useful for chaining commands that depend on each other. For example, mkdir newdir && cd newdir creates a new directory and then changes into it only if the directory creation was successful.
    • command1 || command2 – Executes command2 only if command1 failed (returned a non-zero exit status). For example, grep "pattern" file || echo "Pattern not found". Here, if grep does not find the pattern (and thus exits with status 1), the echo runs; if grep succeeds, the echo is skipped.
    • command1; command2 – Executes command1 and then, regardless of its success or failure, executes command2. Commands separated by ; always run sequentially, ignoring exit status. For example: echo "Starting update"; brew update will print the message, then run brew update afterward.

    Using && and || can make one-liners and scripts more concise and robust by handling success or failure in-line. They function like logical AND and OR, but for command execution flow.

14. Command Substitution: Using $(...) to Embed Commands

    Command substitution allows the output of one command to be inserted into the arguments of another command. In Bash and Zsh, this is done with the syntax $( ... ). (Older syntax using backticks `...` is still supported but $(...) is generally preferred for readability and nesting.)

    For example, suppose there are many .txt files and you want to move those that contain the word “Kansas” to a directory named “kansas”. This can be done by combining grep with mv using command substitution:

    $ mv $(grep -l "Artist: Kansas" *.txt) kansas/

    Here’s how it works:

    • grep -l "Artist: Kansas" *.txt lists all .txt filenames in the current directory that contain the string “Artist: Kansas”. The -l option makes grep output only the names of files with matches, rather than the matching lines.
    • The $( ... ) around that command captures its output. The shell will replace the entire $(grep ...) expression with the list of files that grep found.
    • The mv command then effectively sees: mv file1.txt file2.txt ... kansas/, which moves all those files into the kansas/ directory (which should exist beforehand).

    Command substitution is extremely useful for embedding the result of one command into another. Another simple example: echo "Today is $(date +%A)" might output “Today is Wednesday” (if run on a Wednesday), since $(date +%A) is replaced by the output of the date +%A command (which prints the weekday name).

15. Process Substitution: Treating a Command’s Output Like a File

    Process substitution is a feature of Bash and Zsh that allows the output of a command (or input to a command) to be referenced like a file. The syntax is <(command) for substituting output and >(command) for substituting input (the latter is less commonly used). It’s often used with commands that normally take file arguments, enabling them to instead operate on dynamic data.

    One classic example is comparing the outputs of two commands using diff. Normally diff compares two files, but with process substitution:

    $ diff <(ls *.jpg | sort) <(seq -f "%g.jpg" 1 1000)

    This compares two “virtual” files:

    • The output of ls *.jpg | sort (a sorted list of .jpg files in the current directory).
    • The output of seq -f "%g.jpg" 1 1000 (which generates the sequence 1.jpg, 2.jpg, ..., 1000.jpg).

    The diff command sees each of those outputs as a file (the shell creates temporary file descriptors for them) and thus can report differences between the actual files present and the expected sequence of files.

    The <( ) syntax runs the commands inside the parentheses and provides paths like /dev/fd/63 that refer to their outputs. This can be used in many situations where a command expects a filename, allowing you to feed in process outputs directly. Keep in mind that process substitution is not a POSIX feature; it is specific to certain shells (like Bash and Zsh).

16. Executing Commands from a Pipe or File

    Usually, commands are executed by typing them in the shell or running a script file. However, it’s also possible to execute commands that come from standard input or a file, effectively treating command text as a script on the fly.

    For instance:

    $ echo "ls -l /Applications" | bash

    This pipeline sends the string "ls -l /Applications" into the Bash shell’s standard input. Bash will execute the commands it reads from stdin, in this case listing the /Applications directory with details. The effect is similar to writing that command in a file and then running bash file, but it’s done without an intermediate file, using a pipe.

    Similarly, if a file (say commands.txt) contains a series of commands, running bash < commands.txt will execute those commands in sequence as if they were typed into the shell. This is a simple way to run a batch of commands without creating a separate script file (though functionally it is like a temporary script).

    When using this technique, especially with input from untrusted sources, be cautious: feeding text into sh or bash will execute whatever commands are present, which can be dangerous if the content is not known to be safe.

17. Running Commands in the Background

    On the command line, appending an ampersand (&) to a command will execute it in the background. This means the shell prompt returns immediately and the command runs concurrently, allowing you to continue other work. Background jobs are useful for long-running processes that you don’t need to monitor actively.

    For example:

    $ long_running_script.sh &
    [1] 4821

    When run, this command outputs a job identifier in brackets (here [1]) and a process ID (4821). The shell prompt returns while long_running_script.sh continues executing in the background.

    The shell provides built-in commands to manage these background jobs:

    • jobs – Lists current jobs, showing their job number and status (Running, Stopped, etc.).
    • fg %1 – Brings job number 1 to the foreground (so it becomes active and occupies the shell again). Simply fg without an argument brings the most recent background job to foreground.
    • bg %1 – If a job is stopped (perhaps via Ctrl-Z) or currently in the foreground, this resumes it in the background.
    • kill – Sends a signal to a process, usually to terminate it. You can use either the process ID or the job number with a percent sign (e.g., kill 4821 or kill %1) to stop a background job. Use kill -9 (SIGKILL) only if a normal kill doesn’t work, since it forces termination.

    If the output of a background process isn’t redirected, it will still appear in the terminal (possibly intermixing with your typed commands). To avoid this, it’s common to redirect the output to a file (using > filename 2>&1) or to /dev/null if it’s not needed.

    The shell will notify you when a background job completes or stops, typically with a message like [1]+ Done long_running_script.sh the next time you press Enter. This is a helpful reminder that the job finished (or was suspended).

    For processes that need to continue even after logging out, consider using nohup (no hang-up) or a terminal multiplexer like screen or tmux. Running nohup command & will make the process ignore the hang-up signal (so it isn’t terminated when the session ends), and by default it will write output to a file named nohup.out if not redirected.

18. Using Subshells in Pipelines (Tar Archiving Examples)

    A subshell is a separate shell process launched by the current shell. Parentheses ( ... ) around a sequence of commands will execute them in a subshell. A common use of subshells is to localize a change in state (like the working directory or shell options) so that it doesn’t affect the main shell. This is frequently seen in advanced one-liners and scripting patterns, especially when working with archiving or file operations.

    Consider these use cases:

    • Extracting an archive to a target directory: cat archive.tar.gz | (mkdir -p /tmp/other && cd /tmp/other && tar xzvf -) – This one-liner creates a directory /tmp/other (if it doesn’t exist), then changes into it within a subshell, and extracts the archive from standard input (the - tells tar to read from stdin). The current shell’s working directory is unchanged by the cd because it occurs in the subshell. After running, the files from the archive will be in /tmp/other.
    • Copying a directory tree locally: tar czf - dir1 | (cd /tmp/dir2 && tar xzf -) – Here, tar czf - dir1 creates a tarball of the directory dir1 and writes it to stdout (the - as the output file means “standard output”). That output is piped into a subshell where we first cd /tmp/dir2 and then run tar xzf - to extract from stdin into /tmp/dir2. This effectively copies the contents of dir1 into /tmp/dir2 in one line, without creating an intermediate tar file on disk.
    • Copying to a remote host: tar czf - dir1 | ssh user@remote "(cd /path/on/remote && tar xzf -)" – This variation uses ssh to run a subshell on a remote host. The local tar writes to stdout, which ssh then feeds into the remote shell’s tar xzf - command. The subshell on the remote side ensures that the extraction happens in the directory /path/on/remote. This is a convenient way to transfer an entire directory to a remote location in one command.
    • Creating a backup archive with a timestamp: A shell function can leverage subshells to ensure archives are created in the correct directory. For example, a function could cd into a directory in a subshell and run tar there so that the archive’s content doesn’t include unwanted path prefixes. (See the Tar Archiving and Backups section below for an example.)

    These examples show how subshells can be combined with pipes to perform tasks in isolated environments. By using && between commands inside the subshell, each step (e.g., directory creation or changing directory) must succeed for the next to run, adding a measure of safety. Using - with tar instructs it to use standard input or output, which, combined with ssh for remote commands, enables powerful one-liners for copying or deploying files.

    In summary, subshells (with ( ... )) allow you to create a temporary shell with its own environment, which is very useful for containing side effects. After the subshell command completes, you’re back in the original shell environment as if nothing changed, except for whatever output or files the subshell’s commands produced.

19. Subshells vs. Replacing the Shell (Using exec)

    When you run a command like bash (or zsh) from your current shell, you create a new subshell (a child process of your current shell). Your prompt will change (perhaps to indicate a new shell level), and you can exit this subshell to return to the original shell. Any changes (like changed directories or variables) made in the subshell won’t affect the parent shell once it’s exited.

    For example:

    $ echo $PS1
    \u@\h:\w\$ 
    $ bash          # start a new Bash subshell
    $ PS1="subshell$ "   # change the prompt in the subshell
    subshell$ exit   # exit the subshell
    $ 
    

    In this snippet, the prompt \u@\h:\w\$ is the default format (user@host:cwd$). After starting bash, we’re in a subshell; changing PS1 only affects the subshell’s prompt (shown here as subshell$). Exiting the subshell returns to the original shell, which still has the original prompt and environment.

    Sometimes, however, you might want to completely replace the current shell process with another program. The exec command achieves this: when you exec program, your current shell is replaced by program, and when program terminates, there is no shell process to return to.

    For instance, running exec bash in a shell will not create a nested subshell. It will replace the current shell with a new Bash process. When you then exit that Bash, the session ends (since the original shell was replaced). This can be useful in scripts (to hand off control to another program) or when logging in (some configurations exec a login shell to replace the initial process).

    In summary, parentheses ( ... ) launch a child shell (subshell) to execute commands without affecting the parent shell, whereas exec replaces the current shell process entirely with a new program. Subshells are great for containment and local scope; exec is for when you are done with the current process and want to run something else in its place.

20. Useful One-Liner Examples

    Throughout this guide, many one-liner commands have been demonstrated – powerful combinations of tools strung together to perform complex tasks in one go. Mastering one-liners can significantly boost efficiency on the command line. Here are a few additional examples of handy one-liners:

    • Find the ten largest files in a directory tree: du -ah . | sort -h -r | head -n 10. This uses du (disk usage) with -a to list sizes of all files and directories under the current directory, and -h for human-readable sizes. The list is then sorted by size (sort -h understands the size suffixes due to -h) in reverse order (-r, so largest first), and head -n 10 picks the top 10 entries.
    • Recursively search for a string in all files: grep -R "needle" .. This prints all occurrences of "needle" in files under the current directory. Add -n for line numbers or -l to list only the filenames containing the string. (For even faster recursive searching with ignore patterns, a tool like rg (ripgrep) can be used similarly.)
    • Count occurrences of a word in a file: grep -o "apple" file.txt | wc -l. Here, grep -o outputs each occurrence of “apple” on a new line, and wc -l counts those lines, yielding the number of times “apple” appears in the file. (Using grep -c would count matching lines, which would count a line only once even if the word appears multiple times on that line.)
    • Batch rename file extensions: for f in *.jpeg; do mv "$f" "${f%.jpeg}.jpg"; done. This loop iterates over all .jpeg files in the directory, and for each file, constructs a new name by replacing the .jpeg extension with .jpg (${f%.jpeg}.jpg) and then renames the file. The ${f%.jpeg} parameter expansion strips the .jpeg suffix from $f, and then .jpg is appended.
    • Monitor disk usage in real time: watch -n 5 'df -h'. The watch command runs the given command repeatedly (every 5 seconds, in this case) and updates the display. This example re-runs df -h (disk free space in human-readable units) every 5 seconds, so you can observe changes over time. (On macOS, watch is not installed by default, but you can install it via Homebrew or use alternatives like running a loop in a script.)

    With practice, it becomes easier to create one-liners to solve problems on the fly. It is always a good practice to break down a complex one-liner (as done above) to ensure it is understood and accomplishes exactly what is intended.

Quick Reference Cheat Sheet

File and Text Processing

• wc -l file.txt – Print the number of lines in file.txt. (Use -w to count words, -c for bytes.)
• head -n 20 file.txt – Show the first 20 lines of a file. (tail -n 20 shows the last 20 lines.)
• cut -d ',' -f1-3 data.csv – Extract the 1st through 3rd fields from a CSV file (assuming comma delimiter).
• sort -r -n numbers.txt – Sort the lines numerically (-n) and reverse (-r) the order to get highest to lowest.
• uniq -c sorted.txt – Collapse adjacent duplicate lines in a sorted file, prefixing each unique line with its occurrence count (-c).
• tac file.txt – Output the file with lines in reverse order (last line first). (Not available by default on macOS; use tail -r as an alternative.)
• paste file1.txt file2.txt – Combine two files line-by-line (joining lines with a tab by default).
• tr '[:upper:]' '[:lower:]' – Translate uppercase to lowercase from standard input (outputs to standard output). Use tr -d to delete characters (e.g., tr -d '\n' to remove newlines).
• rev file.txt – Reverse each line of the file (characters in each line are reversed).
• diff -u old.txt new.txt – Compare two files and produce a unified diff (lines prefixed with +/– to show changes).

Finding and Searching

• which command – Show the path of the executable that would run for "command". (E.g., which python might output /usr/bin/python.)
• type command – Show how "command" is interpreted by the shell (alias, function, built-in, or file path).
• find /path -type f -name "*.log" – Recursively find files under /path that match the pattern *.log. (Use -type d to search for directories.)
• find . -size +100M -exec ls -lh {} \; – Find files larger than 100MB in the current directory (and subdirs) and list them with details (ls -lh). The \; terminates the -exec clause.
• find . -name "*.tmp" -exec rm {} \; – Find and remove all files ending with .tmp in the current directory tree. (Do a dry-run first: replace rm with echo rm to verify.)
• grep -R "foo" /path – Recursively search for "foo" in all files under /path. Add -n for line numbers, -i for case-insensitive search.
• grep -v "^#" – From input, output all lines that do not start with # (useful for filtering out comment lines).
• xargs – Construct argument lists and execute a command. Often used with find: e.g., find . -type f -print0 | xargs -0 grep "hello" to search for "hello" in all files found (using null-terminated strings for safety).

Shell and Environment

• printenv – Display all environment variables (or use printenv VAR for a specific variable).
• export VAR=value – Set an environment variable for the current shell (and its subprocesses). E.g., export EDITOR=vim.
• alias ll='ls -la' – Create an alias ll for the command ls -la. Add alias definitions to ~/.zshrc (or ~/.bashrc) to make them permanent.
• source ~/.zshrc (or . ~/.zshrc) – Apply changes from the Zsh configuration file to the current session (reload the startup file). Replace ~/.zshrc with ~/.bashrc for Bash.
• HISTSIZE, HISTFILESIZE – Variables controlling how many commands to keep in history (in memory and in the history file, respectively).
• HISTCONTROL=ignoredups:ignorespace (Bash) / HIST_IGNORE_DUPS, HIST_IGNORE_SPACE (Zsh) – Settings to ignore consecutive duplicate commands and those starting with a space in history.
• Ctrl-R – (interactive search) Search backward through command history. Type to filter, press Ctrl-R repeatedly to cycle through matches.
• set -o vi / set -o emacs – Switch the command-line editing mode to Vi or Emacs style. (Emacs mode is default; Vi mode for Vi/Vim key bindings.)
• cd - – Switch to the previous directory (toggle back and forth between two recent directories).
• pushd dir / popd – Use a directory stack. pushd changes directory (like cd) but pushes the old directory onto a stack; popd pops it off to return. Useful for jumping around and then back.

Command Combinations and Scripting

• command1 && command2 – Run command2 only if command1 succeeds (exit status 0).
• command1 || command2 – Run command2 only if command1 fails (non-zero exit status).
• command1; command2 – Run command1 and then command2 unconditionally (sequentially).
• $(command) – Command substitution: use the output of command in place in another command. (E.g., echo "Files: $(ls | wc -l)" prints “Files: X”.)
• diff <(cmd1) <(cmd2) – Process substitution: treat the output of cmd1 and cmd2 as files and compare them with diff (or use as inputs to other commands that expect files).
• echo "command" | bash – Pipe a dynamically generated command or script into a new shell (here Bash) to execute it.
• cmd & – Run cmd in the background. Use jobs to list background jobs, fg to foreground a job, kill to terminate.
• nohup cmd & – Run cmd immune to hangup (logout), in the background. Output goes to nohup.out by default.
• (cd /tmp && do_something) – Execute commands in a subshell (here changing directory to /tmp only within that subshell). The current directory of the main shell remains unchanged.
• exec program – Replace the current shell with program. The current shell process ends, and program takes over (when program exits, the session ends).
• for i in {1..5}; do ...; done – Loop from 1 to 5 (inclusive). Inside the loop, $i holds the current number. Useful for simple iterative tasks in shell scripts.

Written on November 12, 2025

macOS Zsh Shell Setup: Environment, Quick qcd Navigation, Tar Backups, and Find Utilities (Written November 12, 2025)

Environment Setup on macOS (Zsh vs Bash)

Apple’s macOS now uses Zsh as the default shell (since macOS Catalina), replacing Bash:contentReference[oaicite:20]{index=20}. This means configuration that would go into ~/.bashrc on Linux is instead placed in a Zsh startup file. For login shells (the default in Terminal app), Zsh reads ~/.zprofile (and ~/.zshrc for interactive shells). To permanently add environment variables or aliases on macOS, edit your ~/.zprofile (or ~/.zshrc) and add the desired export, alias, or function definitions. After saving, source the file (or reopen Terminal) to apply changes.

For example, to ensure Homebrew’s paths are set up in Zsh, include the lines below in ~/.zprofile. This runs Homebrew’s shell setup and adds Homebrew’s sbin directory to PATH:contentReference[oaicite:21]{index=21}:

# Homebrew environment setup (for macOS)
eval "$(/opt/homebrew/bin/brew shellenv)"      # Homebrew: set PATH and variables
export PATH="/opt/homebrew/sbin:$PATH"         # include Homebrew sbin in PATH

Explanation: The brew shellenv command outputs the necessary export statements to configure Homebrew, which we evaluate in the shell. We also explicitly prepend Homebrew’s sbin directory so commands like Nginx (if installed via Homebrew) are found:contentReference[oaicite:22]{index=22}. After adding these, you can use source ~/.zprofile (or restart the shell) to apply them.

Quick Directory Navigation with a qcd Function

If you frequently work in certain directories, you can define a “quick change directory” function qcd to jump to them by a short keyword. This avoids typing long paths each time. For instance, define qcd in your ~/.zprofile as follows:

# Quick change directory function (qcd) for common locations
qcd() {
    case "$1" in
        web)    cd /opt/homebrew/var/www ;;    # jump to web root
        nginx)  cd /opt/homebrew/etc/nginx ;;  # jump to Nginx config
        *)      # fallback for unknown keyword
            echo "qcd: unknown key '$1'"
            return 1 ;;
    esac
    pwd  # print the current directory to confirm location
}

In this example, running qcd web will cd into /opt/homebrew/var/www, and qcd nginx will go to /opt/homebrew/etc/nginx. The function uses a Bash/Zsh case statement to map a short key to each target directory:contentReference[oaicite:23]{index=23}. If an unsupported key is given, it prints an error message and returns a non-zero status. After changing directory, it prints the new working directory (pwd) so you always know where you ended up:contentReference[oaicite:24]{index=24}.

Tip: You can extend the qcd function with as many locations as needed by adding more case patterns. In Bash, you could even set up tab-completion for the keywords (e.g. using complete -W "web nginx" qcd), though this particular complete command works in Bash but not Zsh:contentReference[oaicite:25]{index=25}. (Zsh has its own completion system if desired.)

Alternatively, if you prefer separate commands per location, simple aliases can suffice. For example, you might add:

alias gonginx="cd /opt/homebrew/etc/nginx/"
alias gohttp="cd /opt/homebrew/var/www/"

in ~/.zprofile, which would let you type gohttp to jump to the web directory, etc. These aliases achieve a similar result:contentReference[oaicite:26]{index=26}, but using the single qcd function is a more scalable approach for multiple destinations.

Tar Archiving and Backups (with Subshell Pipelines)

Regular backups of your web directory are important. We can create a shell function to compress the /opt/homebrew/var/www folder into a timestamped tar.gz archive. A basic approach is: change to /opt/homebrew/var, run tar -zcvf on the www folder, then return. However, a cleaner method is to use a subshell to temporarily change directories for tar, which avoids affecting your current shell’s directory.

For example, here’s a tar_backup function that archives the www folder with a filename prefixed “WEB” plus the current date (YYYYMMDD). It ensures uniqueness by appending a counter if an archive of the same day already exists:

tar_backup() {
    local base="WEB$(date +"%Y%m%d")"       # e.g. WEB20251112 for Nov 12, 2025
    local filename="${base}.tar.gz"
    local counter=1
    # If a file with this name exists, append _1, _2, ... to the base name
    while [ -e "$filename" ]; do
        filename="${base}_${counter}.tar.gz"
        counter=$((counter + 1))
    done
    # Create the tar.gz archive of the www directory
    ( cd /opt/homebrew/var && tar -zcvf "$filename" www )
}

This function uses a subshell ( ... ) to cd into /opt/homebrew/var for the tar command, then automatically returns to the original directory when done. The archive will contain the www/ folder at its top level (because we ran tar from /opt/homebrew/var) rather than an absolute path. Using a subshell in this way isolates the directory change to that subshell process:contentReference[oaicite:27]{index=27}. (Alternatively, GNU tar’s -C option can change directory for you: e.g. tar -C /opt/homebrew/var -zcvf "$filename" www, achieving the same effect without a subshell:contentReference[oaicite:28]{index=28}.)

The naming logic above checks if WEB20251112.tar.gz exists, and if so, tries WEB20251112_1.tar.gz, WEB20251112_2.tar.gz, etc., ensuring no backup is overwritten:contentReference[oaicite:29]{index=29}. The date +"%Y%m%d" part embeds the date in the filename for easy reference.

Usage: Simply call tar_backup in the terminal; it will create the archive in the current directory (which, if you ran it from your home or anywhere, will still place the .tar.gz in /opt/homebrew/var because of the subshell cd). You can extend this idea further: for instance, a tar_web <name> function could take a custom name and produce <name>.tar.gz from the www folder:contentReference[oaicite:30]{index=30}, or a simplified tar_backup_prototype without uniqueness checks:contentReference[oaicite:31]{index=31}—but the above version is robust for daily use.

Extracting (Untarring) the Backup

To restore or inspect a backup, you can extract the tar.gz archive. For safety, it’s wise to extract into a specific directory. Here’s a tar_extract function to do that:

tar_extract() {
    if [ -z "$1" ]; then
        echo "Usage: tar_extract  [target_dir]"
        return 1
    fi
    local archive="$1"
    local target="${2:-/opt/homebrew/var}"   # default target is /opt/homebrew/var
    mkdir -p "$target"
    tar -zxvf "$archive" -C "$target"        # extract here (z=gunzip, x=extract, v=verbose)
}

This will extract the given tarball into /opt/homebrew/var by default (or into a specified target directory if you provide one). It uses tar -zxvf where -z handles gzip compression and -C <dir> changes to the target directory before extraction. For example, tar_extract WEB20251112.tar.gz would unpack the www folder from that archive back under /opt/homebrew/var. (We created the archive with www/ as the top-level folder, so extracting into /opt/homebrew/var recreates /opt/homebrew/var/www and its contents.)

Verification: You can list the contents of a tar file without extracting using tar -tvf archive.tar.gz (t=list, v=verbose). Or perform a quick integrity check by combining commands: for instance, cat WEB20251112.tar.gz | tar -tzvf - will concatenate and list the archive contents in one step:contentReference[oaicite:32]{index=32}. This one-liner uses a pipe to feed the tar file into tar for listing, similar to how one might combine split files or process an archive from standard input.

Note on Subshell Pipelines: The backup function above used a subshell for convenience. This idea comes from the classic “tar pipe” trick to copy or move directory trees. For example, one can clone a directory hierarchy with permissions preserved by running: (cd src && tar -cf - .) | (cd dst && tar -xpf -). Here each tar runs in a subshell that cd to the proper directory before executing, and a pipe connects the archive stream between them:contentReference[oaicite:33]{index=33}. In modern usage, the tar -C flag can achieve the same without subshells:contentReference[oaicite:34]{index=34}, but it’s good to recognize this pattern as it shows how subshells isolate the working directory and environment for each part of a pipeline.

Advanced Find/Grep Functions for Searching

Sometimes you need to search through files or find files by name pattern. The following custom functions (to put in ~/.zprofile) make use of find and grep to simplify common search tasks. All searches are case-insensitive by default (using -i or -iname):

• file_grep <text> <path> – Search within files for a given text string under the specified directory. It will recurse through all files under <path> and print any lines that match <text>. For example, file_grep "nginx" /opt/homebrew will find all occurrences of “nginx” in files under /opt/homebrew. Implementation: it uses find <path> -type f -print0 | xargs -0 grep -i "<text>" to handle filenames safely and pass them to grep:contentReference[oaicite:35]{index=35}. (The -print0 and xargs -0 ensure file names with spaces/newlines are handled correctly.) Note: sudo is included before find in case some subdirectories require root access:contentReference[oaicite:36]{index=36}; you can drop the sudo if not needed. The output will include filenames and the matching lines. If you prefer to see line numbers, add the -n option to the grep command (making it grep -in) so each result shows the line number of the match.
• find_re <path> '<regex>' – Find files by a regular expression pattern on their full path. This uses find <path> -type f -iregex "<regex>", which matches the entire file path against the regex (case-insensitively):contentReference[oaicite:37]{index=37}. For example, find_re /opt/homebrew '.*frank.*' would list all files under /opt/homebrew whose path (including filename) contains “frank” as a substring (the .* wildcards before and after ensure the regex matches anywhere in the path). This is powerful for flexibly filtering files by name pattern beyond simple wildcards.
• find_str <path> <substring> – Find files by a simpler substring match in the filename. It runs find <path> -type f -iname "*<substring>*", so any file whose name (case-insensitive) contains the given text will be listed:contentReference[oaicite:38]{index=38}. For instance, find_str /opt/homebrew 'conf' might find files like nginx.conf or myconfig.txt. This is essentially a shortcut for using the -iname wildcard match of find.

Each of these functions prints results to standard output. They can be piped to other tools (for example, append | less for easier browsing if the output is very long). They illustrate the combination of find and grep for powerful searching in shell scripts. In fact, you could accomplish similar text search with a single grep command (e.g. grep -R "nginx" /opt/homebrew to recursively search within files, or adding -l to list filenames only), but the find | xargs grep approach is more controlled and can be more flexible – for example, you could modify it to restrict by file type or size using find options, or to avoid searching in certain subdirectories.

Conclusion and Usage

Place the above function and alias definitions into your ~/.zprofile (or ~/.zshrc), then run source ~/.zprofile to load them immediately. New Terminal sessions will load them automatically. With this setup on macOS, you benefit from a customized shell environment:

• Homebrew and other environment variables are configured on startup.
• Quick shortcuts like qcd web take you to project directories instantly.
• Simple commands like tar_backup safely archive your work with timestamps, and tar_extract helps restore them.
• Search tools file_grep, find_re, find_str let you locate files and content with ease.

These improvements streamline common tasks and align with macOS’s Zsh shell conventions:contentReference[oaicite:39]{index=39}, making your command-line workflow more efficient and enjoyable.

Written on November 12, 2025

Tar Archiving and Backups (Written November 12, 2025)

Tar ( tar ) is the standard archiving tool on macOS for combining multiple files and directories into a single archive file, often with compression. It’s commonly used for backups, transferring directory structures, and bundling projects. This section covers how to create and extract tar archives in the Terminal, best practices for safe archiving, and advanced techniques for using tar in backup scripts and remote operations.

Creating Archives with tar

The basic syntax to create a tar archive is:

tar -czf archive-name.tgz source-folder

In this command, -c means “create a new archive,” -z compresses it with gzip (producing a .tgz or .tar.gz file), and -f specifies the archive filename. For example, to archive a folder LotsOfFiles into a compressed tarball:

tar -czf LotsOfFiles.tgz LotsOfFiles

If the folder is large, add the verbose flag -v to monitor progress. This will list each file as it’s added:

tar -czvf LotsOfFiles.tgz LotsOfFiles

LotsOfFiles/
LotsOfFiles/file1.txt
LotsOfFiles/file2.txt
...

By default, tar archives store paths relative to the current working directory. To avoid capturing unnecessary path components and to isolate the operation, it’s best practice to either run tar from the directory containing the files or use tar’s -C option. For example, to archive the contents of /opt/homebrew/var/www without embedding the full path:

tar -czf ~/backups/www_site.tgz -C /opt/homebrew/var www

This changes to /opt/homebrew/var while archiving, so that inside www_site.tgz the files are stored under a relative www/ directory rather than /opt/homebrew/var/www . Equivalently, one can use a subshell to similar effect:

(cd /opt/homebrew/var && tar -czf ~/backups/www_site.tgz www)

Tip: Running archive commands in a subshell (as shown above with parentheses) ensures your current shell’s working directory is not changed by the cd . Using -C is functionally the same and improves path safety by preventing unintended parent directories from ending up in the archive.

Extracting Archives

To extract a tar archive, use the -x (extract) flag. For instance, to extract LotsOfFiles.tgz in the current directory:

tar -xzf LotsOfFiles.tgz

Adding -v will show each file as it’s unpacked:

tar -xzvf LotsOfFiles.tgz

LotsOfFiles/
LotsOfFiles/file1.txt
LotsOfFiles/file2.txt
...

It’s often wise to control where the files go when extracting. Use -C to specify a target directory. For example, to extract into a new folder on the Desktop:

mkdir ~/Desktop/Recovery
tar -xzvf project_backup.tgz -C ~/Desktop/Recovery

This ensures all files from project_backup.tgz end up in the Recovery directory. Before extracting unfamiliar archives, it’s good practice to list their contents first with tar -tf (or tar -ztf for gzip-compressed archives) to preview the files and folder structure. This helps avoid any surprises such as “tarbombs” (archives that unpack files all over the current directory). Always extract to an isolated directory or use -C to avoid overwriting important files in your working location.

Backup Script Example: tar_backup

To automate backups, you can use a shell function that creates a timestamped archive. Below is an example function that compresses a directory (e.g. a local website in /opt/homebrew/var/www ) into a date-stamped tar.gz file. It ensures each backup filename is unique by appending a number if a file for that date already exists:

tar_backup() { local src=\"/opt/homebrew/var/www\" local dest=\"~/backups\" local base=\"www_$(date '+%Y%m%d')\" local archive=\"$dest/${base}.tar.gz\" mkdir -p \"$dest\" # ensure destination directory exists local n=0 while [ -e \"$archive\" ]; do n=$((n+1)) archive=\"$dest/${base}_$n.tar.gz\" done tar -czf \"$archive\" -C \"$src\" . echo \"Backup archive created: $archive\" }

This tar_backup function will create an archive like www_20251112.tar.gz (using the current date). If you run it again on the same day, it will produce www_20251112_1.tar.gz , www_20251112_2.tar.gz , and so on, avoiding overwrites. The use of -C \"$src\" . means “change to the source directory and archive . (everything in it)”, so the archive contains only the contents of www , not the entire path. The function also creates the backup directory ( ~/backups ) if it doesn’t exist. For example:

$ tar_backup Backup archive created: ~/backups/www_20251112.tar.gz

After running tar_backup , the ~/backups folder will contain the new compressed archive of the www directory. You can adjust the src and dest paths in the function to suit what you’re backing up and where you want the backup stored.

Safe Extraction Script: tar_extract

Just as creating backups can be scripted, extracting archives safely can be automated with a function. The following tar_extract function takes an archive filename and an optional target directory. It will create a directory (if not provided, it derives one from the archive name) and extract the archive into that location, keeping the files contained:

tar_extract() { local archive=\"$1\" local dest=\"$2\" if [ -z \"$archive\" ]; then echo \"Usage: tar_extract <archive-file.tar.gz> [destination-folder]\" return 1 fi if [ -z \"$dest\" ]; then dest=\"${archive%%.tar*}\" # remove .tar, .tar.gz, .tar.bz2, etc. fi mkdir -p \"$dest\" tar -xzf \"$archive\" -C \"$dest\" echo \"Extracted $archive to $dest/\" }

With this function, extracting an archive is as simple as calling tar_extract archive.tar.gz . If destination-folder isn’t specified, it will create a folder (here, named after the archive file) to hold the contents. For example, running tar_extract ~/backups/www_20251112.tar.gz will make a directory www_20251112 (in the current directory) and unpack the archive into it. The function’s approach of always using a dedicated extraction directory and the -C flag helps prevent the accidental scattering of files into the wrong location. The output confirms where the files were extracted:

$ tar_extract ~/backups/www_20251112.tar.gz Extracted ~/backups/www_20251112.tar.gz to www_20251112/

Tar over SSH for Remote Archiving

Tar’s ability to read from standard input and write to standard output makes it powerful for copying or deploying files between systems without intermediate files. By combining tar with SSH, an entire directory can be sent over the network and unpacked on the fly. For example, to copy a local directory to a remote server:

tar -czf - -C /opt/homebrew/var www | ssh user@remote-server \"tar -xzf - -C /opt/homebrew/var\"

This one-liner does the following: on the local machine, tar archives the /opt/homebrew/var/www directory and sends the compressed archive to stdout (the - after -czf signifies “write to standard output”). The SSH command connects to remote-server and runs another tar that reads from stdin ( -xf - ) and extracts into /opt/homebrew/var . In effect, it streams the www directory to the remote host and unpacks it there, all in a single step. This is an efficient way to deploy files or back up data to a remote machine without creating a temporary archive file on either side. (Add a -v to one or both tar commands if you want to see the list of files transferred.)

The same technique can be reversed to pull files from a remote system. For instance, to back up /var/logs from a remote server onto your Mac:

ssh user@remote-server \"tar -czf - /var/logs\" | tar -xzf - -C ~/Downloads/remote-logs

Here, the tar on the remote side writes a compressed archive of /var/logs to stdout, which is piped over SSH to the local tar command that immediately extracts it into ~/Downloads/remote-logs . This way, the logs are retrieved and unpacked on the Mac without needing an intermediate file.

Using Pipes and Process Substitution with tar

Because tar can work with standard input/output, it excels in advanced shell scenarios using pipes and process substitution. Here are a couple of powerful techniques:

Diffing two archives: To compare the contents of two tar archives (for example, two different backups) without extracting them, you can list their file names and use the diff command with process substitution:

diff -u <(tar -tzf backup-old.tar.gz) <(tar -tzf backup-new.tar.gz)

This command runs tar -tzf (list contents of a gzip-compressed archive) on each archive in a subshell. The <( ) syntax feeds those outputs to diff as if they were files. The result is a unified diff listing differences in file names between backup-old.tar.gz and backup-new.tar.gz – for instance, it will show which files were added or removed. This approach quickly spot-checks changes between archive versions without manual inspection.

Combining split archives: If an archive was split into multiple parts (e.g., by using the split command), you can recombine and extract it on the fly. Suppose a large archive was split into backup.tar.gz.partaa , backup.tar.gz.partab , etc. You can use cat to join the parts and pipe into tar for extraction:

cat backup.tar.gz.part* | tar -xzf - -C ~/Downloads/Recovered

Here, cat concatenates all the part files in order and sends the complete archive stream to tar via the pipe. The tar -xzf - reads from stdin and extracts the contents into the Recovered directory. This saves time and disk space by avoiding the creation of a full recombined archive on disk first.

Useful tar One-Liners

Tar’s flexibility lends itself to many convenient one-line commands for everyday tasks. Below are a few practical examples:

Listing archive contents: To quickly see what’s inside an archive without extracting, use tar -tf (or tar -ztf for compressed archives). For example:

tar -tvf www_20251112.tar.gz | head -n 5

This will display the first five entries of the archive www_20251112.tar.gz in long format (permissions, owner, size, date, and name). You can omit the v to just list file names, or pipe to grep to search for a specific file. This is useful for verifying the contents of a backup archive before doing a full extract.

Streaming a backup to a file or processor: Tar can be combined with other commands via pipelines. For instance, tar’s output could be piped directly into a compression tool or an encryption program. However, since flags like -z (gzip) are built-in, you often don’t need an external compressor. One common pattern is piping tar to openssl for encryption:

tar -czf - /path/to/important/data | openssl enc -aes-256-cbc -e -out secure_backup.tar.gz.enc

Here tar produces a compressed archive on stdout, which is encrypted by openssl and written to secure_backup.tar.gz.enc . The reverse (decrypting and extracting) can be done similarly by piping openssl decryption into tar -xzf - . This demonstrates how tar streams allow on-the-fly processing of archives.

Remote download and unpack: You can download and extract an archive in one step using curl (available by default on macOS). This is handy for installing tarball releases from the internet without manual steps:

curl -L https://example.com/project.tar.gz | tar -xz -C /opt/projects/

The -L option tells curl to follow redirects, and the pipeline feeds the downloaded archive straight into tar -xz , which extracts it into /opt/projects/ . No temporary file is saved locally; the data is streamed directly into the extraction process. This one-liner is frequently used for quick deployments or retrieving sample projects.

In summary, the tar command is a cornerstone for archiving and backups on the macOS command line. By following best practices (such as using -C for directory isolation and verifying contents with tar -t ) and leveraging shell pipelines, tar can handle everything from simple folder compression to complex backup workflows and remote file transfers, all while maintaining safety and efficiency.

Written on November 12, 2025

Advanced Unix-Like Command Explanatory

Collections of Unix-Like Commands

(A) Advanced Listing with ls

Recursive Listing (-R)

The ls -R command lists all files and directories recursively. This means it will display the contents of the current directory and all subdirectories, which is useful for viewing a complete directory structure.

Human-Readable Sizes (-lh)

The ls -lh command displays file sizes in a human-readable format, showing sizes in kilobytes (KB), megabytes (MB), or gigabytes (GB), as appropriate. This makes it easier to understand file sizes at a glance compared to the default byte-based format.

Detailed Listing with Time Sorting (-lart)

The ls -lart command combines several options to provide an advanced view of files:

• -l: Displays files in a long format, including permissions, ownership, size, and modification date.
• -a: Includes hidden files (those starting with a dot) in the listing.
• -r: Reverses the order, showing the oldest files first.
• -t: Sorts files by modification time, placing the most recently modified files at the end of the list when used with -r.

Sorting by File Size in Reverse (-lSr)

The ls -lSr command lists files by size in ascending order:

• -l: Shows detailed file information, such as file size, permissions, and ownership.
• -S: Sorts files by size, starting with the largest.
• -r: Reverses the order, showing the smallest files first.

Displaying Output with Colors (--color=auto)

The ls --color=auto command adds color to the output, distinguishing files, directories, and symbolic links by color. This visual enhancement simplifies identification of different types of files within the terminal.

(B) Deleting Files with rm

Using Quotes for Files with Spaces

When a file name contains spaces, quotes are necessary to ensure the shell interprets it correctly. For example:

rm "file with spaces.txt"

In this case, either single or double quotes can be used to handle the spaces in the file name properly.

Escaping Spaces

An alternative method for handling file names with spaces is to escape each space with a backslash (\):

rm file\ with\ spaces.txt

This approach is particularly effective when dealing with multiple files or file names containing special characters directly in the terminal.

Preventing Critical Deletions (--preserve-root)

The rm -rf --preserve-root command adds an extra safeguard, ensuring the root directory (/) is never deleted. This is crucial to prevent accidental system-wide deletion.

Deleting Files by Pattern (rm !(*.txt))

The rm !(*.txt) command deletes all files except those matching a specific pattern, such as text files. This requires enabling extglob with the command:

shopt -s extglob

This method provides control over batch deletion while protecting specific file types.

(C) Enhancing Copy and Move Commands

cp – Enhanced Copying

The cp command offers several enhancements:

• cp -u source destination: Only copies files if the source is newer than the destination, making it useful for backup scripts.
• cp --parents file /path/to/destination: Copies the file along with its directory structure, preserving the hierarchy.

For more efficient copying, consider using rsync:

rsync -avh source destination

rsync is optimized for large transfers, preserving permissions and utilizing compression.

mv – Moving with Precision

The mv command can be used with additional options:

• mv -u source destination: Moves files only if the source is newer or if the destination file does not exist.
• find . -name "*.bak" -exec mv {} /backup/ \;: Moves all .bak files from the current directory to /backup/, combining find with mv for complex file operations.

(D) Advanced Searching and File Management

find – Complex Search

• find /path -mtime -1: Finds files modified in the last day.
• find /path -type f -exec chmod 644 {} \;: Finds files and changes their permissions in bulk.
• find /path -name '*.log' -size +10M -delete: Deletes log files larger than 10MB.

awk – Advanced Text Processing

• awk '{print $1, $3}' file: Extracts and prints specific columns from a file (e.g., column 1 and column 3).
• awk '/pattern/ {print $0}' file: Prints lines matching a pattern.

Combined with process monitoring, the following command lists users and processes consuming more than 50% CPU:

ps aux | awk '$3 > 50 {print $1, $3, $11}'

sed – Streamlined Text Editing

• sed -i 's/old/new/g' file: Replaces all occurrences of old with new within a file.
• sed '/pattern/d' file: Deletes lines matching a pattern.

(E) Disk Usage and Process Monitoring

df and du – Disk Usage Analysis

• df -hT: Displays disk usage in a human-readable format with file system type.
• du -sh * | sort -rh: Sorts files and directories by size, providing a clear overview of storage usage.
• du --max-depth=1 /path: Shows disk usage for directories up to a specific depth, helping to identify space hogs quickly.

ps and top – Process Monitoring

• ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem: Lists processes sorted by memory usage, helping to monitor resource-intensive tasks.
• top -o %MEM: Sorts processes by memory usage, prioritizing resource consumption monitoring.

(F) Networking and Efficient Command Chaining

netstat and ss – Network Monitoring

• netstat -tuln: Lists all open ports and their associated services.
• ss -tuln: A faster alternative to netstat, providing similar insights into network activity.

xargs – Efficient Command Chaining

• find /path -name '*.log' | xargs rm: Finds all log files and passes them to rm for deletion.
• cat file.txt | xargs -n 1 echo: Feeds each line of a file into echo one by one, enabling efficient multi-line processing.

(G) Useful Aliases and Synchronization

alias – Command Shortcuts

Setting an alias in ~/.bashrc or ~/.zshrc helps reduce repetitive typing:

alias proj="cd /home/user/projects"
alias ll="ls -lh"
alias cls="clear"  

rsync – Smart Synchronization

• rsync -avz --progress source/ destination/: Synchronizes files between two directories with compression and displays progress.
• rsync -avz --delete source/ destination/: Deletes files from the destination that do not exist in the source, ideal for mirroring directories.

These advanced commands and techniques offer powerful control over file management, system monitoring, and data transfers in Linux.

Searching Through Files for Text

(A) grep -r "###" /path/to/designated/folder

This command searches recursively for the string "###" in all files and subdirectories under the specified directory.

• -r: The recursive flag, which enables searching through all files and subdirectories within the provided directory.

Limitations:

• Does not handle special characters in filenames efficiently.
• Outputs both filenames and matching lines, which may not always be desired.

(B) grep -rl "###" /path/to/designated/folder

This command functions similarly to the first, with the addition of the -l flag, which modifies the output to display only filenames containing the matching text, without showing the matching lines.

• -r: Recursively searches through all directories and files.
• -l: Prints only the filenames where matches are found, excluding the matching lines from the output.

Advantages:

• Provides a cleaner output by listing filenames only, making it ideal for quick identification of files containing the specified text.

Limitations:

• Case sensitivity is not accounted for unless the -i flag is added.
• Filenames with spaces or special characters are not handled effectively.

(C) find /path/to/designated/folder -type f | xargs grep -i "###" 1> tmp1 2> tmp2

This command begins by using find to list all files, then pipes the results to grep via xargs, which searches for the specified string "###".

• -type f: Restricts the search to regular files only, excluding directories or other file types.
• |: The pipe operator, which takes the output of find and passes it as input to the next command (xargs).
• xargs: Takes the output from find and uses it as arguments for another command, in this case grep.
• 1> tmp1: Redirects the standard output (such as matching lines and filenames) to a file named tmp1.
• 2> tmp2: Redirects error messages (such as permission denied errors) to a file named tmp2.

Advantages:

• Supports case-insensitive searching through the inclusion of the -i flag.

Limitations:

• Filenames with spaces or special characters are not managed properly without the use of -print0 and xargs -0.

(D) find /path/to/designated/folder -type f -print0 | xargs -0 grep -i "###" 1> tmp1 2> tmp2

This command enhances the previous one by handling filenames that contain spaces or special characters more effectively.

• -print0: Instructs find to output filenames terminated by a null character (\0), rather than a newline. This is beneficial for managing filenames that include spaces, special characters, or newlines.
• |: The pipe operator, used to pass the output from find to xargs.
• xargs -0: Instructs xargs to expect null-terminated input, ensuring compatibility with find's -print0 option and properly handling filenames with special characters.

Advantages:

• Well-suited for directories with filenames that include spaces or special characters.
• Allows case-insensitive searching with the -i flag.

Limitations:

• May introduce minimal overhead when using -print0 and xargs -0.

Locating Installed Command-Line Tools on macOS: find "$(brew --prefix)" -name [tool_name] -type f

This document explains how to determine the installation path of a command-line tool on macOS. The instructions focus on scenarios involving Homebrew installations, though they can be adapted to any other method of tool installation. Every step and detail is preserved from earlier discussions but rearranged and generalized to maintain a high level of clarity and professionalism.

Purpose of the find "$(brew --prefix)" -name [tool_name] -type f Command

This command is a convenient way to locate a specific file—such as a binary for a tool—within the directory structure managed by Homebrew. The example below uses [tool_name] as a placeholder; substitute the actual tool’s name (e.g., ffmpeg, git, or any other executable).

find "$(brew --prefix)" -name [tool_name] -type f

1. brew --prefix: Returns the base directory where Homebrew is installed. On Apple Silicon (M1/M2) systems, this is commonly /opt/homebrew; on Intel-based Macs, /usr/local; some custom Homebrew setups may differ.
2. Substitution with $(...): When the shell processes brew --prefix inside $(...), it replaces that portion of the command with the actual path (e.g., /opt/homebrew), resulting in:

   find "/opt/homebrew" -name [tool_name] -type f

3. find "$(brew --prefix)": Instructs the find utility to begin searching in the Homebrew prefix directory returned by brew --prefix.
4. -name [tool_name]: Tells find to look for files named exactly [tool_name].
5. -type f: Restricts the search to regular files, excluding directories, symlinks, or other file types.

Outcome: The command scans Homebrew’s installation tree for files named [tool_name] and helps pinpoint the exact location of the installed binary.

Searching Beyond Homebrew

Some tools may not have been installed using Homebrew, or they may be installed in a location not covered by the Homebrew directory structure. In such cases, the following approaches can be used:

1. Quick Checks with which or command -v

   which [tool_name]

   or

   command -v [tool_name]

   If [tool_name] is found in the PATH, these commands return the absolute path (for example, /usr/local/bin/[tool_name]). If nothing is returned, the tool is not on the system’s PATH.

2. Comprehensive System-Wide Search

   If the exact location remains unknown, it may be necessary to search the entire file system:

   sudo find / -name [tool_name] -type f 2>/dev/null

   • /: Starts the search from the root directory.
   • 2>/dev/null: Redirects error messages (such as permission denials) to /dev/null, creating a cleaner output.

   This approach can take significantly longer than searching only the Homebrew prefix because it scans every accessible directory on the system.

Benefits of Using find "$(brew --prefix)" -name [tool_name] -type f

• Faster Search: When there is high confidence that the tool was installed by Homebrew, restricting the search to Homebrew’s directory structure saves time by avoiding a full-system scan.
• Automatic Prefix Detection: Because Homebrew installations can vary (e.g., Intel vs. Apple Silicon or custom paths), brew --prefix ensures the command dynamically targets the correct location without manual path specification.

Additional Notes about the find Utility

1. No Extra Installation Required: The find command is included by default on macOS (as well as most Linux and Unix-like systems). There is no need to install additional software to run find.
2. Flexibility in Search Scope: If a tool is installed manually or obtained from a different package manager, it is still possible to use find by directing it to a suspected location or by conducting a system-wide search.
3. Applicability to Any Tool: Although the examples here often reference media-processing utilities (e.g., ffmpeg), the same syntax applies to any file name. Substitute [tool_name] to locate other binaries or resources.

Appendix: macOS — Search file contents for “lecroy” (case-insensitive)

Use one of the two patterns below. Both search recursively from the current directory and safely handle filenames with spaces.

1. Show matching lines with line numbers (skip binary files):

   find . -type f \( -name '*.txt' -o -name '*.html' -o -name '*.py' -o -name '*.js' \) -print0 \
   | xargs -0 grep -I -n -i 'lecroy'

2. List filenames only that contain the keyword:

   find . -type f \( -name '*.txt' -o -name '*.html' -o -name '*.py' -o -name '*.js' \) -exec grep -I -i -l 'lecroy' {} +

• -i = case-insensitive; -n = show line numbers; -l = print filenames only.
• -I tells grep to ignore binary files.
• -print0 with xargs -0 safely handles spaces and newlines in filenames.
• Adjust the -name patterns to include other extensions as needed.

Written on February 12, 2025

Managing Processes in Linux and macOS

Managing processes is a fundamental aspect of system administration in both Linux and macOS environments. Understanding how to check, search for, and control processes is essential for maintaining system performance and stability. This guide provides detailed instructions on managing processes, incorporating tools and commands available in both Linux and macOS.

(A) Checking Processes

A-1) Using the ps Command

The ps (process status) command provides a snapshot of current processes.

• List All Processes:

  ps aux

  • a: Displays processes from all users.
  • u: Shows processes with a user-oriented format.
  • x: Includes processes without a controlling terminal.
• Alternative Format:

  ps -ef

  • -e: Selects all processes.
  • -f: Displays a full-format listing.

A-2) Using the top Command

The top command provides a dynamic, real-time view of running processes.

• Start top:

  top

• Features:
  • Displays CPU and memory usage.
  • Processes are sorted by CPU usage by default.
  • Press M to sort by memory usage.
  • Press P to sort by CPU usage.
  • Press q to quit.

Note: On macOS, top has some differences in options and display.

• macOS top Command Differences:
  • By default, displays processes sorted by CPU usage.
  • Press o to change sort order. For example, to sort by memory:

  o mem

  • Press / to search for a process.
  • Change refresh interval (default 1s) using the -s option:

  top -s 5

  • List a specific number of processes:

  top -n 20

  • Display processes for a specific user:

  top -u username

A-3) Using the htop Command

htop is an interactive process viewer with a user-friendly interface.

• Install htop:

  sudo apt-get install htop    # For Debian-based systems
  	  brew install htop        # For macOS

• Start htop:

  htop

• Features:
  • Scroll through the list of processes vertically and horizontally.
  • Kill processes without typing their PIDs.
  • Visual indicators for CPU and memory usage.

A-4) Using the pstree Command

Displays processes in a tree format, showing parent-child relationships.

• Display Process Tree:

  pstree

(B) Searching for Specific Processes

B-1) Using pgrep

Searches for processes based on name and other attributes.

• Find Processes by Name:

  pgrep process_name

• Options:
  • -l: Lists the process name alongside the PID.
  • -u user_name: Searches for processes owned by a specific user.

B-2) Using ps with grep

Filters the list of processes to find specific ones.

• Search Processes:

  ps aux | grep process_name

• Exclude the grep Process Itself:

  ps aux | grep [p]rocess_name

B-3) Filtering in top and htop

• In top: Press / and type the process name to search.
• In htop: Press F3 and enter the process name.

(C) Identifying Problematic Processes

C-1) High Resource Usage

Processes consuming excessive CPU or memory can degrade system performance.

• Identify High CPU Usage: Use top or htop and sort by CPU usage.
• Identify High Memory Usage: Sort processes by memory usage in top or htop.

C-2) Unresponsive or Zombie Processes

Processes that are not functioning correctly or have become defunct.

• Check Process State:
  • In ps output, the STAT column indicates the state.
  • D: Uninterruptible sleep (usually I/O).
  • Z: Zombie (terminated but not reaped by parent).
  • T: Stopped.
  • R: Running.
  • S: Sleeping.

(D) Analyzing Open Files and Network Connections

D-1) Using the lsof Command

Lists open files and the processes that opened them.

• Find Processes Using a Specific File:

  lsof /path/to/file

• Find Processes Listening on a Port:

  lsof -i :port_number

• Examples:
  • Identify Processes Using Port 80 (HTTP):

  sudo lsof -i :80

  • Identify Processes Using SSH Port 22:

  sudo lsof -i :22

  • List Files Opened by a Specific Process:

  lsof -p PID

D-2) Using the netstat Command

Displays network-related information.

• On Linux:

  netstat -tulpn

• On macOS:

  netstat -anv | grep LISTEN

  • -a: Display all sockets.
  • -n: Show numerical addresses without resolving hostnames.
  • -v: Verbose output.

(E) Killing Specific Processes

E-1) Using the kill Command

Sends a signal to a process to terminate it.

• Basic Syntax:

  kill PID

• Common Signals:
  • SIGTERM (15): Requests a graceful shutdown.
  • SIGKILL (9): Forces immediate termination.
  • SIGSTOP (19): Stops (pauses) a process.
  • SIGCONT (18): Continues a stopped process.

E-2) Using the killall Command

Terminates processes by name rather than PID.

• Kill All Instances of a Process:

  killall process_name

• Options:
  • -u user_name: Kills processes owned by a specific user.
  • -signal: Sends a specific signal.

Note: On macOS, killall targets processes by their full command name as displayed in ps or top. The command is case-sensitive and requires the exact process name.

E-3) Using the pkill Command

Similar to pgrep, but sends signals to processes.

• Kill Processes by Name:

  pkill process_name

• Options:
  • -u user_name: Targets processes of a specific user.
  • -signal: Specifies the signal to send.

E-4) Killing Processes in htop

• Steps:
  1. Navigate to the desired process using the arrow keys.
  2. Press F9 or k to initiate the kill menu.
  3. Select the signal to send (default is SIGTERM).
  4. Confirm to send the signal.

(F) Monitoring System Performance

F-1) Using vmstat

Reports virtual memory statistics.

vmstat 2 5

Note: On macOS, use vm_stat (with an underscore).

vm_stat

F-2) Using iostat

Reports CPU and input/output statistics.

iostat 2 5

F-3) Using sar (Linux Only)

Collects, reports, or saves system activity information.

sar -u 2 5

(G) Best Practices and Security

G-1) Verify Before Termination

Ensuring the correct process is targeted helps prevent unintended system behavior.

• Double-check PIDs and process names before using termination commands.

G-2) Use Signals Appropriately

• Start with SIGTERM: Allows the process to close files and release resources gracefully.
• Use SIGKILL Only if Necessary: Forcefully terminates the process without cleanup.

G-3) Consider Process Hierarchies

• Parent and Child Processes: Killing a parent process may affect child processes.
• Use pstree to understand relationships.

G-4) Monitor System Logs

• Linux Logs: Located in /var/log/. Use tail for real-time monitoring:

tail -f /var/log/syslog

• macOS Logs: Use Console application via Spotlight Search or Terminal commands:

log show --last 1h

G-5) Permissions and Security

Some processes require root privileges to manage. Operate with the least privilege necessary.

• Use sudo when needed:

sudo kill PID

By understanding and utilizing these tools and commands, processes can be effectively managed in both Linux and macOS environments, ensuring optimal system performance and stability.

Accessing and Mounting External Drives

In Unix-like environments and macOS, external drives such as SD cards or USB disks are typically mounted in specific directories, making them accessible from the command line. These drives may be automatically mounted in designated directories, or manual mounting can be employed for greater control.

(A) Accessing External Drives on macOS

On macOS, external drives are automatically mounted in the /Volumes directory. Each drive appears as a folder within this directory, named according to the drive’s label, allowing for organized and predictable access.

cd /Volumes
ls

After navigating to /Volumes, using the ls command lists the mounted drives. For instance, if an SD card is labeled "SDCARD," access it directly by specifying the drive’s path:

cd /Volumes/SDCARD

(B) Accessing External Drives on Linux (Unix-like)

In most Linux distributions, external drives are generally mounted in either /media or /mnt, with specific mounting practices depending on distribution and user configuration:

Automatic Mounting

Drives are usually mounted automatically in /media/username/DRIVENAME, where username represents the logged-in user.

cd /media/username/DRIVENAME

Manual Mounting

For manual mounting, /mnt is commonly used as a directory for temporary mounts. This process requires the use of the mount command.

sudo mount /dev/sdX1 /mnt
cd /mnt

Replace /dev/sdX1 with the correct device name for the external drive. For example, /dev/sdb1 often denotes the first partition on a USB disk.

(C) Using the mount Command for Manual Mounting

The mount command provides flexibility for mounting devices, allowing access to a variety of filesystems and external storage. The command is structured as follows:

sudo mount -o options device mount_point

• device: Specifies the device to mount, such as /dev/sdb1.
• mount_point: Indicates the directory where the device will be accessible, commonly /mnt or a subdirectory within /media.
• options: Optional parameters, such as ro for read-only access or loop for mounting ISO files.

Example Commands

1. Mounting a USB Drive Manually: To mount a USB drive (e.g., /dev/sdb1) to /mnt/usb, use:

sudo mount /dev/sdb1 /mnt/usb

Before accessing the device, ensure that the /mnt/usb directory exists, creating it if necessary:

sudo mkdir -p /mnt/usb

2. Mounting an ISO File as a Loop Device: ISO files are often mounted as loop devices, making their files accessible without burning them to physical media. This example mounts an ISO file as a read-only loop device using the iso9660 filesystem type:

sudo mount -o loop,ro -t iso9660 /path/to/file.iso /mnt/iso

• loop: Treats the ISO file as a block device.
• ro: Mounts the file in read-only mode to prevent modifications.
• iso9660: Specifies the ISO 9660 filesystem, standard for ISO files.

3. Unmounting a Device: To safely remove a mounted device, unmount it using the umount command:

sudo umount /mnt/usb

Ensuring the device is unmounted before physically disconnecting it helps prevent data loss or corruption.

These practices allow for flexible and efficient management of external drives and ISO files across Unix-like environments, providing consistent access through automatic and manual mounting techniques.

Viewing Shell History Using Emacs: emacs ~/.zsh_history

The following instructions describe the process for accessing the complete shell command history using Emacs. The procedure is outlined in a systematic manner, providing details for both bash and zsh shells.

1. Determine the Active Shell

   A shell history file must be identified before proceeding:

   • bash: The history is typically stored in the file ~/.bash_history
   • zsh: The history is usually stored in the file ~/.zsh_history

2. Flush the Current Session’s History

   Before opening the history file, it is advisable to ensure that the session’s command history is fully written to the history file.

   Shell	Command	Description
   bash	history -a	Appends the session's recent commands to the history file.
   zsh	fc -W	Writes the current session's history to the history file.

   Execute the corresponding command in the terminal to update the history file.

3. Open the History File in Emacs

   Once the history file is updated, Emacs can be used to view and search the command history. Launch Emacs with the appropriate file as follows:

   • For bash: emacs ~/.bash_history
   • For zsh: emacs ~/.zsh_history

   Opening the file in Emacs allows navigation, search, and editing of the complete command history.

Written on April 1, 2025

Frequently used macOS command-line reference for practical workflows

• ls -lh — human-readable long listing

  Purpose. Provides a detailed overview of files and directories within the current working directory, presenting sizes in readable units (KB, MB, GB) and showing metadata such as permissions, owner, and modification date.

  Syntax. ls -lh

  Example output.

  -rw-r--r--   1 user  staff   12M Oct 12 14:32 sample.mov
  drwxr-xr-x   5 user  staff  160B Oct 12 13:10 project/
      

  Usage notes. This command does not traverse subdirectories. To include hidden files, use ls -alh.

• find . -type f -exec du -h {} + | sort -hr | head -n 5 — largest five files

  Purpose. Identifies the most space-consuming files across the current directory and all nested subdirectories, supporting targeted cleanup and optimization efforts.

  Syntax. find . -type f -exec du -h {} + | sort -hr | head -n 5

  Example output.

  413M    ./20251005_RickshawRide02.mov
  128M    ./20251004_keyboard.mov
   61M    ./20251003_nexus02.MOV
   24M    ./20251005_yufuin05.MOV
   24M    ./20251002_walk01.png
      

  Usage notes. The pipeline performs three key tasks: du -h calculates human-readable sizes, sort -hr orders results from largest to smallest, and head -n 5 limits the output to the top five results.

• du -sh */ — per-subfolder size summary

  Purpose. Generates a concise summary of disk usage for each immediate child directory, enabling quick identification of space distribution.

  Syntax. du -sh */

  Example output.

  2.0M   _dev/
  193M   attic/
   23M   driver/
   94M   media/
  6.5M   secure/
  929M   src/
  5.4G   travel/
      

  Usage notes. The -s flag produces a total summary, -h formats sizes in human-readable units, and the */ glob limits the scope to directories one level below the current working path.

  Interpretation nuance. The “total” value shown by ls counts 512-byte blocks, whereas du -h reports sizes using binary units and rounds to one decimal. Minor discrepancies are normal and expected.

Written on October 12, 2025

Debian OS from macOS

Downloading Debian ISO Using Jigdo on macOS

This guide provides detailed instructions for downloading Debian ISO files using Jigdo on a macOS system. The steps are organized to ensure clarity and efficiency, addressing potential challenges that may arise during the process.

(A) Install Jigdo via Homebrew

Homebrew serves as a package manager for macOS, facilitating the installation of various software packages, including Jigdo.

A-1) Install Jigdo:

brew install jigdo

Execute this command in the Terminal to install Jigdo.

A-2) Verify Jigdo Installation:

brew list jigdo

Confirm that Jigdo has been installed correctly. Typical output includes executable files located in /opt/homebrew/Cellar/jigdo/0.8.2/bin/, such as:

• jigdo-file
• jigdo-lite
• jigdo-mirror

echo $PATH

echo 'export PATH="/opt/homebrew/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

echo 'export PATH="/opt/homebrew/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

which jigdo-lite

brew unlink jigdo && brew link jigdo
which jigdo-lite

(C) Download and Prepare Debian Jigdo Files

To reconstruct Debian ISO files, both .jigdo and .template files are required. These files provide the necessary information and structure for the ISO assembly.

C-1) Download the .jigdo and .template Files:

Navigate to the Debian Jigdo DVD Images. Download the corresponding .jigdo and .template files and save them in ~/Desktop/Debian/.

(D) Execute Jigdo to Download and Assemble the Debian ISO

With the necessary Jigdo files in place, proceed to download and assemble the Debian ISO.

D-1) Navigate to the Jigdo Files Directory:

cd ~/Desktop/Debian/

D-2) Run Jigdo-lite for a Single ISO:

jigdo-lite debian-12.7.0-i386-DVD-1.jigdo

D-3) Batch Download Multiple ISOs:

For scenarios involving multiple Jigdo and template files (e.g., 21 ISOs), scripting can automate the download process. Below are two versions of the Zsh script, each with different approaches for automating responses to Jigdo prompts.

For Zsh (using yes for continuous "Enter" presses):

#!/bin/zsh

# Directory containing all .jigdo and .template files
JIGDO_DIR=~/Desktop/Debian/

# Navigate to the Jigdo directory
cd "$JIGDO_DIR" || exit

# Loop through all .jigdo files and execute jigdo-lite, with continuous "Enter" key presses
for jigdo_file in *.jigdo; do
   echo "Processing $jigdo_file..."

   # Use 'yes' to simulate continuous "Enter" presses for each file
   yes '' | jigdo-lite "$jigdo_file"
done

This version of the script employs the yes command to repeatedly send an empty string, simulating continuous "Enter" presses until all Jigdo prompts are satisfied. This method is useful if the number of prompts varies or if additional confirmations are required during the download process.

For Zsh (using printf for exactly two "Enter" presses):

#!/bin/zsh

# Directory containing all .jigdo and .template files
JIGDO_DIR=~/Desktop/Debian/

# Navigate to the Jigdo directory
cd "$JIGDO_DIR" || exit

# Loop through all .jigdo files and execute jigdo-lite, with exactly two "Enter" key presses
for jigdo_file in *.jigdo; do
   echo "Processing $jigdo_file..."

   # Use 'printf' to simulate pressing "Enter" twice for each file
   printf '\n\n' | jigdo-lite "$jigdo_file"
done

This version of the script uses printf to send exactly two newline characters, simulating two "Enter" key presses. It is beneficial when only two prompts are expected, as it avoids continuous input and provides controlled interaction with the Jigdo process.

#!/bin/bash

#Directory containing all .jigdo and .template files
JIGDO_DIR=~/Desktop/Debian/

# Navigate to the Jigdo directory
cd "$JIGDO_DIR" || exit

# Loop through all .jigdo files and execute jigdo-lite, with continuous "Enter" key presses
for jigdo_file in *.jigdo; do
   echo "Processing $jigdo_file..."

   # Use 'yes' to simulate continuous "Enter" presses for each file
   yes '' | jigdo-lite "$jigdo_file"
done

#!/bin/bash

# Directory containing all .jigdo and .template files
JIGDO_DIR=~/Desktop/Debian/

# Navigate to the Jigdo directory
cd "$JIGDO_DIR" || exit

# Loop through all .jigdo files and execute jigdo-lite, with exactly two "Enter" key presses
for jigdo_file in *.jigdo; do
   echo "Processing $jigdo_file..."

   # Use 'printf' to simulate pressing "Enter" twice for each file
   printf '\n\n' | jigdo-lite "$jigdo_file"
done

Choose the script version based on the expected interaction with Jigdo prompts. The yes command version is suitable for continuous responses, while the printf version provides a precise number of responses.

D-4) Execute the Script:

chmod +x ~/Desktop/download_isos.zsh
~/Desktop/download_isos.zsh

Replace with download_isos.sh for Bash.

Creating a Bootable SD Card for Debian Installation Using Windows and macOS

To facilitate Debian installation from an SD card, the ISO file must be properly written to the SD card. This guide provides a brief overview for Windows users utilizing Rufus, followed by detailed steps for macOS users using built-in tools.

Windows Instructions

Download Rufus from the official website. Rufus is a straightforward tool for creating bootable media from ISO files on Windows.

macOS Instructions (Using Built-In Tools)

This section provides detailed steps to create a bootable Debian SD card on macOS without third-party software.

(A) Identify the SD Card in Terminal:

Insert the SD card and open Terminal. Use the following command to display all drives and identify the SD card by its size:

diskutil list

Note the identifier for the SD card (such as /dev/disk9).

(B) Unmount and Erase the SD Card:

Unmount the SD card with the command:

diskutil unmountDisk /dev/disk9

Next, erase and format the SD card:

sudo diskutil eraseDisk FAT32 BOOT MBRFormat /dev/disk9

The command eraseDisk initiates the process of removing all existing data on the SD card. The FAT32 parameter specifies the filesystem to be used, ensuring compatibility across various operating systems. The label BOOT names the new partition, and MBRFormat sets the partition scheme to Master Boot Record, which is suitable for booting purposes.

/dev/disk0 (internal, physical):
#:                       TYPE NAME                    SIZE       IDENTIFIER
0:      GUID_partition_scheme                        *2.0 TB     disk0
1:             Apple_APFS_ISC Container disk1         524.3 MB   disk0s1
2:                 Apple_APFS Container disk3         2.0 TB     disk0s2
3:        Apple_APFS_Recovery Container disk2         5.4 GB     disk0s3

/dev/disk3 (synthesized):
#:                       TYPE NAME                    SIZE       IDENTIFIER
0:      APFS Container Scheme -                      +2.0 TB     disk3
                              Physical Store disk0s2
1:                APFS Volume Macintosh HD - Data     807.5 GB   disk3s1
2:                APFS Volume Macintosh HD            10.8 GB    disk3s3
3:              APFS Snapshot com.apple.os.update-... 10.8 GB    disk3s3s1
4:                APFS Volume Preboot                 12.3 GB    disk3s4
5:                APFS Volume Recovery                1.9 GB     disk3s5
6:                APFS Volume VM                      2.1 GB     disk3s6

/dev/disk4 (disk image):
#:                       TYPE NAME                    SIZE       IDENTIFIER
0:      GUID_partition_scheme                        +10.4 GB    disk4
1:                 Apple_APFS Container disk5         10.4 GB    disk4s1

/dev/disk5 (synthesized):
#:                       TYPE NAME                    SIZE       IDENTIFIER
0:      APFS Container Scheme -                      +10.4 GB    disk5
                              Physical Store disk4s1
1:                APFS Volume watchOS 10.5 21T575 ... 10.1 GB    disk5s1

/dev/disk6 (disk image):
#:                       TYPE NAME                    SIZE       IDENTIFIER
0:      GUID_partition_scheme                        +17.6 GB    disk6
1:                 Apple_APFS Container disk7         17.6 GB    disk6s1

/dev/disk7 (synthesized):
#:                       TYPE NAME                    SIZE       IDENTIFIER
0:      APFS Container Scheme -                      +17.6 GB    disk7
                              Physical Store disk6s1
1:                APFS Volume iOS 17.5 21F79 Simul... 17.0 GB    disk7s1

/dev/disk9 (internal, physical):
#:                       TYPE NAME                    SIZE       IDENTIFIER
0:     FDisk_partition_scheme                        *32.0 GB    disk9
1:             Windows_FAT_32 bootfs                  268.4 MB   disk9s1
2:                      Linux                         31.7 GB    disk9s2

(C) Convert the ISO to IMG Format:

macOS requires the ISO to be in .img format for the dd utility. Convert the Debian ISO by running:

hdiutil convert -format UDRW -o ~/Desktop/debian.img ~/Desktop/debian-12.7.0-arm64-DVD-1.iso

The hdiutil command-line tool is utilized for working with disk images in macOS. The UDRW format specifies an uncompressed read/write image, which is necessary for the subsequent dd operation. The output location is set to ~/Desktop/debian.img, and the source file is ~/Desktop/debian-12.7.0-arm64-DVD-1.iso.

If macOS appends .dmg to the output file (resulting in debian.img.dmg), this extension remains acceptable for the next step.

(D) Write the IMG to the SD Card Using dd:

Use dd to transfer the IMG file to the SD card. The dd utility performs a low-level copy of data from one location to another. The parameters used are:

• if=~/Desktop/debian.img.dmg: Specifies the input file.
• of=/dev/disk9: Specifies the output file (the SD card).
• bs=1m: Sets the block size to 1 megabyte, optimizing the copy process for speed.

The command to execute is:

sudo dd if=~/Desktop/debian.img.dmg of=/dev/disk9 bs=1m

Prior to executing this command, it is necessary to unmount the SD card using diskutil unmountDisk /dev/disk9. This ensures that no other processes are accessing the disk, preventing potential data corruption during the write operation.

Be cautious when using dd, as it can overwrite any specified drive without warning. This process can take a few minutes; no progress is shown by default.

(E) Eject the SD Card:

Once the process is complete, safely eject the SD card with:

diskutil eject /dev/disk9

The SD card is now prepared as a bootable Debian installation medium.

Configuring Debian to Use Local ISO Repositories with Fallback to Online Sources

Configuring Debian to utilize local ISO files as repositories enhances package management efficiency, particularly in environments with limited or unreliable internet connectivity. This guide outlines the process of mounting multiple ISO files, updating the package manager’s sources list to prioritize local repositories, and automating the mounting process for sustained convenience.

(A) Prepare Mount Directories

A fundamental step involves creating directories designated for mounting each ISO file. Organizing these directories under /media maintains system orderliness.

sudo mkdir -p /media/debian-iso{1..21}

• mkdir -p: Creates the specified directories along with any necessary parent directories.
• /media/debian-iso{1..21}: Generates directories named /media/debian-iso1 through /media/debian-iso21.

(B) Mount ISO Files Using a Bash Script

Automating the mounting process ensures efficiency when handling multiple ISO files. A Bash script is employed to mount each ISO to its corresponding directory as a loop device.

B-1) Create the Mount Script:

emacs ~/mount_debian_isos.sh -nw

#!/bin/bash

# Base directory where ISO files are located
ISO_DIR=/home/frank/Downloads
MOUNT_DIR=/media

# Loop through all 21 ISOs and mount them
for i in {1..21}; do
    ISO_FILE="$ISO_DIR/debian-12.7.0-arm64-DVD-$i.iso"
    MOUNT_POINT="$MOUNT_DIR/debian-iso$i"

    if [ -f "$ISO_FILE" ]; then
        echo "Mounting $ISO_FILE to $MOUNT_POINT..."
        sudo mount -o loop,ro "$ISO_FILE" "$MOUNT_POINT"
    else
        echo "Warning: $ISO_FILE does not exist."
    fi
done

• Shebang (#!/bin/bash): Specifies that the script should be executed in the Bash shell.
• Variables:
  • ISO_DIR: Directory containing the ISO files. Modify this path if the ISOs are stored elsewhere.
  • MOUNT_DIR: Base directory for mounting the ISOs.
• Loop: Iterates from 1 to 21, handling each corresponding ISO file.
• File Check: Verifies the existence of each ISO before attempting to mount.
• Mount Options:
  • loop: Mounts the ISO as a loop device.
  • ro: Mounts the ISO as read-only to prevent modifications.

B-2) Run the Script:

chmod +x ~/mount_debian_isos.sh
~/mount_debian_isos.sh    

sudo mount -t iso9660 -o loop /home/frank/Downloads/debian-9.5.0-amd64-DVD-1.iso /media/d1

sudo mount /dev/sdb1 /mnt/usb

sudo mount -t ntfs-3g /dev/sda1 /mnt/windows

sudo mount -t nfs 192.168.1.100:/shared-folder /mnt/nfs

sudo mount -t cifs -o username=frank,password=yourpassword //192.168.1.101/shared-folder /mnt/cifs

sudo mount -o ro /dev/sdc1 /mnt/readonly

sudo mount --bind /var/www/html /mnt/website

(C) Update sources.list to Include Local and Online Repositories

Configuring the APT package manager to prioritize local ISO repositories while retaining the ability to access online sources involves editing the sources.list file. The order of entries dictates the priority, with earlier entries being preferred.

C-1) Edit sources.list:

sudo emacs /etc/apt/sources.list

C-2) Add Repository Entries:

Append the following lines to the end of the file. Ensure that bookworm is the correct codename for the Debian release in use. Adjust accordingly if a different release is active.

# Local Debian ISO repositories
deb [trusted=yes] file:/media/debian-iso1/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso2/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso3/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso4/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso5/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso6/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso7/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso8/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso9/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso10/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso11/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso12/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso13/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso14/ bookworm main contrib	
deb [trusted=yes] file:/media/debian-iso15/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso16/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso17/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso18/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso19/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso20/ bookworm main contrib
deb [trusted=yes] file:/media/debian-iso21/ bookworm main contrib

# Online Debian repositories
deb http://deb.debian.org/debian bookworm main contrib
deb-src http://deb.debian.org/debian bookworm main contrib

deb http://deb.debian.org/debian-security bookworm-security main contrib
deb-src http://deb.debian.org/debian-security bookworm-security main contrib

deb http://deb.debian.org/debian bookworm-updates main contrib
deb-src http://deb.debian.org/debian bookworm-updates main contrib

• Local Repositories:
  • deb [trusted=yes] file:/media/debian-isoX/ bookworm main contrib: Specifies each local ISO as a trusted repository. The trusted=yes option bypasses signature verification; ensure ISOs are obtained from official sources to maintain security.
• Online Repositories:
  • Listed after local repositories to serve as fallback sources.
  • Both binary (deb) and source (deb-src) repositories are included as needed.
• Ordering: Local repositories are listed first to ensure APT searches them prior to online sources.

(D) Update the Package List

Refreshing the APT package database allows the system to recognize the newly configured repositories.

sudo apt update

This command updates the package index, enabling APT to acknowledge packages available from both local ISO repositories and online sources.

(E) Install Packages from Repositories

Packages can now be installed using APT, with the system prioritizing local ISO repositories before consulting online sources.

sudo apt install <package_name>

Replace <package_name> with the desired package to install.

apt-get install build-essential linux-headers-$(uname -r)
	
# Raspberry Pi
apt-get install build-essential gcc raspberrypi-kernel-headers raspberrypi-kernel
apt-get install build-essential emacs raspberrypi-kernel-headers git bc bison flex libc6-dev libncurses5-dev make
sudo apt install crossbuild-essential-armhf    

(F) Automate ISO Mounting on Startup (Optional)

To ensure that all ISO files are mounted automatically upon system boot, entries can be added to the /etc/fstab file. This guarantees that the local repositories are available without manual intervention each time the system starts.

F-1) Edit /etc/fstab:

sudo emacs /etc/fstab

F-2) Add Mount Entries:

Append the following lines to the end of the file.

# Mount Debian ISO repositories
~/Downloads/debian-12.7.0-i386-DVD-1.iso /media/debian-iso1 iso9660 loop,ro 0 0
~/Downloads/debian-12.7.0-i386-DVD-2.iso /media/debian-iso2 iso9660 loop,ro 0 0
# ...
~/Downloads/debian-12.7.0-i386-DVD-21.iso /media/debian-iso21 iso9660 loop,ro 0 0

• Source: Path to the ISO file.
• Mount Point: Directory where the ISO will be mounted.
• Filesystem Type (iso9660): Standard filesystem for ISO images.
• Mount Options:
  • loop: Mounts the file as a loop device.
  • ro: Mounts the ISO as read-only.
• Dump and Pass (0 0): Commonly set to 0 0 for ISO mounts, indicating no dump and no filesystem check.

F-3) Reload the Filesystem Mounts:

sudo mount -a

This command mounts all filesystems specified in /etc/fstab without necessitating a system reboot.

Linux, for Kernel-Level Device Driver

Linux Commands for Kernel and Device Programmers

1. System Information and Architecture

Before diving into device and kernel programming, it is crucial to understand the core system information. The following commands offer insights into the Linux system's hardware, kernel, and architecture.

1.1 System Information

The hostnamectl command provides a basic overview of the system, including the hostname, operating system, and kernel version.

# System Information
hostnamectl 

1.2 Architecture and Kernel Information

The uname command displays kernel and architecture information.

# Kernel Version
uname -r

# Machine Architecture
uname -m 

• uname -r: Displays the kernel version, which is essential for module development or custom kernel builds.
• uname -m: Reveals the machine architecture (e.g., x86_64 or arm64), ensuring compatibility with drivers and modules.

To check the Linux distribution and version details:

# Linux Distribution and Version
lsb_release -a 

1.3 CPU Information

Detailed CPU information can be obtained using lscpu or by inspecting /proc/cpuinfo.

# Human-readable CPU Information
lscpu

# Detailed CPU Information
cat /proc/cpuinfo 

• lscpu: Provides CPU architecture, cores, threads, and cache sizes in a human-readable format.
• cat /proc/cpuinfo: Offers detailed CPU information, useful for performance benchmarking when developing CPU-bound kernel modules.

2. Device and Driver Information

Linux exposes detailed information about devices and their drivers through the /sys and /proc filesystems, as well as utility commands.

2.1 Listing Devices and Drivers

To check the drivers associated with devices and view system buses:

# List PCI Devices with Kernel Modules
lspci -k

# View Input Devices
cat /proc/bus/input/devices 

• lspci -k: Lists PCI devices along with their associated kernel modules.
• cat /proc/bus/input/devices: Displays information about input devices connected to the system.

2.2 Exploring the /sys Filesystem

The /sys filesystem provides a way to interact with kernel objects.

For example, to check the status of a network interface:

# Network Interface Status
cat /sys/class/net/eth0/operstate 

To list available filesystems supported by the kernel:

# Available Filesystems
cat /proc/filesystems 

3. Hardware and Device Inspection

Understanding the hardware landscape and relationships between different buses and devices is essential for advanced device programming.

3.1 PCI Devices

The lspci command is used to inspect PCI devices:

# PCI Devices in Tree View
lspci -tv

# PCI Devices with Kernel Modules
lspci -k

# Verbose PCI Device Information
lspci -vv

# Filter PCI Devices (e.g., USB Controllers)
lspci -v | grep USB 

• lspci -tv: Shows a hierarchical tree of PCI devices and their connections.
• lspci -k: Links kernel modules to devices, verifying which drivers are in use.
• lspci -vv: Provides detailed device information, such as capabilities, power management, and interrupt settings.
• lspci -v | grep USB: Filters specific device classes, like USB controllers.

3.2 USB Devices

For USB device driver development:

# USB Devices in Tree View
lsusb -tv

# Verbose USB Device Information
lsusb -v

# Inspect Specific USB Device
lsusb -d <vendor_id:product_id> 

• lsusb -tv: Provides a tree view of connected USB devices.
• lsusb -v: Gives verbose information about each USB device, including descriptors, vendor IDs, and product IDs.
• lsusb -d <vendor_id:product_id>: Inspects a specific USB device.

For debugging or interacting with USB devices programmatically, tools like usbmon and Wireshark can capture USB traffic.

3.3 Block Devices

To view block devices such as disks and partitions:

# List Block Devices
lsblk

# List Block Devices with Filesystem Information
lsblk -f 

• lsblk: Provides a clear hierarchical structure of block devices.
• lsblk -f: Adds filesystem and UUID information, useful for understanding mounted devices and their properties.

3.4 SCSI Devices

For low-level inspection of SCSI devices:

# List SCSI Devices
lsscsi 

• lsscsi: Lists all SCSI devices connected to the system.

4. Interrupts and I/O Monitoring

Tracking interrupts and I/O performance is vital for optimizing system interactions with hardware.

4.1 Monitoring Interrupts

To view the number of interrupts per CPU for each I/O device:

# Interrupts per Device
cat /proc/interrupts 

This is useful when optimizing interrupt handling or diagnosing hardware IRQ conflicts.

4.2 Monitoring I/O Performance

The iostat and iotop commands help monitor I/O device performance:

# Extended I/O Statistics (updates every second)
iostat -x 1

# Block Device Statistics
iostat -d 1

# Monitor I/O Usage by Process
iotop 

• iostat -x: Provides extended statistics like utilization, throughput, and wait time.
• iostat -d: Focuses on block devices, updating statistics every second.
• iotop: Displays real-time I/O usage by process, useful for identifying processes performing heavy I/O operations.

5. Memory and Cache Information

Memory management and caching are critical when programming kernel modules or drivers.

5.1 Memory Information

To view detailed memory statistics:

# Memory Information
cat /proc/meminfo 

This provides information such as total memory, free memory, available swap, and cached memory.

5.2 Cache Information

To list memory cache information:

# Cache Information
sudo lshw -C memory 

• lshw -C memory: Displays information about the system's memory hierarchy, including caches.

6. Detailed Hardware Information

Delving into specific hardware capabilities is facilitated by tools for inspecting buses, devices, and subsystems.

6.1 Using lshw

The lshw command provides detailed information about the hardware configuration.

# Hardware Path Tree View
sudo lshw -short

# Class-specific Information
sudo lshw -short -C bus
sudo lshw -short -C cpu
sudo lshw -short -C storage

# Bus Information
sudo lshw -businfo 

• lshw -short: Provides a concise list of hardware.
• lshw -short -C [class]: Focuses on specific classes like bus, cpu, or storage.
• lshw -businfo: Shows how devices are connected to various buses.

7. Device Query and Configuration

Device-specific information can be extracted using hwinfo and other utilities.

7.1 Filesystems

To list all supported filesystems in the kernel:

# Supported Filesystems
cat /proc/filesystems 

This is useful when developing storage drivers or working with filesystems.

7.2 Using hwinfo

The hwinfo command provides detailed information about hardware components.

# Concise Hardware Summary
sudo hwinfo --short

# Detailed USB Information
sudo hwinfo --usb 

• hwinfo --short: Gives a concise summary of all detected devices.
• hwinfo --usb: Provides detailed information about USB devices, including vendor and product IDs.

8. SCSI and Disk Management

For storage systems or SCSI devices, in-depth tools are necessary to inspect and configure devices.

8.1 SCSI Devices

To show the hierarchy of SCSI devices:

# SCSI Device Hierarchy
lsblk -s 

• lsblk -s: Displays block devices in reverse dependencies, showing the relationship between devices.

8.2 Disk Performance Monitoring

To obtain detailed SMART information for storage devices:

# SMART Information
sudo smartctl -a /dev/sda 

• smartctl: Provides disk health, performance data, and potential failure indicators.

9. Monitoring Devices in Real-Time

Real-time monitoring of device and kernel activity is essential for performance tuning and debugging.

9.1 Process Monitoring

The top or htop commands can be used to monitor processes and system load:

# Real-time Process Monitoring
top

# Enhanced Process Monitoring
htop 

• top: Displays system processes and resource usage.
• htop: An interactive process viewer with more details and a user-friendly interface.

9.2 Block Device Monitoring

To monitor block device I/O:

# Block Device I/O Statistics
iostat -d 

• iostat -d: Shows I/O statistics for block devices.

9.3 Network Monitoring

To monitor network interfaces:

# Real-time Network Interface Monitoring
iftop 

• iftop: Displays bandwidth usage on network interfaces.

10. Performance Monitoring

Tools like perf are used to monitor kernel and application performance, helping identify bottlenecks.

10.1 Using perf

To monitor CPU performance, system calls, and events:

# Live CPU Performance Monitoring
sudo perf top 

To record a performance profile:

# Record Performance Data
sudo perf record -a 

To display the recorded performance data:

# Report Performance Data
sudo perf report 

• perf top: Provides a real-time view of system performance.
• perf record: Collects performance data over time.
• perf report: Analyzes and displays the collected data.

11. Kernel Configuration and Modules

Kernel modules extend system functionality without requiring a reboot and are commonly used in device driver development.

11.1 Listing Kernel Modules

To list all loaded kernel modules:

# List Loaded Kernel Modules
lsmod 

11.2 Loading and Unloading Modules

To load a kernel module manually:

# Load a Kernel Module
sudo modprobe <module_name> 

To unload a kernel module:

# Unload a Kernel Module
sudo modprobe -r <module_name> 

11.3 Module Information

To get detailed information about a specific module:

# Module Information
modinfo <module_name> 

This provides information such as module parameters, dependencies, and author.

11.4 Installing Kernel Headers

Kernel headers are required when building or debugging modules:

# Install Kernel Headers
sudo apt-get install linux-headers-$(uname -r) 

12. Kernel Logs and Debugging

Kernel logs are essential for tracking system errors and debugging device driver issues.

12.1 Viewing Kernel Logs

To view real-time kernel logs:

# Real-time Kernel Logs
dmesg -w 

Access kernel logs through the systemd journal:

# Kernel Logs via systemd journal
journalctl -k 

12.2 Filtering Logs

To filter logs and focus on specific messages:

# Filter Kernel Logs
dmesg | grep <keyword> 

Replace <keyword> with the module name or any relevant term to isolate specific log messages.

13. Kernel Modules and Hardware Interaction

Monitoring how kernel modules interact with hardware is crucial for device driver development.

13.1 Tracking Module Messages

To locate messages related to a specific kernel module:

# Module-related Kernel Messages
dmesg | grep <module_name> 

This assists in debugging issues like module loading or initialization.

14. Building and Inserting Kernel Modules

Kernel module development often requires building, inserting, and testing custom modules.

14.1 Compiling Kernel Modules

To compile the kernel module for the currently running kernel:

# Compile Kernel Module
make -C /lib/modules/$(uname -r)/build M=$(pwd) modules 

14.2 Inserting and Removing Kernel Modules

To insert the compiled module into the running kernel:

# Insert Kernel Module
sudo insmod mymodule.ko 

To remove the kernel module:

# Remove Kernel Module
sudo rmmod mymodule 

Note: Use the module name without the .ko extension when removing.

15. Debugging Kernel Modules

For kernel module developers, debugfs and ftrace are invaluable for exposing internal kernel data and tracing function calls.

15.1 Mounting debugfs

To mount the debugfs filesystem:

# Mount debugfs
sudo mount -t debugfs none /sys/kernel/debug 

15.2 Using ftrace

To trace function calls or events in the kernel:

# Set the Current Tracer to 'function'
echo function | sudo tee /sys/kernel/debug/tracing/current_tracer 

To view the trace output:

# View Tracing Output
cat /sys/kernel/debug/tracing/trace 

This allows tracing of function calls, which is invaluable for kernel debugging.

16. Kernel Parameters and Tuning

Kernel parameters can be inspected and tweaked using the /proc/sys directory or via the sysctl command.

16.1 Viewing Kernel Parameters

To view all kernel parameters:

# View All Kernel Parameters
sysctl -a 

16.2 Modifying Kernel Parameters

To modify a kernel parameter (e.g., increasing the maximum number of open files):

# Increase Maximum Open Files
sudo sysctl -w fs.file-max=100000 

To make the change permanent, add the parameter to /etc/sysctl.conf.

17. Building Custom Kernels and Modules

Building custom kernels or kernel modules is sometimes necessary when developing for the Linux kernel.

17.1 Compiling the Kernel

To configure and build a custom kernel:

# Configure Kernel Options
make menuconfig

# Compile the Kernel
make

# Install the Kernel and Modules
sudo make modules_install install 

17.2 Compiling and Inserting Custom Kernel Modules

For building and inserting custom kernel modules:

# Compile Kernel Module
make -C /lib/modules/$(uname -r)/build M=$(pwd) modules

# Insert Kernel Module
sudo insmod mymodule.ko

# Remove Kernel Module
sudo rmmod mymodule 

Linux Kernel Programming and Module Compilation on Raspberry Pi

Linux kernel programming on the Raspberry Pi presents unique challenges and considerations compared to traditional desktop or server environments. The Raspberry Pi, being a single-board computer based on the ARM architecture, requires specific approaches for kernel development, module compilation, and device driver integration. This guide explores the differences, methodologies, and best practices for effective kernel programming on the Raspberry Pi.

(A) Differences in Linux Kernel Programming on Raspberry Pi

• Processor Architecture:
  • The Raspberry Pi utilizes ARM-based processors, whereas many traditional systems use x86 architecture.
  • ARM architecture has different instruction sets, calling conventions, and memory management techniques.
• Endianness:
  • ARM processors can be either little-endian or big-endian; Raspberry Pi uses little-endian mode.
• Hardware Resources:
  • Limited resources such as CPU speed, memory, and storage compared to desktop systems.
  • Specific hardware peripherals like GPIO, SPI, I2C, and UART interfaces.

• Definition:
  • Cross-compilation involves compiling code on a host system (typically x86) to run on a target system (ARM-based Raspberry Pi).
• Necessity:
  • The Raspberry Pi's limited resources make native compilation time-consuming.
  • Cross-compiling on a more powerful host system significantly reduces build times.
• Toolchains:
  • Specialized cross-compilation toolchains like GCC for ARM are required.
  • Toolchains must match the target architecture and ABI (Application Binary Interface).

• Customized Kernels:
  • The Raspberry Pi often uses customized kernels provided by the Raspberry Pi Foundation.
  • These kernels include specific drivers and configurations for Raspberry Pi hardware.
• Upstream vs. Downstream Kernels:
  • Upstream kernels are the mainline Linux kernels from kernel.org.
  • Downstream kernels are modified versions tailored for the Raspberry Pi.
• Compatibility:
  • Modules and drivers must be compatible with the specific kernel version in use.

(B) Module Compilation on Raspberry Pi

• Obtaining Kernel Headers:
  • Kernel headers matching the running kernel are necessary for module compilation.
  • Install using package managers:

    sudo apt-get install raspberrypi-kernel-headers

• Installing Required Tools:
  • Essential tools include build-essential, gcc, make, and git.

• Setting Up Cross-Compiler:
  • Install an ARM cross-compiler on the host system.
  • For Debian-based systems:

    sudo apt-get install gcc-arm-linux-gnueabihf

• Configuring the Build Environment:
  • Set environment variables to point to the cross-compiler and kernel sources.
  • Example Makefile adjustments:

    ARCH=arm
    CROSS_COMPILE=arm-linux-gnueabihf-
    KERNELDIR=/path/to/kernel/sources

• Building the Module:
  • Use the make command with appropriate flags:

    make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-

• Transferring Modules:
  • Copy the compiled .ko module file to the Raspberry Pi.
• Inserting the Module:
  • Use insmod or modprobe:

    sudo insmod module_name.ko

• Removing the Module:
  • Use rmmod:

    sudo rmmod module_name

• Verifying Module Load:
  • Check with lsmod and dmesg for logs.

(C) Kernel Compilation on Raspberry Pi

• Customization:
  • Enabling or disabling specific features and drivers.
• Performance Optimization:
  • Tailoring the kernel for specific applications or workloads.
• Testing and Development:
  • Developing custom kernel modules or patches.

• Cloning from GitHub:
  • The Raspberry Pi kernel sources are available on GitHub.

    git clone --depth=1 https://github.com/raspberrypi/linux

• Selecting the Correct Branch:
  • Ensure the branch matches the Raspberry Pi model and desired kernel version.

• Using Default Configurations:
  • Start with the default Raspberry Pi configuration.

    cd linux
    KERNEL=kernel7
    make bcm2709_defconfig

• Customizing Configurations:
  • Use menuconfig or xconfig:

    make menuconfig

• Configuration Options:
  • Enable or disable modules, set kernel parameters, and optimize settings.

• Compiling the Kernel:
  • Use make with appropriate flags for cross-compilation.

    make -j4 zImage modules dtbs

• Installing Modules and Firmware:
  • Modules:

    sudo make modules_install

  • Device Tree Blobs and Kernel Image:

    sudo cp arch/arm/boot/zImage /boot/kernel7.img
    sudo cp arch/arm/boot/dts/*.dtb /boot/
    sudo cp arch/arm/boot/dts/overlays/*.dtb* /boot/overlays/
    sudo cp arch/arm/boot/dts/overlays/README /boot/overlays/

• Rebooting:
  • Restart the Raspberry Pi to load the new kernel.