This is a guide to the terminal, the raw essence of the computing experience. I cover four of its tools: Croc for transferring files between machines, Gallery-DL for downloading image galleries, ImageMagick for batch processing those images once you have them, and Wget for archiving websites.
The terminal predates everything you know about your computer, even down to the mouse. Instead of clicking buttons, you type commands and press enter to execute them. If you type erroneously or your command otherwise fails, the terminal will let you know. Your commands tend to follow their own patterns, many of which are simple abbreviations of words such as "concatenate" and "directory". Every computer of every operating system has its own terminal, and you can use it to make system-wide changes, some of them devastating if you're not prudent. It is a tool that rewards patience and offers a reprieve from the overstimulation of contemporary GUIs.
Croc allows you to transfer files directly and securely between two computers, anywhere on the internet. It is cross-platform, end-to-end encrypted, needs no port forwarding or IP address configuration, does not even require you to be on the same network and has no file size limit.
File sharing services are convenient until they are not. You do not know who runs them, what they consider acceptable to host, or more to the point, what kind of files they had in mind when they built the thing. Croc sidesteps all of that. Your file goes directly from one machine to the other, end-to-end encrypted, with no server in the middle holding a copy. It is just you and whoever you are sending to.
sudo apt install crocsudo pacman -S crocsudo dnf install croccurl https://getcroc.schollz.com | bashbrew install croccurl https://getcroc.schollz.com | bashsudo port install crocnix-shell -p crocchoco install croccurl https://getcroc.schollz.com | bashscoop install crocwinget install schollz.crocYou can test the whole send/receive flow on a single machine using two terminal windows side by side. No second computer needed, though when you get to that it will need its own installation of croc.
Create a test file:
Linux/macOS: echo "Hello from croc!" > test.txt
Windows: echo Hello from croc! > test.txt
In Terminal 1, run: croc send test.txt
croc will display a code phrase like: 7776-gorilla-london-turtle
Copy it.
In Terminal 2, navigate to where you want the file to land. If you are not sure where your terminal is pointing, run pwd first to check.
Run croc followed by the code phrase.
Type y to confirm. Once it reaches 100%, the file is in Terminal 2's current directory.
If that worked, you are ready to send actual files. The code phrase is single-use, so croc will generate a new one each time. Make sure the sender starts first and shares the code before the receiver tries to connect.
Ways to specify what to send:
By name from the current folder: croc send filename.txt
By full path from anywhere: croc send ~/Pictures/mypicture.jpg
An entire folder: croc send ~/MyFolder
With a custom code phrase: croc send --code "mycustomcode" filename.txt
croc send (with a trailing space) into the terminal, then drag the file from your file manager into the terminal window. The path will paste automatically. Press Enter to send.Navigate to the folder where you want the file to land: cd ~/Downloads
Run croc followed by the code phrase the sender gives you.
Type y to confirm. Once it reaches 100%, the file is in your current folder.
~/ or /tmp and moving the file afterwards. If a file with the same name already exists, croc will not overwrite it silently, so delete the old one with rm filename first.Send a file: croc send filename.txt
Send with a custom code: croc send --code "mycode" filename.txt
Receive a file: croc code-phrase-here
Receive to a specific folder: cd ~/TargetFolder && croc code-phrase-here
Send a file: croc send filename.txt
Send with a custom code: croc send --code "mycode" filename.txt
Receive a file: croc code-phrase-here
Receive to a specific folder: cd C:\TargetFolder && croc code-phrase-here
You can now transfer files directly between machines without routing them through anyone else's server. For a full list of options, run croc --help or visit the croc GitHub page at https://github.com/schollz/croc.
Gallery-DL is the main terminal tool for downloading images from hundreds of galleries and boorus. Whether you want to pull albums off Flickr, browse through a deviantArt gallery, or add to your niche porn collection, you heard that last one right, it handles all of it. Once you have rigged the script to your liking, you can tab out the window, do your other tasks and come back to a full archive.
python3 -m pip install -U gallery-dlsudo pacman -S gallery-dlsudo dnf install gallery-dlpython3 -m pip install -U gallery-dlbrew install gallery-dlsudo port install gallery-dlnix-shell -p gallery-dlpython3 -m pip install -U gallery-dlscoop install gallery-dlchoco install gallery-dlpython3 -m pip install -U gallery-dlThe basic syntax is simple. Open your terminal and pass a URL: gallery-dl "https://www.example.com/gallery/URL"
gallery-dl will automatically detect the site, find all images in the gallery, and download them. By default, files are saved to a gallery-dl/ subfolder inside your current directory, organised by site and username.
If you hit a rate limit, go down to Step 3 to learn how to handle them. If you have not yet, you can proceed as normal to Step 2.
Download a gallery or user page: gallery-dl "URL"
Download to a specific folder (Linux and macOS): gallery-dl -d ~/Pictures/gallery-dl "URL"
Download to a specific folder (Windows): gallery-dl -d "C:\Users\YourUsername\Pictures\gallery-dl" "URL"
List the URLs that would be downloaded (without downloading): gallery-dl -g "URL"
See what keywords are available for a URL (useful for custom filenames): gallery-dl -K "URL"
Username and password (command line): gallery-dl -u "yourusername" -p "yourpassword" "URL"
See all available options: gallery-dl --help
Most websites do not like having their servers pounded by legions of robots.
Because we are the robots for once, this is a bit of a problem. Thankfully this requires one minimal input in addition to your existing configuration, which goes as such:
--sleep 10-25
Websites know that bots are hitting them, but they will tolerate that activity as long as it is not too aggressive. And by default, as you may have seen in your test run, gallery-dl is quite aggressive. So it is not enough to just work with the normal settings. You will want to download larger galleries at better qualities, and you will eventually hit this one way or another.
Rate-limiting is essentially how a website protects itself from bad actors, excessive bandwidth use and potential downtime. Every time your browser asks for something, whether that is an image, a page, or anything else, that counts as a request. If you send too many, whether you are a human or a robot, the site will start pushing back.
Websites use a few different techniques for this. They may track your IP address, your login session, your browser session, or even an API key. If you exceed a certain threshold of requests per minute, you will get hit with a limit.
Some sites also use what is called a sliding window or token bucket system. If you are familiar with LLMs, you will know exactly what this is. The service will allow you a certain number of requests, and if you go through them too quickly, you have to wait for it to refill. This is why large bursts cause problems, whereas spacing things out over time does not. The latter method does not hit the same limits in the same way.
Now, while gallery-dl does allow you to retry if 429s happen, this particular method avoids that entirely. I have downloaded thousands of pictures using this approach and have not encountered a 429 since. You do not want to be working in bursts followed by five minute wait times. I have tried that, and it does not work. It does not matter how long you wait between bursts, the moment you start hammering the server again, it will push back with rate limits.
What this random timer helps to achieve is a more organic appearance. By resembling a human user, you do not trip the server's bot detection. I went with 10 to 25 to begin with, but you can alter this however you see fit.
Also, you might see some guides suggesting the use of accounts or API keys. I have deliberately avoided that here. Archiving is a free activity by default, and if you do not want to be tracked or tied to an account, it is just not necessary for this setup. An ounce of prevention, and all that.
It will expect an e-mail. Bothersome, but if you have a legion of burners, it's a small price to pay for its collection. Here's an example code you can use to get things off there:
gallery-dl --sleep 2.0-6.0 --sleep-request 10.0-25.0 --sleep-429 120 --user-agent "gallery-dl/1.0 (https://github.com/mikf/gallery-dl; EMAIL ADDRESS HERE)" "URL HERE"You can now download image galleries from the command line. For a full list of configuration options, see the gallery-dl configuration documentation at https://gdl-org.github.io/docs/configuration.html.
ImageMagick is a free, open-source software suite used for editing and manipulating digital images. It can work with a vast range of image formats and can be fully automated through command-line tools, making it perfect for batch processing large numbers of images. This guide covers installation, your primary workflow, and a comprehensive command reference.
Download ImageMagick from https://imagemagick.org/script/download.php. Look for the Windows binary installer, typically named something like: ImageMagick-7.x.x-x-Q16-x64-dll.exe
Run the installer and follow the defaults. When asked about "Install development headers and libraries", you can skip this unless you plan to use ImageMagick as a library in other programs.
Most importantly: check the box for "Add application directory to your system path". This lets you use the magick command from any terminal without typing the full path.
Complete the installation and restart your terminal (or computer) for the PATH changes to take effect.
Verify installation by opening PowerShell and typing: magick -version. You should see output showing the ImageMagick version and supported formats.
If you have Homebrew installed, the simplest method is: brew install imagemagick
Otherwise, download the macOS installer from https://imagemagick.org/script/download.php and run it.
Verify with: magick -version
Most Linux distributions include ImageMagick in their package manager. For Ubuntu/Debian: sudo apt-get install imagemagick
For Fedora/RHEL: sudo dnf install ImageMagick
Verify with: magick -version
Your primary use case is reducing colors in JPEGs using Floyd-Steinberg dithering. This command does exactly that:
magick *.jpg -colors 256 -dither FloydSteinberg output_%d.jpg
Here is what each part of this command does:
magick : the main ImageMagick command. Use this in version 7+. Version 6 uses convert instead.
*.jpg : process all JPEG files in the current directory.
-colors 256 : reduce the image to 256 colors maximum.
-dither FloydSteinberg : use Floyd-Steinberg dithering, which creates a better visual result than solid color blocks.
output_%d.jpg : save processed images as output_0.jpg, output_1.jpg, and so on.
ImageMagick commands follow this general pattern:
magick [input files] [operations] [output file]
Operations are applied in sequence, so order matters. For example:
magick image.jpg -resize 800x600 -quality 85 output.jpg
This reads image.jpg, resizes it, reduces quality to 85%, then saves it as output.jpg.
ImageMagick excels at batch processing. A few useful patterns:
Process all JPEGs in a folder: magick *.jpg -resize 50% output_%d.jpg
Process all images of any format: magick * -resize 50% output_%d.jpg
Process a specific range of numbered files: magick image_[0-5].jpg -resize 50% resized_%d.jpg
Use the adjoin option to combine into one file (if format supports it): magick *.jpg -adjoin output.pdf
Before processing, you might want to know details about your images:
magick identify image.jpg
This shows dimensions, color depth, file size, and other metadata. Add -verbose for exhaustive details.
Resize to fit within 800x600, preserving aspect ratio: magick input.jpg -resize 800x600 output.jpg
Resize to exact dimensions (will distort if needed): magick input.jpg -resize 800x600! output.jpg
Resize by percentage: magick input.jpg -resize 50% output.jpg
Crop a specific area (width x height + offset_x + offset_y): magick input.jpg -crop 500x400+100+50 +repage output.jpg
Auto-trim whitespace or borders: magick input.jpg -trim output.jpg
Increase brightness and contrast: magick input.jpg -brightness-contrast 15x25 output.jpg
Convert to grayscale: magick input.jpg -colorspace Gray output.jpg
Invert colors (create a negative): magick input.jpg -negate output.jpg
Blur an image: magick input.jpg -blur 0x10 output.jpg
Sharpen an image: magick input.jpg -sharpen 0x2 output.jpg
Add a border: magick input.jpg -border 5x5 -bordercolor black output.jpg
Create a shadow effect: magick input.jpg -shadow 50x3+5+5 output.jpg
Standard command with 256 colors: magick *.jpg -colors 256 -dither FloydSteinberg output_%d.jpg
No dithering, posterized: magick *.jpg -colors 256 -dither None output_%d.jpg
Alternative dither pattern: magick *.jpg -colors 256 -dither Riemersma output_%d.jpg
Reduce to fewer colors for retro effect: magick *.jpg -colors 16 -dither FloydSteinberg output_%d.jpg
Combine with resize for smaller files: magick *.jpg -resize 800x600 -colors 256 -dither FloydSteinberg output_%d.jpg
Convert all PNGs to JPEGs: magick *.png -quality 85 converted_%d.jpg
Convert with quality setting (1 to 100, default is 92): magick input.jpg -quality 75 output.jpg
Convert PNG to JPG: magick input.png output.jpg
-resize geometry : resize image, preserving aspect ratio. Example: -resize 800x600
-crop geometry : crop to specified area. Example: -crop 500x400+50+50
-scale geometry : fast resize, lower quality than -resize. Example: -scale 50%
-trim : remove borders and whitespace.
-extent geometry : expand canvas to size. Example: -extent 1000x800
-colors number : reduce to N colors (quantize). Example: -colors 256
-dither type : apply dithering algorithm. Example: -dither FloydSteinberg
-colorspace type : change color space. Example: -colorspace Gray
-negate : invert colors.
-brightness-contrast val : adjust brightness and contrast. Example: -brightness-contrast 10x20
-modulate brightness,saturation,hue : adjust HSB values as percentages. Example: -modulate 120,100,100
-normalize : auto-normalize contrast.
-blur radiusxsigma : apply Gaussian blur. Example: -blur 0x10
-sharpen radiusxsigma : sharpen image. Example: -sharpen 0x2
-edge radius : detect edges. Example: -edge 3
-emboss radius : 3D emboss effect. Example: -emboss 2
-despeckle : remove noise.
-posterize levels : reduce color levels. Example: -posterize 4
-rotate degrees : rotate image. Example: -rotate 90
-flip : mirror vertically (top to bottom).
-flop : mirror horizontally (left to right).
-transpose : rotate 90 degrees then mirror.
-swirl degrees : swirl or vortex effect. Example: -swirl 60
-quality percent : JPEG compression quality, 1 to 100. Example: -quality 85
-density xxy : DPI for PDF or vector input. Example: -density 150x150
-strip : remove all profiles and metadata.
-interlace type : progressive JPEG or PNG. Example: -interlace Plane
-adjoin : join images into single file. Example: -adjoin output.pdf
identify -verbose : get detailed image info. Example: identify -verbose image.jpg
-verbose : output detailed processing info.
-define key=value : set internal format options. Example: -define jpeg:dct-method=float
You can chain multiple operations together. They execute left to right:
magick input.jpg -resize 50% -blur 0x5 -colors 128 -dither FloydSteinberg output.jpg
This resizes, blurs, reduces colors, applies dithering, and saves, all in one command.
You now know how to bulk edit images from the command line. Online dithering tools are useful for a single image, but ImageMagick operates on entire folders at once, gives you far more control over the output, and does not require uploading your files anywhere. If you have used something like Dithering.app before, ImageMagick can do everything it does and considerably more.
Wget is the premier tool for downloading entire websites to your machine. It is a staple of Linux and has persisted since 1996. As a result it is heavily documented, has decades worth of versions and is an invaluable asset to your efforts.
sudo apt install wgetsudo pacman -S wgetsudo dnf install wgetbrew install wgetsudo port install wgetnix-shell -p wgetwinget install GnuWin32.Wgetscoop install wgetchoco install wgetThe following command will download a full local copy of a website, converting all links so they work offline:
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent URL
Replace URL with the address of the site or page you want to mirror. Here is what each flag does:
--mirror : enables mirroring mode, equivalent to turning on recursive downloading, timestamping, and infinite recursion depth all at once. wget will follow links and pull the full structure of the site.
--convert-links : after downloading, converts all links in the HTML to point to the local files instead of the original web addresses. This is what makes the archived site browsable offline: clicking a link opens the local copy rather than trying to fetch from the internet.
--adjust-extension : adds the correct file extension (e.g. .html) to downloaded files that do not already have one, so your browser knows how to open them.
--page-requisites : downloads everything needed to display each page correctly: images, stylesheets, scripts, and other assets, even if they are hosted on a different domain.
--no-parent : restricts downloading to the URL you specified and anything below it in the directory tree. Without this, wget could wander up to the site's root and start pulling the entire domain. This keeps the download focused.
By default, wget saves files into the current directory, organised into subfolders named after the domain.
cd ~/Documents/Archives
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent URL
Or specify an output directory directly with -P:
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent -P ~/Documents/Archives URL
The download is taking forever or pulling too much : some sites are enormous. You can limit recursion depth with --level=N (e.g. --level=2 to only go two links deep) or add a wait between requests with --wait=1 to be polite to the server and avoid being rate-limited.
wget is being blocked by the server : some servers reject requests that do not look like a browser. Try spoofing a user agent by adding --user-agent="Mozilla/5.0" to your command.
Links still point to the live site after downloading : make sure you included --convert-links. If you forgot it, you can re-run the command with -c (--continue) to resume and only fetch what is missing.
Some assets are missing from the local copy : if images or styles are hosted on a different domain (a CDN, for instance), --page-requisites should catch them. However, some sites load assets dynamically via JavaScript, which wget cannot execute. For heavily JavaScript-dependent sites, a headless browser tool will produce better results than wget alone.
You can now mirror websites to your local machine. For a full list of wget options, run wget --help or consult the GNU wget manual at https://www.gnu.org/software/wget/manual/.
You've got a bank of powerful terminal commands, workarounds for common problems and your less technically-inclined friends will regard you as a computer wizard. From here you can try more programs and enjoy the smooth, lightweight experience of a computer's command line. You might only use it for scraping, or you might make the jump to a terminal-based workflow. People constantly develop for the terminal environment, so you're never going to be out of options. CLI programs have been here since the 1960s, so you're also not out of history. Thanks for reading, and happy scraping!
| Title | Description |
|---|---|
| Beautiful Soup | My HTML Python scraper of choice. |
| Croc (GitHub) | The official repository for Croc, including documentation and releases. |
| Explain Shell | Paste any command and it breaks down every flag against the actual man page. |
| fzf (GitHub) | A fuzzy finder for the terminal that pairs well with almost everything. |
| Gallery-DL (GitHub) | The official repository for Gallery-DL, including the full list of supported sites. |
| GNU Coreutils Manual | The authoritative reference for the standard Unix tools. |
| GNU Wget | The official GNU Wget page, including the full manual. |
| ImageMagick | The official ImageMagick website, with documentation and download links. |
| Ripgrep (GitHub) | A faster grep, worth knowing. |
| Terminal Trove | A curated directory of terminal tools and CLI programs. |
| The Art of Command Line | A dense, practical single-page reference for command line fluency. |
| TLDR Pages | Community-maintained cheat sheets for terminal commands, far more readable than man pages. |
| yt-dlp (GitHub) | The video equivalent of Gallery-DL, and just as capable. |