James Ramsay

I use Arq to backup my computer to Dropbox (off-site backup) and a Synology NAS (on-site backup). The most obvious approach is to use a folder stored on your NAS. I previously explained how using Autofs can make this easier, but I’ve still found it unreliable and slow.

Arq calculates checksums when performing a backup to verify uploads, and when validating backup data every 60 days. When using the folder approach, Arq needs to download the data to calculate a checksum. S3-compatible storage, however, can calculate the checksum, allowing this work to be offloaded to the NAS making backups and validation faster.

Arq supports S3-compatible storage including MinIO, which can be run on a Synology NAS using Docker. Unfortunately, only Synology devices with an x64 CPU can run Docker. My DS216play has an ARMv7 CPU, so I can’t use the Docker package.

Here’s how you can compile MinIO to run natively on an ARMv7 Synology NAS without Docker. 🎉

Compile MinIO

Before you begin, you need Git and Docker to be installed.

Clone the MinioSPK repository, which contains build scripts for creating a Synology MinIO package.

# Clone the project
git clone https://github.com/darkmuggle/MinioSPK.git

# Change directory
cd MinioSPK

Two commands are needed to compile MinIO.

# Download the MinIO source and logos
make init

# Build the MinIO package for Synology
make arm7

Install MinIO on our Synology NAS

With our newly built package, we are ready to install MinIO.

From Synology DiskStation, the web interface for managing your Synology NAS, open Package Center.

Next, from the Installed section (top left), select Manual Install (top right) to launch the installation and upgrade wizard.

Click Browse to locate the package you compiled, likely similar to minio-2020.04.04-arm-7.spk. Then click Next.

After a short while you’ll be shown a warning because package hasn’t been signed. You will need to click Yes to proceed.

The wizard will prompt you to set the MINIO_ACCESS_KEY and MINIO_SECRET_KEY, and then install MinIO. Once installed you will be able to open MinIO from Package Center. Use the access key and secret key you set during the installation to login.

Configure Arq

Now that you’ve installed MinIO, you can use the guide on the Arq blog Backup to Synology NAS using Minio and Arq.

(Summarized from my post for GitLab, One simple trick to make your screenshots 80% smaller)

When you capture a screenshot on your Mac, it will be saved in the PNG-32 format, with support for 16 million distinct colors and transparency. This means that the screenshot will perfectly capture every pixel on your screen, but having four 8-bit channels for red, green, blue and alpha (transparency) for every pixel makes the file very large. If you’re interested, you can verify this yourself using pngcheck.

In practice, the subjects of my screenshots are buttons and forms, not photographs. Since we don’t need 16 million colors, we can take advantage of the PNG-8 format with it’s more compact 256 color palette.

Lossy Compression: Color Quantization

The first step is to reduce the color palette of the screenshot. This is a type of lossy compression called color quantization, which will reduce the number of distinct colors in the image. The pngquant command line utility is the perfect tool for this job, and if you’ve used the popular ImageAlpha tool, you’ve already used the pngquant library.

# Install pngquant using Homebrew
brew install pngquant

# Quantize 32-bit RGBA PNG to 8-bit (or smaller) RGBA-palette
# pngquant [number of colors] [options] input.png
#   --skip-if-larger  only save converted file if they're smaller than original
#   --strip           remove optional metadata
#   --ext=.png        set output filename to be same as input filename
#   --force           overwrite existing output files
pngquant 256 --skip-if-larger --strip --ext=.png --force example.png

I’ve observed for most screenshots you can comfortably reduce the color palette to as few as 64 colors before it becomes noticeable. If you frequently take screenshots of gradients or more complex images, you may want to stick with 256 colors to avoid noticeable artifacts.

Lossless Compression: DEFLATE

The PNG file format uses DEFLATE compression internally for an added layer of lossless compression, but most PNG libraries do not implement aggressive lossless compression. This provides another opportunity to reduce the file size further.

In 2013, Google released zopfli, which claimed to improve compression by 3-8% compared to zlib. The trade off for this improvement: waiting an extra 1-2 seconds. (There is no decompression penalty when viewing the compressed image).

# Install zopfli using Homebrew, which includes zopflipng
brew install zopfli

# Optimize PNG compression
# zopflipng [options] input.png output.png
#   -y  do not ask about overwriting files
zopflipng -y example.png example.png

Relative to the massive savings from color quantization, improving lossless compression provides a much smaller reduction, but in the context of pages with many images these marginal gains do add up to worthwhile savings.

Automation

The trick is to make this happen automatically every time I capture a screenshot using Automator. This allows you to run commands based on file events, like every time a new screenshot is added to a directory.

I create a dedicated Screenshots directory, so that they don’t clutter your desktop.

# Create a Screenshots directory in the current users Home directory
mkdir -p "$HOME/Screenshots"

# Configure macOS to capture screenshots to this location
# If you want to revert this change, and save screenshots to your desktop,
# instead use "$HOME/Desktop"
defaults write com.apple.screencapture location "$HOME/Screenshots"

Using Automator, create a new Folder Action that receives notifications from the Screenshots folder. Add a Run Shell Script block, and make sure to Pass input as arguments. Combining the commands above, and this time using $@ syntax to handle multiple arguments, and absolute paths for pngquant and zopflipng, the final script is:

for f in "$@"
do
  # Quantize to PNG-8 with 64 colors (lossy, fast)
  # Majority of file size savings come from this
  /usr/local/bin/pngquant 64 --skip-if-larger --strip --ext=.png --force "$f"

  # Rename to communicate the majority of file size savings complete
  # Allows me to grab the file faster
  new_f="$(dirname "$f")/$(date -r "$f" +"%Y-%m-%dT%H.%M.%S").png"
  mv -n "$f" "$new_f"

  # Improve deflate compression for an extra 10%
    /usr/local/bin/zopflipng -y "$new_f" "$new_f"

  # Copy to clipboard (optional)
  # osascript -e "set the clipboard to (read (POSIX file \"$(perl -e "print glob('$new_f')")\") as {«class PNGf»})"
done

Finally, I add the Screenshots folder to my Dock for easy access. This is achieved by dragging the Screenshots folder from Finder to your Dock.

Fresh flowers in preparation for the week ahead.

Working from home certainly makes baking bread convenient. I’m currently using bakers yeast and a poolish preferment in my country loaf for simplicity.

Yesterday I setup Minio on my Synology DS216play using MinioSPK for use with Arq. So far, faster and more reliable than using Autofs.

Seeing the sights before Git Merge.

Pleased to have added an experimental Micropub endpoint so I can post here more easily. This was posted directly to my blog from the Micro.blog app. 🎉

XOXO 2019:

We strongly encourage attendees use any of the options above instead of Uber or Lyft. Both companies are actively undermining public transportation and worker protections, refusing to pay their drivers a living wage or benefits, while artificially and unsustainably lowering the cost of rides.

I prefer to walk or cycle. Failing that, I use public transportation for longer distances. Too many cities make this difficult, or dangerous, if not impossible.

Four RCCNYC members, along with members from Washington DC, Boulder and Portland, headed north to Saint-Urbain-de-Charlevoix in search of gravel at the first edition of Grinduro in Canada. Part race, part party, Grinduro delivered on all fronts.

The 100km race treated us to perfect weather, scenic views, steep gravel climbs and painful miles of sand. All RCCNYC members survived the jumps, tree roots and berms of the timed single track descent on race day (even if not as lucky the day prior when previewing the section). The final major climb – a steep gravel logging road, appearing as if freshly hewn by the bulldozers that littered the roadside – proved challenging for tired legs, but the sight and smell of decks of logs welcomed us to the final summit before the descending to the finish, beer and live music.

The recovery ride was a spirited farewell to the roads of Charlevoix and a welcome opportunity to stretch our legs before driving through the night back to NYC. I’m still not sure if cycling or driving was more exhausting!

A deep-dive into how GitLab reduced median latency by 80% for FindCommit and TreeEntry RPCs by sharing a single Git cat-file process per Rails session.