Mirroring at Savannah

NOTE: This is an updated version of the previous Mirmon page.

Savannah offers two public-facing mirroring services:

  1. ftpmirror.gnu.org - redirects to world-wide mirrors of ftp.gnu.org. These mirrors carry official releases of GNU pacakges. This page refers to it as gnu mirror.

  2. download.savannah.{nongnu,gnu}.org - redirects to world-wide mirrors of Savannah download area. Typically used for non-GNU software, but GNU packages can store files there as well. This page refers to it as nongnu mirror.

Adding mirrors

Occasionally, a new mirror is submitted. Usually the webmasters inform us, but people may come to us directly.

When we are told of a new mirror, we verify it, then

  1. On mgt1, add contact information to /root/administration/mirrors-contacts.txt;
  2. On download0.savannah.gnu.org, add the mirror URLs to /srv/download/00_MIRRORS.html (alphabetical, check list online) and /srv/download/00_MIRRORS.txt (no order). Supported protocols are FTP, HTTP, HTTPS, RSYNC.

Client-side overview

A user requesting a download from ftpmirror.gnu.org will receive an HTTP/302 response redirecting to a mirror closest to the user's originating IP:

$ curl -o /dev/null -D /dev/stdout --silent http://ftpmirror.gnu.org/sed/sed-4.4.tar.xz
HTTP/1.1 302 Found
Location: http://gnu.mirrors.hoobly.com/gnu/sed/sed-4.4.tar.xz
[...]

A user requested a download from download.savannah.gnu.org will be redirected to an intermediate redirection URL, and then to the mirror:

$ curl -o /dev/null -D /dev/stdout --silent http://download.savannah.gnu.org/releases/test-project/README
HTTP/1.1 302 Found
Location: http://download.savannah.gnu.org/releases-redirect/test-project/README
[...]

$ curl -o /dev/null -D /dev/stdout --silent http://download.savannah.gnu.org/releases-redirect/test-project/README
HTTP/1.1 302 Found
Location: http://nongnu.askapache.com//test-project/README
[...]

The technical reason for this extra redirection is that download.savannah.gnu.org serves directory listing directly, and only redirects actual files. See nginx details below.

Server-side redirection overview

Redirection is implemented using a Perl CGI script called from nginx running on download0.sv.gnu.org.

Both gnu and nongnu redirections are served from download0.savannah.gnu.org VM. ftpmirror.gnu.org is CNAME'd to ftpmirror.sv.gnu.org, which points to the same IP as download0 (remember Savannah admins only control *.sv.gnu.org DNS entries).

The nginx configuration files are: download0:/etc/nginx/sites-available/ftpmirror, download0:/etc/nginx/sites-available/ftpmirror-common.inc, download0:/etc/nginx/sites-available/ftpmirror-common-https.inc, download0:/etc/nginx/sites-available/download, download0:/etc/nginx/sites-available/download-common.inc, download0:/etc/nginx/sites-available/download-common-https.inc. The *-common*.inc files contain the CGI configurations. The other files include them twice (once for HTTP, once for HTTPS).

The CGI configuration for gnu mirror is:

location / {
    gzip off;

    # In this case, the ENTIRE path is passed on the to cgi script, e.g.
    # given http://ftpmirror.sv.gnu.org/datamash/datamash-1.1.0.tar.gz we pass
    # '/datamash/datamash-1.1.0.tar.gz' to the CGI script.
    # So there is no need to use 'fastcgi_split_path_info' command.
    fastcgi_param PATH_INFO            $uri;

    fastcgi_param LOG_DIR         /opt/savannah/stat/ftp/log;
    fastcgi_param MIRROR_FILE     $mirror_file;
    fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect;
    fastcgi_param GEOIP_DB        /usr/share/GeoIP/GeoIP.dat;
    fastcgi_param GEOIP6_DB       /usr/share/GeoIP/GeoIPv6.dat;
    include fastcgi_params;
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
}

The CGI configuration for nongnu mirror is:

# Redirect to the CGI if it's a file URL.
rewrite ^/releases/(.*[^/])$ /releases-redirect/$1 last;

# Directory listing if it's a directory.
location /releases {
    alias /srv/download;
    fancyindex on;              # Enable fancy indexes.
}

# Actual redirection mechanism.
location /releases-redirect {
    gzip off;

    # This command tells NGINX how to extract the path info part
    # from the requested URL: it trims the 'releases-redirect' part,
    # leaving only what follows it.
    fastcgi_split_path_info            ^(/releases-redirect)(/?.+)$;

    # This command takes NGINX's path_info variable (extracted above),
    # and sends it to the FastCGI daemon as the PATH_INFO environment variable.
    fastcgi_param PATH_INFO            $fastcgi_path_info;

    fastcgi_param LOG_DIR         /opt/savannah/stat/dl/log;
    fastcgi_param MIRROR_FILE     $mirror_file;
    fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect;
    fastcgi_param GEOIP_DB        /usr/share/GeoIP/GeoIP.dat;
    fastcgi_param GEOIP6_DB       /usr/share/GeoIP/GeoIPv6.dat;

    include fastcgi_params;
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
}

The $mirror_file variable is set to one of /opt/savannah/mirrors/active-mirror-lists/{gnu,download}{,-https}.txt depending on whether it's served via HTTP or HTTPS and whether it's used for gnu mirror or nongnu mirror.

The mirror list text files (gnu.txt and download.txt) contain an auto-generated list of active mirrors and region redirections:

# head -n5 /opt/savannah/mirrors/active-mirror-lists/gnu.txt
http://us.mirrors.cicku.me/gnu/ us
http://mirror.freedif.org/GNU/ sg
http://fosszone.csd.auth.gr/gnu/ gr
http://mirror.marwan.ma/gnu/ ma
http://ftp.task.gda.pl/pub/gnu/ pl
# tail -n5 /opt/savannah/mirrors/active-mirror-lists/gnu.txt
ye      az
yt      za
za      za
zm      za
zw      za

(Yemen [YE] uses the mirrors located in Azerbaijan [AZ], Zimbabwe [ZW] uses the mirrors located in South Africa [ZA].)

See 'mirmon' section below to learn how these files are generated.

The Perl script mirror-redirect uses Geo::IP module to detect users' regions based on their IP.

Server-side Mirmon overview

  1. The authoritative lists of mirrors are managed manually. For gnu mirror, GNU webmasters update the lists (http://www.gnu.org/prep/ftp.html, generated from http://www.gnu.org/prep/FTP). For nongnu mirror, Savannah admins maintain the lists as explained above.

  2. mirmon is used to check each mirror and determine how up-to-date it is.

    We maintain 3 separate mirmon configurations:

    • gnu - list of HTTP(S)-only mirrors for ftpmirror.gnu.org. The gnu list is used later to compile the list of active mirrors for the redirection CGI script.
    • allgnu - list of HTTP(S), FTP, RSYNC mirrors for ftpmirror.gnu.org. The allgnu list is used only by human admins to check which mirrors are up-to-date.
    • nongnu - list of mirrors for download.savannah.gnu.org. The nongnu list is used later to compile the list of active mirrors for the redirection CGI script.
  3. The mirmon input lists (of mirrors to check) are here:

    The lists are stored here:

    $ ls -lhog /opt/savannah/mirrors/mirmon-lists/
    -rw-r--r-- 1 7.7K Feb 20 23:30 allgnu-mirror-list.txt
    -rw-r--r-- 1 4.0K Feb 20 23:30 gnu-mirror-list.txt
    lrwxrwxrwx 1   28 Feb 20 23:28 nongnu-mirror-list.txt -> /srv/download/00_MIRRORS.txt
    

    The script sv_gnu-mirmon fetches the gnu list and saves it locally in the two versions (gnu and allgnu). The nongnu list is symlinked to the manually-managed file.

    These files are the input to mirmon.

  4. The three mirmon configuration files are:

    $ ls -1 /etc/mirmon/
    mirmon-allgnu.conf
    mirmon-gnu.conf
    mirmon-nongnu.conf
    

    When mirmon is executed (once for each configuration), it reads the corresponding list file, and generates an HTML report page and a 'state' textual file:

    /opt/savannah/www/mirmon/gnu/index.html
    /opt/savannah/www/mirmon/nongnu/index.html
    /opt/savannah/www/mirmon/allgnu/index.html
    
    # ls -lhog /var/lib/mirmon/state-*
    -rw-r--r-- 1  16K Feb 21 19:42 /var/lib/mirmon/state-allgnu
    -rw-r--r-- 1 7.3K Feb 21 20:22 /var/lib/mirmon/state-gnu
    -rw-r--r-- 1 3.5K Feb 21 19:32 /var/lib/mirmon/state-nongnu
    

    A 'state' file contains the status for each mirror:

    $ head -n5 /var/lib/mirmon/state-gnu
    http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.gnu.org/ 1707624961 ok 1707657721 ssssssssssssss 1707632521-ssssssssssssss 1707657721
    http://ftp.acc.umu.se/mirror/gnu.org/gnu/ 1707617761 ok 1707654121 ssssssssssssss 1707626399-ssssssssssssss 1707654121
    http://ftp.cc.uoc.gr/mirrors/gnu/ 1707581761 ok 1707654121 ssssssssssssss 1707626399-ssssssssssssss 1707654121
    http://ftp.eq.uc.pt/software/unix/gnu/ 1707621361 ok 1707657721 ssssssssssssfs 1707628921-ssssssssssssss 1707657721
    http://ftp.fau.de/gnu/ 1707646561 ok 1707657721 ssssssssssssss 1707632521-ssssssssssssss 1707657721
    
  5. Two auxiliary scripts /opt/savannah/mirrors/scripts/update-active-mirrors-{gnu,nongnu}.sh trivially invoke sv_mirmon-to-geoip; that script reads the state file and the mirror list and generates simple text files containing the active and up-to-date mirrors and region redirections:

    # ls /opt/savannah/mirrors/active-mirror-lists/
    download-https.txt  download.txt  gnu-https.txt  gnu.txt
    

    These are the four files used in the CGI scripts (see above section).

  6. The 3 mirmon invocations are executed as cronjobs in /etc/cron.d/sv-mirmon:

    # Update the authoritative list of GNU ftp mirrors (mirroring ftp.gnu.org).
    3 1 * * * root sv_gnu-mirmon --http-only /opt/savannah/mirrors/mirmon-lists/gnu-mirror-list.txt
    6 1 * * * root sv_gnu-mirmon --all       /opt/savannah/mirrors/mirmon-lists/allgnu-mirror-list.txt
    
    # Check the status of each mirror (and regenarate the HTML report page).
    # For gnu/nongnu - also update the list of active mirrors (for the mirror CGIs).
    22 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-gnu.conf -q -get update >/var/log/mirmon/gnu 2>&1 && /opt/savannah/mirrors/scripts/update-active-mirrors-gnu.sh
    
    32 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-nongnu.conf -q -get update >/var/log/mirmon/nongnu 2>&1 && /opt/savannah/mirrors/scripts/update-active-mirrors-nongnu.sh
    
    42 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-allgnu.conf -q -get update >/var/log/mirmon/allgnu 2>&1
    

Diagnostics and Troubleshooting

  1. The three mirmon HTML reports are viewable here: http://download.savannah.gnu.org/mirmon/.

  2. An additional page is used for testing: http://download.savannah.gnu.org/mirror-check/, http://ftpmirror.savannah.gnu.org/mirror-check/, https://download.savannah.gnu.org/mirror-check/, https://ftpmirror.savannah.gnu.org/mirror-check/.

    The NGINX configurations are *-common.inc:

    location /mirror-check/ {
        gzip off;
        fastcgi_split_path_info       ^(/mirror-check)(/?.+)$;
        fastcgi_param PATH_INFO       $fastcgi_path_info;
        fastcgi_param MIRROR_FILE     $mirror_file;
        fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect;
        fastcgi_param GEOIP_DB        /usr/share/GeoIP/GeoIP.dat;
        fastcgi_param GEOIP6_DB       /usr/share/GeoIP/GeoIPv6.dat;
        fastcgi_param TEST            1;
        include fastcgi_params;
        fastcgi_pass unix:/var/run/fcgiwrap.socket;
    }
    

    The web page accepts two optional parameters,

    • addr overrides the detected IP address
    • cc overrides the detected country code

    For example,

    $ wget -o /dev/null -O - \
      'https://ftpmirror.gnu.org/mirror-check/?addr=209.51.188.113&cc=lt'
    GNU Savannah - mirror check
    
    Requested path: /
    Query string:   addr=209.51.188.113&cc=lt
    
    Detected remote address: 209.51.188.113 (overridden in query string)
    Detected country: United States
    Country code: lt (overridden in query string)
    Suggested mirror: https://sunsite.icm.edu.pl/pub/gnu
    
    All mirrors from pl:
    
      https://ftp.task.gda.pl/pub/gnu/
      https://ftp.man.poznan.pl/gnu/
      https://sunsite.icm.edu.pl/pub/gnu/
    
  3. Mirmon cronjob logs are saved in /var/log/mirmon. Current implementation saves only the most recent run.

  4. Mirmon state files should be generated at 22/32/42 minutes of each hour (defined in the cron file):

    # ls -lhog /var/lib/mirmon/
    total 84K
    -rw-r--r-- 1  27K Feb 11 09:17 debian-mirror-check.html
    -rw-r--r-- 1 3.9K Feb 11 09:17 state
    -rw-r--r-- 1  24K Feb 11 08:42 state-allgnu
    -rw-r--r-- 1  18K Feb 11 08:22 state-gnu
    -rw-r--r-- 1 4.2K Feb 11 08:32 state-nongnu
    

    The active mirror list files should have similar timestamp:

    # ls -lhog /opt/savannah/mirrors/active-mirror-lists/
    total 24K
    -rw-r--r-- 1 1.7K Feb 11 08:32 download-https.txt
    -rw-r--r-- 1 2.1K Feb 11 08:32 download.txt
    -rw-r--r-- 1 4.4K Feb 11 08:22 gnu-https.txt
    -rw-r--r-- 1 4.2K Feb 11 08:22 gnu.txt
    
  5. Redirection script statistics are available at https://download.savannah.gnu.org/dl-stats/ (for download) and https://download.savannah.gnu.org/ftp-stats/ (for ftpmirror).

  6. Other summaries based on latest nginx logs are available (for ftpmirror only) at https://download.savannah.gnu.org/ftpmirror-stats/.