ecbuild_get_test_data

Download a test data set at build time.

ecbuild_get_test_data( NAME <name>
                       [ TARGET <target> ]
                       [ DIRNAME <dir> ]
                       [ DIRLOCAL <dir> ]
                       [ MD5 <hash> ]
                       [ EXTRACT ]
                       [ NOCHECK ]
                       [ INSECURE ])

curl or wget is required (curl is preferred if available).

Options

NAMErequired

name of the test data file

TARGEToptional, defaults to test_data_<name>

CMake target name

DIRNAMEoptional

use when there is a directory structure on the server that hosts test files

DIRLOCAL : optional, defaults to “.”, local directory in which the test data is copied

MD5optional, ignored if NOCHECK is given

md5 checksum of the data set to verify. If not given and NOCHECK is not set, download the md5 checksum and verify

EXTRACToptional

extract the downloaded file (supported archives: tar, zip, tar.gz, tar.bz2)

NOCHECKoptional

do not verify the md5 checksum of the data file

INSECUREoptional

explicitly allows curl and wget to perform “insecure” SSL connections

Usage

Download test data from <ECBUILD_DOWNLOAD_BASE_URL>/<DIRNAME>/<NAME>

If the ECBUILD_DOWNLOAD_BASE_URL variable is not set, the default URL https://sites.ecmwf.int/repository is used.

If the DIRNAME argument is not given, test data will be downloaded from <ECBUILD_DOWNLOAD_BASE_URL>/<project>/test-data/<relative path to current dir>/<NAME> where <project> is the name of the project with all ‘_’ replaced by ‘-‘.

By default, the downloaded file is verified against an md5 checksum, either given as the MD5 argument or downloaded from the server otherwise. Use the argument NOCHECK to disable this check.

The default timeout is 30 seconds, which can be overridden with ECBUILD_DOWNLOAD_TIMEOUT. Downloads are by default only tried once, use ECBUILD_DOWNLOAD_RETRIES to set the number of retries.

Further download behaviour, all curl-only:

ECBUILD_DOWNLOAD_STALL_TIMEOUT

seconds a transfer may stall below 1 kB/s before it is abandoned and retried (default 60). ECBUILD_DOWNLOAD_TIMEOUT bounds only the connect.

ECBUILD_DOWNLOAD_HTTP_VERSION

1.1, 2, or empty to let curl negotiate. Defaults to empty, except on a curl too old for --retry-all-errors, where it defaults to 1.1: such a curl cannot retry an HTTP/2 framing error, so it is not asked to speak HTTP/2.

ECBUILD_DOWNLOAD_EXTRA_FLAGS

further flags appended to the curl command line

Examples

Do not verify the checksum:

ecbuild_get_test_data( NAME msl.grib NOCHECK )

Checksum agains remote md5 file:

ecbuild_get_test_data( NAME msl.grib )

Checksum agains local md5:

ecbuild_get_test_data( NAME msl.grib MD5 f69ca0929d1122c7878d19f32401abe9 )

ecbuild_get_test_multidata

Download multiple test data sets at build time.

ecbuild_get_test_multidata( NAMES <name1> [ <name2> ... ]
                            TARGET <target>
                            [ DIRNAME <dir> ]
                            [ DIRLOCAL <dir> ]
                            [ LABELS <label1> [<label2> ...] ]
                            [ EXTRACT ]
                            [ NOCHECK ]
                            [ INSECURE ] )

curl or wget is required (curl is preferred if available).

Options

NAMESrequired

list of names of the test data files. Each name may contain a relative path and may be followed by an md5 checksum separated with a :.

TARGETrequired

name of the download test (and prefix for its internal CMake targets)

DIRNAMEoptional

use when there is a directory structure on the server that hosts test files

DIRLOCAL : optional, defaults to “.”, local directory in which the test data is copied

LABELSoptional

list of labels to assign to the test

Lower case project name and download_data are always added as labels.

This allows selecting tests to run via ctest -L <regex> or tests to exclude via ctest -LE <regex>.

EXTRACToptional

extract downloaded files (supported archives: tar, zip, tar.gz, tar.bz2)

NOCHECKoptional

do not verify the md5 checksum of the data file

INSECUREoptional

explicitly allows curl and wget to perform “insecure” SSL connections

Usage

Download test data from <ECBUILD_DOWNLOAD_BASE_URL>/<DIRNAME> for each name given in the list of NAMES. Each name may contain a relative path, which is appended to DIRNAME and may be followed by an md5 checksum, separated with a : (the name must not contain spaces). Characters that are not valid in CMake target names, including = and , in filenames, are supported.

Each file is assigned an internal target name containing readable directory and filename components followed by a hash of the complete relative path. The hash keeps the targets distinct when different paths have the same readable C identifier. These internal target names are implementation details and should not be invoked directly; use the target passed with TARGET.

Files in different relative directories must have different basenames. The relative directory is used for the remote location and internal target name, but the downloaded file is stored directly in DIRLOCAL using its basename. For example, foo/a.txt and bar/a.txt would both produce <DIRLOCAL>/a.txt and are therefore not supported in the same invocation.

If the ECBUILD_DOWNLOAD_BASE_URL variable is not set, the default URL https://get.ecmwf.int/repository/test-data is used.

If the DIRNAME argument is not given, test data will be downloaded from <ECBUILD_DOWNLOAD_BASE_URL>/<project>/<relative path to current dir>/<NAME>

By default, each downloaded file is verified against an md5 checksum, either given as part of the name as described above or a remote checksum downloaded from the server. Use the argument NOCHECK to disable this check.

Examples

Do not verify checksums:

ecbuild_get_test_multidata( TARGET get_grib_data NAMES foo.grib bar.grib
                            DIRNAME test/data/dir NOCHECK )

Checksums agains remote md5 file:

ecbuild_get_test_multidata( TARGET get_grib_data NAMES foo.grib bar.grib
                            DIRNAME test/data/dir )

Checksum agains local md5:

ecbuild_get_test_multidata( TARGET get_grib_data DIRNAME test/data/dir
                            NAMES msl.grib:f69ca0929d1122c7878d19f32401abe9 )

Download filenames containing punctuation:

ecbuild_get_test_multidata(
  TARGET get_gridspec_data
  DIRNAME eccodes/test-data/data/gridspec
  NAMES gridType=regular_ll.grib
        gridType=regular_ll,scanningMode=96.grib
  NOCHECK )