DEVR           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 504807.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID504807.0@justin-prod-sched01.dune.hep.ac.uk
Workflow TestingYes
Workflow ID1
Stage ID1
User nameamcnab@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1073741824 (1024 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2026-04-11 06:49:51
SiteES_CIEMAT
EntryDUNE_T2_ES_CIEMAT_condorce1
Last heartbeat2026-04-11 06:57:30
From worker nodeHostnamegaew0206.ciemat.es
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1073741824 (1024 MiB)
Wall seconds limit232200 (64 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-04-11 06:51:23
Input files
JobscriptExit code0
Real time5m (352s)
CPU time0m (49s = 13%)
Max RSS bytes66662400 (63 MiB)
Outputting started2026-04-11 06:57:15
Output files
Finished2026-04-11 06:57:30
Saved logsjustin-logs:504807.0-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']"
DEBUG:dogpile.lock:Calling creation function for not-yet-present value
DEBUG:dogpile.cache.region:Cache value generated in 0.000 seconds for key(s): "host_to_choose_choice['https://dune-rucio.fnal.gov']"
DEBUG:dogpile.lock:Released creation lock
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/?expression=T3_US_NERSC HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC HTTP/1.1" 200 1240
DEBUG:root:Input validation done.
INFO:root:Preparing upload for file awt-1775890295-gOy80A6LKv
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC/attr/ HTTP/1.1" 200 139
DEBUG:root:wan domain is used for the upload
DEBUG:root:Registering file
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /accounts/dunepro/scopes/ HTTP/1.1" 200 870
DEBUG:root:Trying to create dataset: testpro:awt-uploads-202614
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202614 HTTP/1.1" 409 104
INFO:root:Dataset testpro:awt-uploads-202614 already exists - no rule will be created
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /dids/testpro/awt-1775890295-gOy80A6LKv/meta?plugin=DID_COLUMN HTTP/1.1" 404 129
DEBUG:root:File DID does not exist
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /replicas HTTP/1.1" 201 7
INFO:root:Successfully added replica in Rucio catalogue at T3_US_NERSC
DEBUG:root:gfal.NoRename: connecting to storage
DEBUG:root:Checking if davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/2b/cc/awt-1775890295-gOy80A6LKv exists
DEBUG:root:gfal.NoRename: checking if file exists davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/2b/cc/awt-1775890295-gOy80A6LKv
DEBUG:root:gfal.NoRename: closing protocol connection
DEBUG:root:[{'hostname': 'dtn14.nersc.gov', 'scheme': 'root', 'port': 1094, 'prefix': '//global/cfs/cdirs/m3249/dune/RSE', 'impl': 'rucio.rse.protocols.gfal.NoRename', 'domains': {'lan': {'read': 10, 'write': 10, 'delete': 10}, 'wan': {'read': 10, 'write': 10, 'delete': 10, 'third_party_copy_read': 0, 'third_party_copy_write': 0}}, 'extended_attributes': None}, {'hostname': 'dtn14.nersc.gov', 'scheme': 'davs', 'port': 1094, 'prefix': '/global/cfs/cdirs/m3249/dune/RSE', 'impl': 'rucio.rse.protocols.gfal.NoRename', 'domains': {'lan': {'read': 1, 'write': 1, 'delete': 1}, 'wan': {'read': 1, 'write': 1, 'delete': 1, 'third_party_copy_read': 1, 'third_party_copy_write': 1}}, 'extended_attributes': None}]
INFO:root:Trying upload with davs to T3_US_NERSC
DEBUG:root:Processing upload with the domain: wan
DEBUG:root:gfal.NoRename: connecting to storage
DEBUG:root:The PFN created from the LFN: davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/2b/cc/awt-1775890295-gOy80A6LKv
DEBUG:root:gfal.NoRename: checking if file exists davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/2b/cc/awt-1775890295-gOy80A6LKv
DEBUG:root:gfal.NoRename: checking if file exists davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/2b/cc/awt-1775890295-gOy80A6LKv
DEBUG:root:put: Attempt 1
DEBUG:root:gfal.NoRename: uploading file from awt-1775890295-gOy80A6LKv to davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/2b/cc/awt-1775890295-gOy80A6LKv
INFO:root:Successful upload of temporary file. davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/2b/cc/awt-1775890295-gOy80A6LKv
DEBUG:root:skip_upload_stat=False
DEBUG:root:stat: pfn=davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/2b/cc/awt-1775890295-gOy80A6LKv
DEBUG:root:gfal.NoRename: getting stats of file davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/2b/cc/awt-1775890295-gOy80A6LKv
DEBUG:root:Filesize: Expected=26 Found=26
DEBUG:root:Checksum: Expected=5f7306e1 Found=5f7306e1
DEBUG:root:gfal.NoRename: closing protocol connection
DEBUG:root:Upload done.
INFO:root:Successfully uploaded file awt-1775890295-gOy80A6LKv
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /traces/ HTTP/1.1" 404 207
DEBUG:urllib3.connectionpool:Resetting dropped connection: dune-rucio.fnal.gov
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "PUT /replicas HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202614/dids HTTP/1.1" 201 7
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /replicas/list HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /dids/testpro/awt-1775890295-gOy80A6LKv/parents HTTP/1.1" 200 None
--- Upload try 1/1
--- Rucio upload 1/1 returns 0
--- Replica check try 1/1
--- Dataset awt-uploads-202614 check try 1/1
--- Upload, replicas, and datasets checks passed
'justin-rucio-upload --rse T3_US_NERSC --protocol davs --scope testpro --dataset awt-uploads-202614 awt-1775890295-gOy80A6LKv --timeout 1200' returns 0


subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=268591181/CN=177589028342
issuer    : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=268591181
identity  : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=268591181
type      : RFC compliant proxy
strength  : 2048 bits
path      : /home/awt-proxy.pem
timeleft  : 167:54:08
key usage : Digital Signature, Key Encipherment, Key Agreement
=== VO dune extension information ===
VO        : dune
subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk
issuer    : /DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Forward Discovery Group, LLC/CN=voms1.fnal.gov
attribute : /dune/Role=Production/Capability=NULL
attribute : /dune/Role=NULL/Capability=NULL
timeleft  : 164:37:47
uri       : voms1.fnal.gov:15042

===== Results =====

Download/upload commands:
xrdcp --force --nopbar --verbose $read_pfn downloaded.txt
echo '{"namespace":"testpro","name":"FILENAME","size":0}' >tmp.json
metacat file declare --json -f tmp.json "dune:all"
justin-rucio-upload --rse $rse_name --protocol $write_protocol --scope testpro --dataset awt-uploads-202614 --timeout 1200 FILENAME
Use the wrapper job link on the page for the job on the justIN Dashboard to find the full log file, with errors from these commands

Each line: $JUSTIN_SITE_NAME $rse_name $download_retval $upload_retval $read_pfn $write_protocol
==awt== ES_CIEMAT DUNE_CA_SFU 0 0 root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT DUNE_CERN_EOS 0 0 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT DUNE_ES_PIC 0 0 root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT DUNE_FR_CCIN2P3_DISK 0 0 root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT DUNE_IT_INFN_CNAF 0 0 root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT DUNE_UK_GLASGOW 0 0 root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT DUNE_UK_LANCASTER_CEPH 0 0 root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT DUNE_UK_MANCHESTER_CEPH 0 0 root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT DUNE_US_BNL_SDCC 0 0 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT DUNE_US_FNAL_DISK_STAGE 0 0 root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT FNAL_DCACHE 0 99 root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/tape_backed/dunepro//other/awt-staging/awt-download-2023-03-07-01.txt_1749841165 davs
==awt== ES_CIEMAT NIKHEF 0 0 root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT PRAGUE 0 0 root://se1.farm.particle.cz:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT QMUL 0 0 root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT RAL-PP 0 0 root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT RAL_ECHO 0 0 root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT SURFSARA 0 0 root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== ES_CIEMAT T3_US_NERSC 0 0 root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
justIN time: 2026-04-17 15:26:04 UTC       justIN version: 01.07.rc0