NCAR CESM Output

CESM Output Overview

Overview

Teaching: 0 min
Exercises: 0 min
Questions

  • What kinds of output do I get after the model runs?

Objectives

Last week we setup and ran our first CESM experiment. What did it do? How do I find and interpret the output from it? There are several types of output form our model simulation:

  1. Information in the CaseStatus file
  2. Standard error and standard out
  3. Log files
  4. Model output data

In this episode, we will take a look at each of these and understand what they tell us.

Key Points

CaseStatus

Overview

Teaching: 0 min
Exercises: 0 min
Questions

  • What does the CaseStatus file tell me about my model experiment?

Objectives

The CaseStatus file

Let’s start by going to the CASEROOT directory for your first experiment and looking at the CaseStatus file:

$ cd ~/cases/b.day1.0
$ cat CaseStatus

 ---------------------------------------------------
2020-09-15 11:00:30: case.submit starting
 ---------------------------------------------------
2020-09-15 11:00:38: case.submit success case.run:4141153.chadmin1.ib0.cheyenne.ucar.edu, case.st_archive:4141154.chadmin1.ib0.cheyenne.ucar.edu
 ---------------------------------------------------
2020-09-15 11:21:16: case.run starting
 ---------------------------------------------------
2020-09-15 11:21:22: model execution starting
 ---------------------------------------------------
2020-09-15 11:23:58: model execution success
 ---------------------------------------------------
2020-09-15 11:23:58: case.run success
 ---------------------------------------------------
2020-09-15 11:24:10: st_archive starting
 ---------------------------------------------------
2020-09-15 11:24:18: st_archive success
 ---------------------------------------------------

From this we can see model execution success indicates the model ran successfully and st_archive_success indicates that the archiving job ran successfully. Remember that when we submitted the model, two jobs were sent to the queue, the model run and the archive job. Both appear to have run successfully.

Troubleshooting

Does anyone have a CaseStatus file that indicates the model did not run successfully?

Key Points

Standard error and standard out

Overview

Teaching: 0 min
Exercises: 0 min
Questions

  • What do the stderr and stdout files tell me?

Objectives

Standard error and standard out files

In your CASEROOT directory, your model produced files like b.day1.0.run.o* and b.day1.0.st_archive.o*. These files produce output related to our submit script. If the model and/or archiving did not run to completion, you will also have a b.day1.0.run.e* and/or b.day1.0.st_archive.e* file. The ones with *.o*' are called standard out (stdout) and the ones with .e` are called standard error (stderr).

The end of the file usually contains the most useful information, so you can use the Unix tail command to look at the end of the file:

$ tail b.day1.0.run.o4141153

 - Case input data directory (DIN_LOC_ROOT) is /glade/p/cesmdata/cseg/inputdata
 - Checking for required input datasets in DIN_LOC_ROOT
-------------------------------------------------------------------------
2020-09-15 11:21:22 MODEL EXECUTION BEGINS HERE
run command is mpiexec_mpt -p "%g:"  -np 576  omplace -tm open64  /glade/scratch/kpegion/b.day1.0/bld/cesm.exe  >> cesm.log.$LID 2>&1
2020-09-15 11:23:58 MODEL EXECUTION HAS FINISHED
check for resubmit
dout_s True
mach cheyenne
resubmit_num 0

Troubleshooting

Does anyone have a stdout file that does not indicate MODEL EXECUTION HAS FINISHED

Key Points

Log files

Overview

Teaching: 0 min
Exercises: 0 min
Questions

  • What do log files tell me?

Objectives

Log files

Each component of CESM produces a log file that contains output and errors related to the running of that component. If your model experiment ran successfully, then the log files are zipped and stored with the short-term archiving in DOUT_S_ROOT/logs. If you have an error, then the logfiles will be located in RUNDIR/*

Let’s take a look at a log file:

$ ./xmlquery DOUT_S_ROOT
$ cd /glade/scratch/kpegion/archive/b.day1.0
$ ls
$ cd logs
$ gunzip cesm.log.4126230.chadmin1.ib0.cheyenne.ucar.edu.200914-164152.gz
$ more cesm.log.4126230.chadmin1.ib0.cheyenne.ucar.edu.200914-164152
$ tail cesm.log.4126230.chadmin1.ib0.cheyenne.ucar.edu.200914-164152

Log files are very long and contain lots of information which may look like errors, but are usually just diagnostic information unless your run fails and there is a clear error. Useful error information is typically located at the end of the file

$ tail cesm.log.4126230.chadmin1.ib0.cheyenne.ucar.edu.200914-164152

Key Points

History Files

Overview

Teaching: 0 min
Exercises: 0 min
Questions

  • How can I look at my model output?

Objectives

History Files

This is of course why we run the model experiment in the first place. We want to see the model output from our experiment. Where does it go? If the experiment an the archiving ran successfully, model output is located in DOUT_S_ROOT. Where is that?

$ ./xmlquery DOUT_S_ROOT 

	DOUT_S_ROOT: /glade/scratch/kpegion/archive/b.day1.0

Let’s go to this directory and see what is there:

$ cd /glade/scratch/kpegion/archive/b.day1.0
$ ls

atm  cpl  esp  glc  ice  lnd  logs  ocn  rest  rof  wav

There are sub-directories for each of the component models and the output, called history files are located in each component sub-directory.

Let’s go look at output files form the ocn model:

$ cd ocn/hist
$ ls

ocn/hist> ls
b.day1.0.pop.dd.0001-01-01-03600  b.day1.0.pop.dv.0001-01-01-03600           b.day1.0.pop.h.once.nc
b.day1.0.pop.do.0001-01-01-03600  b.day1.0.pop.h.ecosys.nday1.0001-01-01.nc  b.day1.0.pop.hv.nc
b.day1.0.pop.dt.0001-01-01-03600  b.day1.0.pop.h.nday1.0001-01-01.nc

We can do a quick look at the ocean model output file b.day1.0.pop.h.nday1.0001-01-01.nc

$ module load ncview
$ ncview b.day1.0.pop.h.nday1.0001-01-01.nc

It make take a few seconds, but a viewer will appear on your screen. If you select SST, then a small map of SST will appear. You can advance through all 5-times in this file using the arrows. You can advance through all 5-times in this file using the arrows.

Let’s look at the atmosphere history files:

$ cd ../atm
$ ls

There are no history files for the atmosphere. Why? We ran our first experiment for 5-days. The atmosphere is set to only provide monthly mean history files as its default. Since we did not run a full month, there are no atmosphere history files available. That configuration is set in the user_nl_cam file and we will learn how to change it later in class.

Understanding history files

netcdf format

named based on time frequency of data

h0/h contain monthly averaged output

h1,h2 contain other time frequencies (e.g. daily)

contain all variables for a given model component for a given time frequency and produced as the experiment runs

Eventually, we want all timesteps for a specific variable from our model run. These files are called timeseries files and they are produced using a postprocessing step after the model run is completed. We will learn how to create these in a future lesson.

Key Points