NSCL DDAS  1.0
Support for XIA DDAS at the NSCL
 All Classes Namespaces Files Functions Variables Macros Pages
Single-crate System Setup Tutorial

Table of Contents

Author
Ron Fox and Jeromy Tompkins
Date
4/21/2016

Introduction

This document describes the simplest DDAS setup: A single PXI crate with one digitizer. Throughout, there will be suggestions on how to expand this setup so that it has several digitizers.

Other simple setup documents will describe

This material is organized as follows:

Overview of the components of a DDAS data taking system.

DDAS data acquisition and online analysis rely on several hardware and software components.

The figure below shows a simple DDAS hardware configuration:

DDAS_Hardware.jpeg
A DDAS Hardware configuration

The figure below shows a simple DDAS software configuration emphasizing the data flow.

softwarecomponents.jpeg
Software components in a single crate system

A few points are worth noting:

Finally a bit about where the DDAS support software is installed at the NSCL. The NSCL may have several versions of the DDAS software installed concurrently. In general these will be installed in

/usr/opt/ddas/version

Where version above is a software version number of the form major.minor-editlevel major, minor and editlevel are three numbers that are called the major version the minor version and the edit level. Edit level changes reflect defect fixes and very small enhancements. minor version changes reflect enhancements and fixes that are somewhat more involved. Migrating to versions where only the minor version changed will at most require you to recompile without any need to modify your code. Major version changes very significant fixes and enhancements that may require recompilation of any user level code.

In addition to these there is a symbolic link /usr/opt/ddas/current that points to the version of the DDAS software that is recommended for use. This link will only change during significant accelerator shutdowns.

Each version's top-level directory includes a script ddassetup.bash

Source that script into your interpreter to setup several environment variables:

The remainder of this document will assume that you have selected a DDAS version and sourced its ddassetup.bash script into your shell.

Loading the Plx 9054 Kernel Module

Communication with the Pixie-16 modules requires communication over the PXI backplane. In any of the DDAS systems at the NSCL, whether using a fiber interface of embedded single board computer (SBC), a PLX 9054 chip is involved in this communication. There is a kernel module that must be loaded into the kernel prior to using this chip. You can load it by :

sudo /usr/opt/plx/Bin/Plx_load 9054

You will be asked to enter you password and then on success you will get the following output written to the terminal

Install: Plx9054
  Load module......... Ok (Plx9054.ko)
  Verify load......... Ok
  Get major number.... Ok (MajorID = 251)
  Create node path.... Ok (/dev/plx)
  Create nodes........ Ok (/dev/plx/Plx9054)

The user must load the PLX kernel module before using DDAS after every time the computer is booted.

Warning
In a multicrate system, the PLX kernel module must be loaded into every computer attached to the PXI backplane (i.e. every spdaq in your system reading out DDAS modules).

Configuration files

Many of the DDAS software components require a set of configurations files to be in the current working directory. These configuration files describe the configuration of modules in a crate and point at a settings file containing the parameters used by the digital pulse processing (DPP) algorithms for each module in use.

These files are described in the documentation of readout . This section will summarize the files needed for nscope. By convention a directory subtree is used to hold the configuration files for each crate. The tree looks like this:

home-directory
     |
     +-- readout
            |
            +--- crate_1  (configuration for crate 1).
            +--- crate_2  (configuration for crate 2 if needed).
            ...
            +--- crate_n  (configuration for last crate, n).

Since we only have a single crate, this should look like:

Home directory
   |
   +---readout
          |
          +--- crate_1
                 +--- cfgPixie16.txt
                 +--- pxisys.ini
                 +--- crate_1.set
                 +--- modevtlen.txt

The directory $DDAS_SHARE/readout/crate_1 has sample configuration files.

Once you've created ~/readout/crate_1 finish making the structure below by

cp $DDAS_SHARE/readout/crate_1/* ~/readout/crate_1

Of these files:

Setting up the hardware

To set up the hardware of a DDAS single crate, single card system you must

Using Nscope to manage the DPP parameters

The nscope program is used to view input waveforms and set digital signal processing parameters. It should be run by

  1. Setting your default directory to a directory that contains the configuration of the crate nscope should operate on.
  2. Invoking the command $DDAS_BIN/nscope

The terminal window in which you run nscope will output a bunch of stuff. The main bits you want to look at are the crate map that is printed out. This will look something like

...
Reading config file... cfgPixie16.txt

7 modules, in slots: 2 3 4 5 6 7 8

current working directory /scratch/fox/daq/DDAS/main/readout/crate_1
...

where the current working directory should confirm that you are in the correct directory for the crate you are working in and the module map should match the crate you are working in as well (in the example shown that you have 7 modules in slots 2-8).

nscope-startup.jpg
nscope start screen

The minimum set of tasks you must perform are to:

Note that the nscope program controls many more parameters of the DPP algorithm. See nscope for more information about what the nscope program can do.

Setting up the pulser signal shape

Set up your pulser to output pulses that rise quickly but have about a 50μsec fall time. The pulse polarity should be – with an amplitude no larger than 1V. For our test we'll use a 0.5V signal with a rise time of – and a fall time of 50μsec.

Setting up the module

When using several DDAS cards it is important to synchronize the digitizer clocks between them. This is done by setting the jumper block JP01 appropriately. The jumper block is just a set of pins located on the board near the backplane connectors. Configure the connections between these pins with jumpers. In case your setup consists of a single module, be sure to configure the jumpers to match the top diagram for PXI clock master.

ss1crate_jumpers.png
The jumper settings for a single crate system.

In this mode, the digitizer drives the clock on the PXI backplane with its own internal clock, and then reads it in from the backplane.

If you have several modules you must:

Setting up the cfgPixie16.txt config file.

Our simple setup uses a single digitizer module in slot 2 of the PXI crate. We are also going to use the DPP settings file crate_1.set This means our cfgPixie16.txt file will look like this:

1     # Crate number 1
1     # One module.
2     # slot we've loaded the module into.
crate_1.set

Let's walk through the minimum steps required in nscope to set up the readout properly.

First click the Boot button at the left of the user interface. This

Here is some sample output from a successful boot of our single module:

Booting all Pixie-16 modules...

Booting Pixie-16 module #0, Rev=12, S/N=175, Bits=12, MSPS=100
ComFPGAConfigFile:  /usr/opt/ddas/firmware/1.0-000/firmware/syspixie16_current_14b100m.bin
SPFPGAConfigFile:   /usr/opt/ddas/firmware/1.0-000/firmware/fippixie16_current_14b100m.bin
DSPCodeFile:        /usr/opt/ddas/firmware/1.0-000/dsp/Pixie16_current_14b100m.ldr
DSPVarFile:         /usr/opt/ddas/firmware/1.0-000/dsp/Pixie16_current_14b100m.var
--------------------------------------------------------

Start to boot Communication FPGA in module 0
Start to boot signal processing FPGA in module 0
Start to boot DSP in module 0
Boot all modules ok
DSPParFile:        ./crate_1.set

If the boot fails with an error, more detailed error messages will be written to Pixie16msg.txt

In order to set the waveform conditioning parameters, you must look at a waveform. Click the Read WF button. If the digitizer does not capture a full waveform, click it again until one is captured.

For the pulser I used, here's a sample picture:

simple_1module_trace.jpeg
Raw pulser trace

There are two things wrong with this signal:

Click the UV_Setup->Analog Signal Conditioning menu entry. This brings up a screen that looks like this:

nscope_analogsetup.jpg
The nscope analog signal setup panel

Our pulser is plugged into channel 0 which has the Sign set to + Using the pull down on that channel to set the sign to - (be sure to click Apply to write the setting into the module) and acquiring another trace gives:

simple_1module_invertedtrace.jpg
Trace after setting sign to -

Our trace is still saturating the ADC. We'll next adjust the trace baseline. Do this by clicking the Adjust Offsets button of the Analog Signal Setup control panel.

Reading another waveform then gives:

simple_1module_blset.jpg
Trace after setting its baseline.

Our next step is setup the constant fraction (CFD). nscope can compute the parameters it thinks are appropriate. Click the Analyze WF button. The terminal window will output something like:

Filter parameters (From Modules):
    ADC sample deltaT (ns): 60
    Fast rise (ns):         500
    Fast flat (ns):         100
    Energy rise (ns):       6000
    Energy flat (ns):       480
    Tau (ns):               40320
    CFD delay:              80
    CFD scale:              0
Trigger Filter Rise Time is NOT an integer number of dT.  Fixing Now.
Trigger Filter Gap Time is NOT an integer numnber of dT. Fixing Now.
Energy Filter Rise Time is NOT an integer number of dT.  Fixing Now.
CFD Delay is NOT an integer numnber of dT. Fixing Now.
Filter parameters used for ''Analyze Waveform''
Note: Parameters have NOT been altered for the aquisition
    Fast rise (ns):         540
    Fast flat (ns):         120
    Energy rise (ns):       6000
    Energy flat (ns):       480
    CFD delay (ns):         120
Baseline: 410

Since we chose a signal with a good rise and fall time initially, we only care about the computed CFD delay. Click the UV_Setup->CFD to bring up the CFD control panel:

nscope_cfdpanel.jpg
nscope's CFD control panel

Apply the settings. Finally enable the CFD by selecting UV_Setup->CSRA and setting the CF bit in all of the channels for which you are going to use the CFD.

Now that the trigger is set up, you can take data and look at the module histograms. Use the menu entry Expert->Start Run to cause the module to start taking triggers and histogram the energies computed from the waveforms. The Read Histo button will read and display the channel energy histogram accumulated by the module. For my pulser I got:

simple_1module_histo.jpg
Histogram of a pulser

Stop data taking by clicking the End Run button. Normally, you need to repeat the process described above for each channel you are using.

Once all channels are set up, save the settings using the UV_Setup->Save2File menu command. In the file selection dialog that pops up, choose the .set file you specified in your cfgPixie16.txt file. This file will be loaded into the modules automatically when they are booted next.

Using DDAS Readout

Once you have a good settings file, you can take data with Readout. In this section we will:

For full documentation of the Readout program, see readout

Running Readout

The DDAS readout program does not need to be modified by you. It uses a set of configuration files that are expected to live in the current working directory when Readout is run:

The directory $DDAS_SHARE/readout/crate_1 has sample configuration files.

The pxisys.ini file normally does not need to be changed. The crate_1.set file was produced by nscope in the previous section. cfgPixie16.txt was edited to be correct in the previous section as well. The file modevtlen.txt must be edited to reflect the size of the event we expect from each module.

We are using only a single module and not taking waveforms from that module. Since the modevtlen.txt file format is simply the size of an event, in 32-bit units, from each digitizer's channels, one digitizer per line, in the same order as the cfgPixie16.txt file, the contents of this file should be:

4

Since we are not taking waveform data, there are only 4 longwords of data per channel.

Now that all of the configuration files are correct, you can run the readout program:

cd ~/readout/crate_1
$DAQBIN/Readout

Starting the NSCLDAQ dumper program.

When you ran the ddassetup.bash script to incorporate the DDAS environment variable in your shell, that script also ran the daqsetup.bash script for the version of NSCLDAQ against which the DDAS software was built.

To run the dumper, therefore, you only need to open a new terminal window and type:

$DAQBIN/dumper  --count=50

The –count=50 option limits the dumper to dump only the first 50 ring items it sees before exiting.

Taking data and looking at the format:

Start taking data in Readout by switching to that window and typing:

begin

The dumper window should output a lot of text and the dumper should exit back to the shell. Once the dumper exits you can stop data taking by typing

end

in the Readout terminal window.

Let's look at a few events in the dumper terminal window. Note that the actual contents of the event we analyze may be different from what you see in your test, we are only interested in explaining the structure of the event at this time.

-----------------------------------------------------------
Event 24 bytes long
Body Header:
Timestamp:    20924510
SourceID:     0
Barrier Type: 0
000c 0000 0064 0c0c 4020 0008 eda3 001f
0000 1b2f 2970 0000
-----------------------------------------------------------
Event 24 bytes long
Body Header:
Timestamp:    21405220
SourceID:     0
Barrier Type: 0
000c 0000 0064 0c0c 4020 0008 a96a 0020
0000 1bff 2965 0000
-----------------------------------------------------------
Event 24 bytes long
Body Header:
Timestamp:    21885900
SourceID:     0
Barrier Type: 0
000c 0000 0064 0c0c 4020 0008 652e 0021
0000 1ba2 2940 0000
-----------------------------------------------------------
Event 24 bytes long
Body Header:
Timestamp:    22366590
SourceID:     0
Barrier Type: 0
000c 0000 0064 0c0c 4020 0008 20f3 0022
0000 1bdc 296f 0000

First note that the timestamp has been set as the timestamp body header for the event, It is monotonically increasing, as it should be. We'll say more about how the timestamp is derived a bit later.

The first two words of the event body 000c 0000 are the event size in little endian format in 16-bit words. This means that the first word is the least significant part while the second word is the most significant part. The size of all events is 12 words long. The event size includes itself, so the body of the event is 10 words long. Note that data from the digitizer comes in 32-bit longwords. Therefore each pair of values is a digitizer longword. Looking at the last event in the listing above, the remaining words are therefore:

0x0c0c0064    - Module type identification
0x00084020    - Event header.
0x002220f3    - low order timestamp.
0x1bdc0000    - High order timestamp and CFD interpolation.
0x0000296f    - Energy and trace length.

The module type identification has the following fields:

   |  bits         |  Meaning                | Value      |
   +---------------+-------------------------+------------+
   | 0-15          | Sampling freq in Mhz.   | 0x64 = 100 |
   | 16-23         | Bit width of digitizer  | 0x0c = 12  |
   | 24-31         | Module revision number  | 0x0c = 12  |

The key values from this word are that the digitizer runs at 100MHz and waveforms are acquired at 12 bits of precision.

The event header has several bit fields. The table below shows the bit fields and the values they have for this event.

    | bits         |  Meaning                |  Value |
    +--------------+-------------------------+--------+
    | 0-3          | Channel number          |  0     |
    | 4-7          | Slot Identifier         |  2     |
    | 8-11         | Crate ID                |  0     |
    | 12-16        | Header length           |  4     |
    | 17-30        | Event length            |  4     |
    | 31           | Finish code             |  0     |

The following longword is the low order 32 bits of the 48-bit event timestamp. Converted to decimal, this becomes 2,236,649. The actual timestamp placed in the event is in 1ns units. Since our digitizer is 100Mhz (10ns/sample), this is correct.

The next word is divided into two 16-bit fields, The low order 16 bits (0) are the high order 16 bits of the event timestamp. The high order 16 bits (0x1bdc = 7132) is the CFD interpolation value. This is the fraction of a sample time multiplied by 65536. Therefore the CFD zero crossing was interpolated at about .11 samples after the trigger time giving the high precision timestamp of 22366491.1 ns.

Note
The interpretation of the CFD bits above pertain specifically to the 100 MSPS Pixie-16 digitizers at the NSCL. The meanings of the bits changes slightly when working with the 250 MSPS and 500 MSPS digitizers.

Running Readout from the ReadoutGUI

To run the readout GUI you must first have set up your account to allow ssh to login via rsa identity keys. If you have not done this yet, see e.g. http://www.linuxproblem.org/art_9.html

cat .ssh/id_rsa.pub | ssh b@B 'cat >> .ssh/authorized_keys'

s simply

cat .ssh/id_rsa.pub >>.ssh/authorized_keys

We're going to set up the ReadoutShell so that it will later run the ReadoutGUI with the NSCL Event builder. While strictly speaking, the event builder is not needed for a single module, normally it is used. This makes the final event structure uniform.

Setting up the data sources

First we'll setup the ddas Readout as a data source. Data sources are programs that are managed by Readout and create data. We're going to use the convention that Readout programs will produce raw data into rings named username_ddas_n where n is the crate number. In our example, the user is 0400x.

Start ReadoutShell:

$DAQBIN/ReadoutShell
  1. Use the Data Source->Add... menu entry to add a new data source.
  2. From the data source type dialog select SSHPipe from the list of data source types and click Ok
  3. In the resulting dialog, Fill in the fields as follows:
   |  Field Name         |   Contents                           |
   +-------------- ------+--------------------------------------+
   | Host name:          | Name of the host in which Readout    |
   |                     | will run (I used spdaq06).           |
   +---------------------+--------------------------------------+
   | Readout Program:    | ddas bin dir/Readout I used:         |
   |                     | /usr/opt/ddas/1.0-000/bin/Readout    |
   +---------------------+--------------------------------------+
   | Working Directory   | Set this to the directory with the   |
   |                     | card configuration - uncheck         |
   |                     | 'Same as readout' and browse to that |
   |                     | directory                            |
   +---------------------+--------------------------------------+
   | Command line options| I used:                              |
   |                     | --ring=0400x_ddas_1 --sourceid=0     |
   +---------------------+--------------------------------------+

Click the Start button to start your data source (Readout).

Let's test check that all of this works. I ran the dumper as follows:

$DAQBIN/dumper --source=tcp://spdaq06/0400x_ddas_1 --count=50

start a run by clicking the Begin button. Make sure that data is emitted by the dumper. End the run and exit the ReadoutShell.

Setting up the event builder

Starting the event builder is a matter of creating a ReadoutCallouts.tcl script which

Here is an example of a ReadoutCallouts.tcl script that does this for our case:

package require evbcallouts                   (1)


proc OnStart {} {                            

    EVBC::useEventBuilder                    (2)

    EVBC::initialize -gui 1 -destring 0400x -glombuild 1 -glomdt 10 (3)
}
EVBC::registerRingSource tcp://spdaq06/0400x_ddas_1 \
      $::env(DDAS_LIB)/libddastimestampextractor.so 0 {Crate\ 1\ data} (4)
  1. This line incorporates the event builder API into the ReadoutGUI.
  2. EVBC::useEventBuilder installs the event builder state change handlers into the readout gui. The state change handlers allow the event builder API to take actions when the run state changes.
  3. EVBC::initialize starts the event builder. The parameter you'll want to adjust is -glomdt, which specifies the event builder's coincidence window. Generally speaking, the units for this value are the same as the units of the timestamp. If you are only dealing with a single DDAS Readout program, your coincidence window will be in units of nanoseconds, because the timestamps are converted to units of nanoseconds prior to being outputted from the Readout program.
  4. EVBC::registerRingSource adds a ring data source to be started at the beginning of a run. This is a feeder from a ring buffer to an event builder that runs in the localhost. The parameters are, in order, the source ringbuffer URL, the timestamp extraction library the source id (must be the same as the –sourceid parameter for the Readout), and a comment that will be displayed in the event builder for this source. Spaces in the comment must be escaped as shown..

If we point a dumper to the 0400x ring (the output ring from the event builder), here's a typical event:

-----------------------------------------------------------
Event 76 bytes long
Body Header:
Timestamp:    5991399620
SourceID:     0
Barrier Type: 0
004c 0000 80c4 651d 0001 0000 0000 0000
0034 0000 0000 0000 0034 0000 001e 0000
0014 0000 80c4 651d 0001 0000 0000 0000
0000 0000 000c 0000 0064 0c0c 4020 0008
267a 23b6 0000 1ad3 2943 0000

Let's pick apart the data for this event. Note that it is much bigger than before. This is because the data are framed with a fragment header which also encapsulates a ring item header that in turn wraps the event body. To get on familiar territory, note that in the last lines:

0000 0000 000c 0000 0064 0c0c 4020 0008
267a 23b6 0000 1ad3 2943 0000

Our raw event begins at the word 0x000c. We're going to pick apart the data prior to this, much of which is only relevant if you are using multiple crates. Here is the short summary of the information prior to the raw event:

0x0000004c (32 bits)          -- total event size in bytes.
0x00000001651d80c4 (64 bits)  -- The event builder fragment timestamp.  This is 10x
                                 the digitizer timestamp.
0x00000000 (32 bits)          -- Data source id (value of --sourceid on readout).
0x00000034 (32 bits)          -- Size of fragment payload (not self inclusive)

Next is the ring item header:
 
0x00000034 (32 bits)         -- Number of bytes in the ring item (self inclusive)
0x0000001e (32 bits)         -- Ring item type (PHYSICS_EVENT).
0x00000014 (32 bits)         -- Number of bytes in the body header.  The body
                                header contains information used by the event
                                builder.
0x00000001651d80c4 (64 bits) -- Fragment timestamp.
0x00000000 (32 bits)         -- Fragment source id.
0x00000000 (32 bits)         -- Fragment body header

0x0000000c (32 bits)         -- First word of the fragment.

Note that if several fragments got glued together to create a single event (as will happen if you tee off your pulser into several channels), there will be a ring item in the fragment body for each fragment that made the coincidence requirement.


Analyzing the data with SpecTcl.

SpecTcl is the lab-supported framework for analyzing data online. The DDAS comes with some tools to use in unpacking DDAS data within the context of SpecTcl, DAQ::DDAS::DDASUnpacker and DAQ::DDAS::DDASBuiltUnpacker. These both provide support for parsing the raw data of DDAS for you. It is in your best interest to leverage these tools rather than writing your own parser. To learn how to use the DAQ::DDAS::DDASBuiltUnpacker in your SpecTcl, please refer to Analyzing DDAS Data in SpecTcl Tutorial.