Programmers Notes

CSV tables

In some modules CSV tables may by imported (mk_master_cal).

The implemented style is:

serial

chopper

f

a

p

stddev a

stddev p

1

1

1.232800e-01

2.000200e-01

8.823600e+01

2.734738e-04

2.734738e-04

1

0

1.000000e+00

1.892500e-01

1.112900e+02

1.312173e-03

1.312173e-03

serial, chopper, f, a, p, stddev a, stddev p
1, 1, 1.232800e-01, 2.000200e-01, 8.823600e+01, 2.734738e-04, 2.734738e-04
1, 0, 1.000000e+00, 1.892500e-01, 1.112900e+02, 1.312173e-03, 1.312173e-03

CVS tables are always TEXT. When performing actions based on tables, be aware that 1.0e+00 and 1.0E+00 are NOT THE SAME
Also in case of mathematics a string must be converted into a number and back:
.. 1.0e+00 -> double is ok, 1.0e+00 -> int is not ok; so internally we always go 1.0e+00 -> double -> int (in case).
In case of string data we always compare lower case!
All frequencies you use have to be sorted from low to high.

ATS header

ats stands for advanced timeseries.
In languages such as C++ you do simlilar like (here with C++ & Qt)

ATSHeader_80   bin_atsheader;   //!< binaray header for read/write

....
file.setFileName("myatsfile.ats");
file.open(QIODevice::ReadOnly))
QDataStream qds;
qds.setDevice(file);
qds.readRawData((char*) &bin_atsheader, sizeof(bin_atsheader));
qDebug() << "samples: " << bin_atsheader.uiSamples;

In other languages reading a binary file is maybe similar.

#ifndef ATSHEADER_80_DEF_H
#define ATSHEADER_80_DEF_H

#include <cstdint>

//
//
//
//         array of chars are not NULL terminated !
//         achChanType   [2] = "Ex" is not NULL terminated
//
//

#define C_ATS_CEA_NUM_HEADERS   1023     /*!< maximum number of 1023 headers of the ATS sliced file.*/


/*!
\brief The ATSSliceHeader_s struct
A sliced ats file ALWAYS contains a default header and C_ATS_CEA_NUM_HEADERS (1023) = 1024 total.<br>
The FIRST and DEFAULT slice contains ALL information (total samples of all slices)<br>
The 0 ... 1022 slices (=  ATS_CEA_NUM_HEADERS ) contain the individual descriptions<br>
start tsdata = 400h = 1024th byte<br>
start of CEA data is 400h + 3FFh * 20h = 83E0h<br>
                        1024 + 1023 * 32  = 33760<br>

assuming 3 slices of 4096 iSamples will be 12,288 samples in the header,
followed by 3 slice headers with 4096 iSamples each
and 1019 empty slice headers<br>
the data has to sorted out on/after 83E0h according to the slice info

*/

struct ATSSliceHeader_1080
{
    std::uint32_t uiSamples;                    //!< 000h  number of samples for this slice
    std::uint32_t uiStartDateTime;              //!< 004h  startdate/time as UNIX timestamp (UTC)
    double        dbDCOffsetCorrValue;          //!< 008h  DC offset correction value in mV
    float         rPreGain;                     //!< 010h  originally used pre-gain (GainStage1) - LSB is the same for all slices
    float         rPostGain;                    //!< 014h  originally used post-gain (GainStage2) - LSB is the same for all slices
    std::int8_t   DCOffsetCorrOn;               //!< 018h  DC offset correction was on/off for this slice
    std::int8_t   reserved[7];                  //!< 020h  reserved bytes to get to word / dword boundary
};

/*! @todo TX info
maybe UTM Zone number
maybe UTM letter
Tx times six double at least
Tx base freq

*/

/*!
\brief The ATSComments_80 struct for ats header 80,81 and 90 <br>
Some software will automatically convert to higher version<br>
Some fields are external - the user has to set then; examples:<br>
northing, easting, UTM Zone : these can be used for high density grids<br>
*/
struct ATSComments_80 {
    char          achClient        [16];        //!< 000h  UTF-8 storage, used in EDI
    char          achContractor    [16];        //!< 010h  UTF-8 storage, used in EDI
    char          achArea          [16];        //!< 020h  UTF-8 storage, used in EDI
    char          achSurveyID      [16];        //!< 030h  UTF-8 storage, used in EDI
    char          achOperator      [16];        //!< 040h  UTF-8 storage, used in EDI
    char          achSiteName     [112];        //!< 050h  UTF-8 storage, no BOM at the beginning!; WORST case = 28 Chinese/Japanese chars (multibyte chars)
    char          achXmlHeader     [64];        //!< 0C0h  UTF-8 storage  points to the corresponding XML file, containing addtional information for this header; no PATH=XML file is in the same directory as the ats file
    // deleted September 2018 char          achComments   [512];          //!< 100h  UTF-8 storage  comment block starts (comment field can start with "weather:" but has no meaning
    // new September 2018
    char          achComments     [288];        //!< 100h  UTF-8 storage  comment block starts (comment field can start with "weather:" but has no meaning
    char          achSiteNameRR   [112];        //!< 220h  UTF-8 storage, no BOM at the beginning!; WORST case = 28 Chinese/Japanese chars (multibyte chars)
    char          achSiteNameEMAP [112];        //!< 290h  UTF-8 storage, no BOM at the beginning!; WORST case = 28 Chinese/Japanese chars (multibyte chars); INDICATES a default EMAP center station, where we expect the H (magnetic)


};


/*!
\brief The ATSHeader_80 struct<br>
reference is lat and long; additional entries like northing and easting should be empty // not used<br>
these entries can be overwritten by the user in case of high density grids <br>
The site_name (site name, achSiteName, is in the comment field; the site_no, site_num, site_number, site number is nver stored; this is part of external numeration
*/
struct ATSHeader_80 {
    std::uint16_t uiHeaderLength;               //!< 000h  length of header in bytes (default 1024 and  33760 for CEA)
    std::int16_t  siHeaderVers;                 //!< 002h  80 for ats, 81 for 64bit possible / metronix, 1080 for CEA / sliced header

    // This information can be found in the ChannelTS datastructure
    std::uint32_t uiSamples;                    //!< 004h  amount of samples (each is a 32bit / 64bit int INTEL byte order little endian) in the file total of all slices; if uiSamples == UINT32_MAX, uiSamples64bit at address 0F0h will be used instead; uiSamples64bit replaces uiSamples in that case; do not add!
    //!<
    float         rSampleFreq;                  //!< 008h  sampling frequency in Hz
    std::uint32_t uiStartDateTime;              //!< 00Ch  unix TIMESTAMP (some computers will revert that to negative numbers if this number is grater than 32bit signed); 2106-02-07T06:28:14 is the END of this format
    double        dblLSBMV;                     //!< 010h  least significant bit in mV ()
    std::int32_t  iGMTOffset;                   //!< 018h  not used, default 0; can be used as "UTC to GMT"
    float         rOrigSampleFreq;              //!< 01Ch  sampling frequency in Hz as ORIGINALLY recorded; this value should NOT change (for example after filtering)

    //The required data could probably found in the HardwareConfig
    std::uint16_t uiADUSerNum;                  //!< 020h  Serial number of the system (logger)
    std::uint16_t uiADCSerNum;                  //!< 022h  Serial number of the ADC board (ADB)
    std::uint8_t  uiChanNo;                     //!< 024h  Channel number
    std::uint8_t  uiChopper;                    //!< 025h  Chopper On (1) / Off (0); e.g. chopper is on for 512Hz and lower

    // Data from XML Job-specification
    char          achChanType   [2];            //!< 026h  (Ex, Ey, Ez, Hx, Hy, Hz, Jx, Jy, Jz, Px, Py, Pz, Rx, Ry, Rz and so on)
    char          achSensorType [6];            //!< 028h  (MFS06 MFS06e MFS07 MFS07e MFS10e SHFT02 SHF02e FGS02 FGS03 FGS03e etc. e.g. the "-" in MFS-06e is skipped)
    std::int16_t  siSensorSerNum;               //!< 02Eh  Serial number of connected sensor

    float         rPosX1;                       //!< 030h  e.g. South negative [m]
    float         rPosY1;                       //!< 034h  e.g. West negative [m]
    float         rPosZ1;                       //!< 038h  e.g. top, sky [m]
    float         rPosX2;                       //!< 03Ch  e.g. North positive [m]
    float         rPosY2;                       //!< 040h  e.g. East positive [m]
    float         rPosZ2;                       //!< 044h  e.g. bottom, earth [m]

    // not used any more use pos values!!; GUI interfaces using Length and direction MUST calculate pos x,y,z and set above.
    float         rDipLength;                   //!< 048h  e.g. to be calculated; should not be used - my be over written in FUTURE
    // not used any more use pos values!!
    float         rAngle;                       //!< 04Ch  e.g. to be calculated; should not be used - my be over written in FUTURE

    // Data from Selftest
    float         rProbeRes;                    //!< 050h  [ohm]
    float         rDCOffset;                    //!< 054h  [mV]
    float         rPreGain;                     //!< 058h  e.g. Gain Stage 1
    float         rPostGain;                    //!< 05Ch  e.g. Gain Stage 2

    // Data from status information ?
    std::int32_t  iLatitude;                    //!< 060h  must be used, UNIT = milli seconds
    std::int32_t  iLongitude;                   //!< 064h  must be used, UNIT = milli seconds
    std::int32_t  iElevation;                   //!< 068h  must be used, UNIT = cm
    char          byLatLongType;                //!< 06Ch  'G' default, 'U' user, GPS should be used
    char          byAddCoordType;               //!< 06Dh  'U' = UTM, default empty
    std::int16_t  siRefMeridian;                //!< 06Eh  default empty, should not be used (external)



    //!@todo can we store 64bit time and 64bit samples here ??
    double        dblNorthing;                  //!< 070h  also xcoord should not be used, default 0 (external)
    double        dblEasting;                   //!< 078h  also ycoord should not be used  default 0 (external)
    char          byGPSStat;                    //!< 080h  '-' unknown, 'N' no fix, 'C' full fix
    char          byGPSAccuracy;                //!< 081h  '0' - not used
    std::int16_t  iUTCOffset;                   //!< 082h  UTC + iUTCOffset = GPS time for example; can be used for other offsets, not used, default 0
    char          achSystemType[12];            //!< 084h  MMS03e GMS05 ADU06 ADU07 ADU08 ADU09 SPAMMKV SPAMMKIII

    // Data from XML-Job specification
    char          achSurveyHeaderName [12];     //!< 090h  UTF-8 storage  free for usage
    char          achMeasType          [4];     //!< 09Ch  free for usage MT CSMT


    double        DCOffsetCorrValue;            //!< 0A0h  DAC offset double
    std::int8_t   DCOffsetCorrOn;               //!< 0A8h  DC offset was switched on (1) or off(0)
    std::int8_t   InputDivOn;                   //!< 0A9h  inputput divider on(1) off(0); e.g when coil was connected
    std::int16_t  bit_indicator;                //!< 0AAh  0 = 32bit int INTEL byte order, 1 = 64bit INTEL byte order, little endian; since atsheader version 81
    char          achSelfTestResult [2];        //!< 0ACh  'NO' or 'OK'
    std::uint16_t numslices;                    //!< 0AEh  number of slices used (1....1024, 1 is the first, that is list.size() )

    //std::int16_t  calentries    // max 128 entries

    // Were the following fields ever used ?
    std::int16_t  siCalFreqs;                   //!< 0B0h  not used  (external)
    std::int16_t  siCalEntryLength;             //!< 0B2h  not used  (external)
    std::int16_t  siCalVersion;                 //!< 0B4h  not used  (external)
    std::int16_t  siCalStartAddress;            //!< 0B6h  not used, never used  (external)


    char          abyLFFilters [8];             //!< 0B8h  is a bitfield

    char          achUTMZone  [12];             //!< 0C0h  not used  (external)  (formerly abyADU06CalFilename) 32U or 01G, 32UMD7403 : alway NNC od NNCCNNNN
    std::uint32_t uiADUCalTimeDate;             //!< 0CCh  not used  (external)
    char          achSensorCalFilename [12];    //!< 0D0h  not used  ("SENSOR.CAL") (external)
    std::uint32_t uiSensorCalTimeDate;          //!< 0DCh  not used  (external)

    float         rPowerlineFreq1;              //!< 0E0h  e.g. empty  (external)
    float         rPowerlineFreq2;              //!< 0E4h  e.g. empty  (external)
    char          abyHFFilters[8];              //!< 0E8h  is a bitfield


    // IF uiSamples == UINT32_MAX you find
    std::uint64_t uiSamples64bit;               //!< 0F0h  amount of samples (each is a 32bit /64bit int) in the file total of all slices; ONLY used in case uiSamples == UINT32_MAX; else 0; REPLACSES the uiSamples; do not add

    //double        OriginalLSBMV;              //!< 0F0h  NOT USED ANYMORE! orig lsb from selftest without gains; used for ADC values
    float         rExtGain;                     //!< 0F8h  for external satellite box
    char          achADBBoardType[4];           //!< 0FCh  LF HF or MF or BB


    ATSComments_80 tscComment;                  //!< 100h

    // size if comments is total 100h + 100h data  + (512byte comments = 200h) = 400h
    // start tsdata = 400h = 1024th byte
    // start of CEA header is 400h + 3FFh * 20h = 83E0h
    //                        1024 + 1023 * 23  = 33760
};


#endif // ATSHEADER_80_DEF_H

LSB

LSB == least significant bit

The dblLSBMV multiplied with the int32_t data values results in the measured values in mV Gains and others are included in th LSB.
To scale the E field use rPosX1 … rPosZ2 values and NOT! the rDipLength and rAngle; these values are obsolete
metronix software assumes for the magnetic field that in case all rPosX1 … rPosZ2 are zero, that Hx is pointing North, Hy East and Hz down.

ATM header

That is the binary marker file

struct atmheader {
  qint16 siHeaderLength;
  qint16 siHeaderVers;
  quint32 iSamples;
};

Followed by booleans (for each sample); true = will be processed, false not.

Calibration File

E.g. the calibration file has to sections: “chopper on” and “chopper off”.
First column indicateds frequency in Hz.
Second column amplitude in V/(nT * Hz)
Third phase in degree (0-360, not radians).

Be default the chopper should be ON for recordings with equal/less than 512 Hz.
For chopper on the lower frequencies can be extended by the theroretical function. For chopper off this is not possible.

The amplitude is normalized by f (as you can see from the units). So for lower frequencies the amplitude of the MFS-06e converges agains 0.2 V/(nT*Hz) (and gives a horizontal line in a plot for the chopper on data).

For usage in your software you may

  • multiply magnitude by f

  • make a complex number from magnitude and phase

  • if you take the mV data from the ats, multiply calibration by 1000 in case

  • interpolate the calibration data onto your nodes of the FFT

  • divide by the calibration

Calibration measurement with NumetriQ PSM 3750 - 0.1.100000.1.33
Metronix GmbH, Kocherstr. 3, 38120 Braunschweig

Magnetometer: MFS06e#727    Date: 17/01/12    Time: 12:19:57


FREQUENCY    MAGNITUDE    PHASE
Hz           V/(nT*Hz)    deg
Chopper On
+1.0000E-01  +1.9996E-01  +8.8589E+01
+1.2328E-01  +2.0008E-01  +8.8225E+01
+1.5199E-01  +2.0014E-01  +8.7796E+01
+1.8738E-01  +1.9987E-01  +8.7329E+01
+2.3101E-01  +1.9982E-01  +8.6762E+01
+2.8480E-01  +1.9984E-01  +8.5986E+01
+3.5111E-01  +1.9961E-01  +8.5047E+01
+4.3287E-01  +1.9919E-01  +8.3911E+01
+5.3367E-01  +1.9863E-01  +8.2537E+01
+6.5793E-01  +1.9767E-01  +8.0848E+01
+8.1113E-01  +1.9619E-01  +7.8796E+01
+1.0000E+00  +1.9432E-01  +7.6298E+01
+1.2329E+00  +1.9151E-01  +7.3290E+01
+1.5199E+00  +1.8781E-01  +6.9669E+01
+1.8738E+00  +1.8220E-01  +6.5470E+01
+2.3101E+00  +1.7438E-01  +6.0673E+01
+2.8480E+00  +1.6449E-01  +5.5318E+01
+3.5112E+00  +1.5217E-01  +4.9534E+01
+4.3288E+00  +1.3785E-01  +4.3570E+01
+5.3367E+00  +1.2226E-01  +3.7647E+01
+6.5793E+00  +1.0622E-01  +3.2028E+01
+8.1113E+00  +9.0622E-02  +2.6892E+01
+1.0000E+01  +7.6212E-02  +2.2313E+01
+1.2329E+01  +6.3445E-02  +1.8389E+01
+1.5199E+01  +5.2247E-02  +1.5069E+01
+1.8738E+01  +4.2964E-02  +1.2247E+01
+2.3101E+01  +3.5147E-02  +9.8945E+00
+2.8480E+01  +2.8661E-02  +8.0352E+00
+3.5112E+01  +2.3266E-02  +6.4090E+00
+4.3288E+01  +1.8976E-02  +5.1069E+00
+5.3367E+01  +1.5470E-02  +3.9140E+00
+6.5793E+01  +1.2495E-02  +3.0121E+00
+8.1113E+01  +1.0154E-02  +2.2737E+00
+1.0000E+02  +8.2466E-03  +1.5682E+00
+1.2329E+02  +6.6826E-03  +8.7741E-01
+1.5199E+02  +5.4120E-03  +3.2423E-01
+1.8738E+02  +4.3959E-03  -2.9814E-01
+2.3101E+02  +3.5657E-03  -8.7995E-01
+2.8480E+02  +2.8912E-03  -1.5233E+00
+3.5112E+02  +2.3408E-03  -2.1688E+00
+4.3288E+02  +1.9000E-03  -3.0046E+00
+5.3367E+02  +1.5399E-03  -3.9069E+00
+6.5793E+02  +1.2478E-03  -4.9982E+00
+8.1113E+02  +1.0105E-03  -6.2920E+00
+1.0000E+03  +8.1795E-04  -7.8551E+00
+1.2329E+03  +6.6137E-04  -9.7494E+00
+1.5199E+03  +5.3415E-04  -1.2052E+01
+1.8738E+03  +4.3092E-04  -1.4858E+01
+2.3101E+03  +3.4918E-04  -1.8440E+01
+2.8480E+03  +2.7656E-04  -2.3375E+01
+3.5112E+03  +2.1821E-04  -2.8876E+01
+4.3288E+03  +1.6854E-04  -3.6126E+01
+5.3367E+03  +1.2103E-04  -4.4781E+01
+6.5793E+03  +8.0732E-05  -4.5047E+01
+8.1113E+03  +6.6265E-05  -4.4906E+01
+1.0000E+04  +5.3269E-05  -5.2671E+01


FREQUENCY    MAGNITUDE    PHASE
Hz           V/(nT*Hz)    deg
Chopper Off
+1.0000E+00  +1.8929E-01  +1.1098E+02
+1.2329E+00  +1.9877E-01  +1.0178E+02
+1.5199E+00  +2.0310E-01  +9.2408E+01
+1.8738E+00  +2.0205E-01  +8.2979E+01
+2.3101E+00  +1.9525E-01  +7.3551E+01
+2.8480E+00  +1.8380E-01  +6.4373E+01
+3.5112E+00  +1.6822E-01  +5.5546E+01
+4.3288E+00  +1.4990E-01  +4.7282E+01
+5.3367E+00  +1.3074E-01  +3.9765E+01
+6.5793E+00  +1.1170E-01  +3.3138E+01
+8.1113E+00  +9.4077E-02  +2.7353E+01
+1.0000E+01  +7.8347E-02  +2.2450E+01
+1.2329E+01  +6.4595E-02  +1.8346E+01
+1.5199E+01  +5.2931E-02  +1.4879E+01
+1.8738E+01  +4.3376E-02  +1.2120E+01
+2.3101E+01  +3.5403E-02  +9.7871E+00
+2.8480E+01  +2.8760E-02  +7.8304E+00
+3.5112E+01  +2.3408E-02  +6.1176E+00
+4.3288E+01  +1.8972E-02  +4.9934E+00
+5.3367E+01  +1.5398E-02  +4.0097E+00
+6.5793E+01  +1.2525E-02  +2.9188E+00
+8.1113E+01  +1.0166E-02  +2.1264E+00
+1.0000E+02  +8.2561E-03  +1.5565E+00
+1.2329E+02  +6.6903E-03  +8.5820E-01
+1.5199E+02  +5.4239E-03  +2.0043E-01
+1.8738E+02  +4.4025E-03  -3.0994E-01
+2.3101E+02  +3.5698E-03  -9.1526E-01
+2.8480E+02  +2.8946E-03  -1.5153E+00
+3.5112E+02  +2.3489E-03  -2.2518E+00
+4.3288E+02  +1.9026E-03  -2.9940E+00
+5.3367E+02  +1.5419E-03  -3.8998E+00
+6.5793E+02  +1.2494E-03  -4.9771E+00
+8.1113E+02  +1.0118E-03  -6.2748E+00
+1.0000E+03  +8.1898E-04  -7.8198E+00
+1.2329E+03  +6.6221E-04  -9.7244E+00
+1.5199E+03  +5.3482E-04  -1.2031E+01
+1.8738E+03  +4.3142E-04  -1.4824E+01
+2.3101E+03  +3.4914E-04  -1.8435E+01
+2.8480E+03  +2.7691E-04  -2.3267E+01
+3.5112E+03  +2.1863E-04  -2.8766E+01
+4.3288E+03  +1.6899E-04  -3.6023E+01
+5.3367E+03  +1.2149E-04  -4.4705E+01
+6.5793E+03  +8.1011E-05  -4.5075E+01
+8.1113E+03  +6.6395E-05  -4.4922E+01
+1.0000E+04  +5.3337E-05  -5.2637E+01

ATSS header

sats stands for advanced timeseries stream.
The stream data itself is stored in .atss as pure binary data streams, 64 bit double IEEE 754 as most processor architectures use it.
The double is is used by default in Python/NumPy, MatLab, C++ and others.

The JSON header is separated from the stream. This allows a continuous data transmission where the stream data only changes at the end. The sync client can use “rsync –append” in order to resume the data transfer.

The JSON header can be transferred with the “rsync -z” option (compress) because it is an ASCII file.

In addition a .sql3 (SQLite file version 3) is written into the data directory, containing the log data of the recording.

The filename is splitted by the underscore tag _ with a default length of 3 and a letter bit_indicator

008_C01_R000_TEy_512H.sats ADU-08e data stream recorded with channel 1 as Ey with 512 Hz
067_C00_R001_TEx_8s.sats ADU-08e data stream recorded with channel 0 as Ex with one sample every 4 seconds

serial ADU

Channel 0 - N

Run 0 - N

Type (of channel)

Sample Frequency or Sample Period

H(ertz) or S(econds)

ext

008

C001

R000

TEy

512

H

atss

067

C000

R001

TEx

8

s

atss

.