Pttp
From PTAGISWiki
Contents |
PIT Tag Transfer Protocol (PTTP)
PTAGIS initiated an e-mail file upload capability in 1995-1996. While this provided a convenient mechanism for user to submit files to ptagis, it was prone to the corruption of the format of data (various mail interfaces like to change spaces to tabs, indent lines of text, etc). The feedback mechanisms of that process didn’t provide immediate feedback. That process also did not provide a robust security mechanism.
The old process loosely coupled the client data with the archived data stored on PTAGIS. This loose coupling provided opportunities for inconsistencies to develop between the data stored on the client and the archived data stored in the PTAGIS database.
The new PTTP process was architected during the spring of 2000. Its design tightly couples the client data with the archived data by providing an integrated, client-side process (Pittag2 and Minimon) that can provide facilities to submit data to PTAGIS.
See original design documentation.
Original Requirements Specification (August, 2000)
PIT Tag File Transfer Process
Scope
Replace the PTAGIS E-mail file upload interface with a new process, PTTP.
The Problem
PTAGIS initiated an e-mail file upload capability in 1995-1996. While this provides a convenient mechanism for user to submit files to ptagis, it is prone to the corruption of the format of data (various mail interfaces like to change spaces to tabs, indent lines of text, etc). The feedback mechanisms of the existing process don't provide immediate feedback. The existing process also do not provide a robust security mechanism.
The existing process loosely couples the client data with the archived data stored on PTAGIS. This loose coupling provide opportunities for inconsistencies to develop between the data stored on the client and the archived data stored in the PTAGIS database.
The new process tightly couples the client data with the archived data by providing an integrated, client-side process that submits data to PTAGIS.
Assumptions
- Availability of the database should not be required in order to receive files or send an acknowledgment to / from the pttp client.
- Existing process which is to send multi-mon files to intdata with subject of MULTIMON will continue to be supported, but will not be changed - remain as parallel legacy system. Similarly, while in transition, process for INTERROGATION files will remain functional.
Existing Process
The existing process utilizes two e-mail aliases, intdata@ptagis.org and pitdata@ptagis.org. The intdata alias points to process pitdata.pl. The pitdata alias points to fdc_receive_email.pl. Both of these processes decode mime encapsulated attachments, and uncompress the various file types. The pitdata.pl process is responsible for calling the multi-conv.pl process which does some simple validations and converts files created by multimon into the interrogation file format specified by the PIT tag specifications document. The interrogation files are deposited in the
~pittag/data/interrogation/hold directory.
Fdc_receive_email performs the same mime decoding and uncompress functions as pitdata.pl, but for the Field Data file types (TAGGING, RELEASE & MORTALITY). After the files are un-bundled, the rest of the FDVL process take over to validate the files and load them into the database tables.
New Process: PTTP_SERVER
The PTTP process is based upon FTP services. The user collects PIT tag data using PC client tools provided by PTOC pittag2, minimon) and utilizes the new PTTP process to submit the files to the PTAGIS server.
The PTTP on the client, submits a data package to the users PTOC_STAGED directory on the PTAGIS server. The server deamon, /usr/local/admin/bin/watchFtp, looks at the ftp log file. When it sees that a file got sent to any PTOC_STAGED directory, then it calls on the pttp_server process which performs the work necessary to process the package and communicate status of processing to the user.
PTTP SERVER
The pttp requests on the client are always served by the one pttp_server process. The pttp_server process does four things:
- Optionally, process a header file contained in the package. The header file contains configuration information provided by client, and any special processing instructions for the server to perform.
- Un-bundle and dispatch the interrogation files to the interrogation data loader (IDL) via pttp_idl or field data files to the field data validation and loader (FDVL) via pttp_fdvl.
- Provides an "acknowledgment" file for the client which contains status information about the processing of the package.
- Optionally, processes various client requests. This `exchange server' (pttp_services) process will allow opportunities to provide load and processing status of prior package submissions, refreshing of validation tables (valid_codes, rkm), driver updates, software revision updates, and more.
PTTP HEADER FILE
If a package contains a header file, it is named PTTP_HEADER. By default, if the header does not exist in the package, then it is assumed that the files contained in the package are interrogation files. In this instance, a minimal acknowledgement file is returned.
The header file will not only contain configuration information for the clients processing request, but will contain various information about the client. For example, information that identifies the client machine, the user's name, site id, and other information could be stored in the header file.
Information contained in the header file could represent fairly complex objects. A file format that can persist those objects, such as an abstract syntax tree, could be considered upon implementation.
PTTP_FDVL
This process recognizes a test or debug mode flag sent by the client. When this flag is present the entire execution path of this process is exercised, but no data are loaded into the production PTAGIS database tables.
This process provides functionality equivalent to the existing fdc_receive_email.pl process - the main difference being, that we're not collecting files via e-mail. In addition, this process is responsible to bubble the validation status messages back to pttp_server of the files that were submitted in the package.
PTTP_IDL
This process also recognizes a test or debug mode flag sent by the client. When this flag is present the entire execution path of this process is exercised, but no data are loaded into the production PTAGIS database tables.
The existing e-mail interface for uploading interrogation files will remain in place for the foreseeable future. The new process will be in concurrent operation with the e-mail processes.
The pttp_idl process will process two type of interrogation data files. The first type is the MULTIMON file type. For now, focus on this: The second type is the INTERROGATION file type specified in the 2000 PIT Tag Specifications Document (to be published real soon now.).
When pttp_idl (In PK-Zip compatible archive , even if one file .) processes either of these file types, it verifies that it contains valid set of header and trailer records as specified in the SpecDoc. Then creates a comma delimited flat file ($File.detail) union compatible with table obs_temp1 (see start_serial.sh). Remove the following sentence:
This file is to be stored in the location specified by $INTERROGATION_OUTPUT.
Appropriate changes will need to be made to start_serial.sh and other driver processes (e.g., IDL, FDA, etc.).
We need to perform some analysis here: In fact, it is desirable to load the $File.detail as interrogation files arrive, if the database server is running.
Pttp_idl is responsible for calling the appropriate TASS and SBCA processes for each interrogation file. A parser for the INTERROGATION file type will need to be constructed for TASS and SBCA.
The number of interrogation records in the interrogation file (exclusive of any test tags or timer tags) should be stored in the item_count column of the fd_submit table.
Make recommendations as to what messages, and how to send: The acknowledgement messages, created by pttp_idl (and assembled by pttp_server), are to be determined at a future time. One suggestion would be to provide a count of the type of records within the interrogation file (e.g., DATA records, PLC messages, TRANSCEIVER messages, time ticks, etc.).
NOTE: All of this is supposed to work for INT, MULT, and FDVL-style data.
PTTP_SERVICES
Initially, PTTP will provide file load status reports at the request of the pttp client. Status reports will be based upon information stored in the fd_file and fd_submit tables in PTAGIS.
In the future, it is envisioned that this (or a similar) process will have expanded capabilities. Ideas include providing updates to validation and river kilometer codes, device driver updates, software revisions and updates, PTAGIS server status messaging, change user's ptagis password, and other functions.
