CheckIdl
From PTAGISWiki
Useful aliases as user ptagdev
Click the discusion tab to see what's going on...
Add discussion to the top of the discussion page.
jld - Show start and finish times of most recent p3_daily jobs see_idl - shows IDL files to load fc_idl - shows number of files to load rc_idl - shows number of records to load
You may want to use the work-script I've been compiling today as I ride herd on IDL. It's on 'sockeye' at /home/ptagdev/ptagis3/sys_config_chores/ProbLog. The file is:
ProblemDescription_ProcessCreationSpike.txt
In the file you'll find step-by-step the commands I've issued so far today. Also in that directory is the file I used when catching up from the scheduling problem that occurred Monday night. It's called
ProblemDescription_LongRunningP3_daily_4.txt
It may be useful to have a look at this file, to get an overview of what I did on Tuesday - which will be practically identical to things you or someone will need to do this afternoon.
Sorry to leave when things are balled up like this. I'll give you a call before I take off, so you can ask questions that may come up when you look at these work-script files.
Pacific States Marine Fisheries Commission
PTAGIS3 Interrogation Data Loader
Long-Running 'p3_daily_4' Job
Ver 1.0 09 May 2006
(Last Update: 11:30)
History:
--------- -------------- -------------------------------------------------
09 May 06 Doug Clough Creation, at about 07:00 PDT
==OVERVIEW==
'p3_daily_2.sh' stopped because p3_daily_4.sh launched at 18:15 PST ran for 10-
minutes beyond the p3_daily_2.sh launch time of 00:15 PST. This meant 90-plus
files (over 235,000 rows) would have to be processed by the p3_daily_3.sh run
launched at 06:15 PST on 09 May.
As detailed below, the long run-time had several causes: First, this is a very busy
time of year. On top of that, large quantities of PIT-tagged fish are being released
at GOJ and GRJ this year, resulting in unusually large numbers of interrogations at
these sites. Further, two acclimation-pond interrogation files (from CCP and GRP)
were placed in the input queue yesterday, just prior to the 18:15 PST IDL launch.
Similarly, TWX interrogation files spanning 2 thru 9 May had been presented for
loading. Finally, yesterday being a Monday, CEA was tasked with producing 'weekly'
and '2-week' aggregations during the p3_daily_4.sh run in question.
Without the CCP and GRP files, IDL would have finished more than 20-minutes earlier;
without the TWX files, it would have finished about 15-minutes earlier: In either
case, the scheduling problem would not have arisen.
==INTERVENTION==
Done 1. Disable 'cron' processing in the 'pittag' account ...
> cd /home/pittag/bin
> su - pittag
OK > crontab -r
DPC Tue May 9 08:16:25 PST 2006
Done 2. Manually adjust 'cronfile' to launch another p3_daily_3.sh just as
soon as the one launched at 06:15 PST completes
OK > cp -p cronfile cronfile.09may06
DPC Tue May 9 11:01:02 PST 2006
As 'ptagdev', watch for p3_daily_3 completion expected ~12:30 PDT ...
> jld
As 'pittag' ...
> cd /home/pittag/bin
OK > vi cronfile.09may06
Re-schedule to launch p3_daily_3.sh at 12:45 PDT
OK > crontab cronfile.09may06
Once the job has launched ...
OK > crontab -r
DPC Tue May 9 11:45:51 PST 2006
As 'ptagdev', resume watching for p3_daily_3 completion ...
> jld
As 'pittag' ...
> cd /home/pittag/bin
OK > vi cronfile.09may06
Re-schedule to launch p3_daily_3.sh at 15:00 PDT
OK > crontab cronfile.09may06
Once the job has launched ...
OK > crontab -r
DPC Tue May 9 14:07:55 PST 2006
As 'ptagdev', resume watching for p3_daily_3 completion ...
> jld
Done 3. Re-enable 'cron' processing on the normal schedule ...
> cd /home/pittag/bin
> su - pittag
OK > crontab -r
OK > crontab cronfile
OK > crontab -l
DPC Tue May 9 15:34:15 PST 2006
-v-v-v-v-v-v-v-v-v-v-v-v Extra info in case it's needed -v-v-v-v-v-v-v-v-v-v-v-
'snapckp' is launched from 'root' crontab at 23:00
Here's how to enable / disable snapckp ...
> cat /home/pittag/etc/SNAPCKP_FLAG
# The VALUE set below controls invocation of ^snapckp^ from
# the root crontab. VALUE may be either RUN or SKIP.
VALUE: RUN
-^-^-^-^-^-^-^-^-^-^-^-^ Extra info in case it's needed -^-^-^-^-^-^-^-^-^-^-^-
IDL
06:38 PST 91 files 235802 rows
10,471 rows / 25 min = 418 rows / min
07:03 PST 76 files 225331 rows
15,863 rows / 22 min = 721 rows / min
07:25 PST 66 files 209468 rows
77,073 rows / 93 min = 829 rows / min
08:58 PST 44 files 132395 rows
At 800 rows / min, projected completion time is 2hr 45 min, or
11:45 PST --> 12:45 PDT
ANALYSIS
IDL
Dave mentioned that he submitted two acclimation pond interrogation files
at the end of the day yesterday; maybe these had an impact. I checked it
out:
CCP 14,987 rows, 23-minutes elapsed time
GRP 439 1-minute
Yes, without these, p3_daily_4.sh would have finished just before midnight;
15-minutes before the launch of p3_daily_2.sh
CEA
CEA_reportYTDData.pl Begins at Tue May 9 00:24:04 PST 2006
CEA_reportYTDData.pl Ends at Tue May 9 00:25:15 PST 2006
CEA_reportHistoricalData.pl Begins at Tue May 9 00:22:56 PST 2006
CEA_reportHistoricalData.pl Ends at Tue May 9 00:23:58 PST 2006
CEA_processHistoricalData.pl Begins at Mon May 8 19:59:32 PST 2006
daily
Loading data at Mon May 8 20:19:42 PST 2006
CEA_processHistoricalData.pl Begins at Mon May 8 20:20:27 PST 2006
weekly
CEA_processHistoricalData.pl Begins at Mon May 8 22:01:19 PST 2006
2-week
GOJ
Running report at Mon May 8 22:03:28 PST 2006
Loading data at Mon May 8 22:54:34 PST 2006
GRJ
Running report at Mon May 8 22:54:42 PST 2006
Loading data at Mon May 8 23:10:41 PST 2006
Running report at Mon May 8 22:54:42 PST 2006
Loading data at Mon May 8 23:10:41 PST 2006
ICH
Running report at Mon May 8 23:10:47 PST 2006
Loading data at Mon May 8 23:15:11 PST 2006
JDJ
Running report at Mon May 8 23:15:24 PST 2006
Loading data at Mon May 8 23:21:40 PST 2006
LMJ
Running report at Mon May 8 23:21:43 PST 2006
Loading data at Mon May 8 23:50:29 PST 2006
MCJ
Running report at Mon May 8 23:52:28 PST 2006
Loading data at Tue May 9 00:21:45 PST 2006
PRO
Running report at Tue May 9 00:21:58 PST 2006
Loading data at Tue May 9 00:22:18 PST 2006
WEA
Running report at Tue May 9 00:22:51 PST 2006
Loading data at Tue May 9 00:22:53 PST 2006
|--------|---------|---------|---------|---------|---------|---------|---------|
