Troubleshooting queries on bay

From PTAGISWiki

Contents 1 Background 2 Marton's first look 3 Restart ingres on bay and jack 4 Classpath issues 5 SREE portal shows no queries 6 CVS snapshot of changes on bay in the last week 7 known bug with CVS and jars 8 Email from Marton 9 Ryan's response 10 What is a task? 10.1 Answer 10.1.1 replicating this problem 10.1.2 description of the steps above 10.2 Still some problems with queries 11 scheduling queries fails 12 classpath on bay 13 classpath on pitblade 14 diff reports etools.jar is different 15 classpath on reedi 15.1 Next steps 15.2 Recent "replet not found" errors in production

Background

After deploying a new My Messages design module, the ability to run queries on bay and jack is broken. When a using the My Queries page, an attempt to run a query results in this error:

Compilation of '/usr/local/bea81/user_projects/domains/mydomain/./appserver_bay/.wlnotdelete/extract/appserver_bay_sree_sree/jsp_servlet/__runquery.java' failed: /usr/local/bea81/user_projects/domains/mydomain/./appserver_bay/.wlnotdelete/extract/appserver_bay_sree_sree/jsp_servlet/__runquery.java:113: cannot resolve symbol
probably occurred due to an error in /scheduleHelper.jspx line 16:
case inetsoft.report.io.Builder.GENERATED_REPORT:

Full compiler error(s):

/usr/local/bea81/user_projects/domains/mydomain/./appserver_bay/.wlnotdelete/extract/appserver_bay_sree_sree/jsp_servlet/__runquery.java:113: cannot resolve symbol
symbol  : variable GENERATED_REPORT 
location: class inetsoft.report.io.Builder
      case inetsoft.report.io.Builder.GENERATED_REPORT: //[ /scheduleHelper.jspx; Line: 16]
                                     ^
1 error

Marton's first look

Marton found the following:

The referenced inetsoft.report.io.Builder.GENERATED_REPORT is in fact defined in the inetsoft.report.io.Builder 
class, which is part of the sree_pro.jar file. The runquery.jsp imports inetsoft.report.io.Builder. So the reason 
for the error is some class loading issue. We could get this error of sree_pro.jar was not in the path, but that is 
unlikely - there were many more errors, all around. So perhaps different sree_pro.jar versions are in the path and 
an earlier version is taking precedence. Another possibility is that the two weblogic versions are different a 
little on pitblade and bay...

Restart ingres on bay and jack

I realized the ingres clients on bay and jack had not been restarted since blueback was restarted for the ingres patch. I restarted ingres and weblogic on bay and jack, but the problem persists.

Classpath issues

There could very well be class path issues as a result of my naive deployment of the files from pitblade to bay. In a few properties files, I deployed directly from pitblade to bay but this included machine-specific paths that had to be hand edited to make sure only the relevant changes were deployed. That is, I mistakenly copied some pitblade paths ("/dsk2/local/ptagis") into bay's environment ("/usr/local/pitweb"). This problem is probably caused by this sort of mistake.

SREE portal shows no queries

If I go directly to the SREE portal page (http://bay.psmfc.org:7003/sree/sr) instead of working through the PTAGIS interface, I see that SREE thinks I have no queries in the Replet Repository. That would indicate that SREE is confused by the fiddling that has been done to schedule.xml or the copying of repository.xml from reedi.

CVS snapshot of changes on bay in the last week

bash-3.00# cvs history -ea -z PST -D "last week"
M 2007-11-27 11:03 PST rday 1.6 nav.jsp                  ptagis-1.0/web/ptagis                    == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 log4j.properties         ptagis-1.0/web/ptagis/WEB-INF            == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.4 ptagis.properties        ptagis-1.0/web/ptagis/WEB-INF            == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 istatus.html             ptagis-1.0/web/sree                      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 myreporttemplates.jsp    ptagis-1.0/web/sree                      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 myrpttempaction.jsp      ptagis-1.0/web/sree                      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 runquery.jsp             ptagis-1.0/web/sree                      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 schaction.jsp            ptagis-1.0/web/sree                      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 schedule.jsp             ptagis-1.0/web/sree                      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 scheduleHelper.jspx      ptagis-1.0/web/sree                      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.3 sr.properties            ptagis-1.0/web/sree/WEB-INF              == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 web.xml                  ptagis-1.0/web/sree/WEB-INF              == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 audit.log                ptagis-1.0/web/sree/WEB-INF/classes      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 ldap.tmp                 ptagis-1.0/web/sree/WEB-INF/classes      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 repository.bak           ptagis-1.0/web/sree/WEB-INF/classes      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 repository.xml           ptagis-1.0/web/sree/WEB-INF/classes      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.3 sree.log                 ptagis-1.0/web/sree/WEB-INF/classes      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 ReportHelper.class       ptagis-1.0/web/sree/WEB-INF/classes/util == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 11:03 PST rday 1.2 queryList.jsp            ptagis-1.0/web/sree/qb                   == /usr/local/pitweb/ptagis-1.0/web
A 2007-11-27 11:04 PST rday 1.1 MyMessages.class         ptagis-1.0/web/ptagis/WEB-INF/classes/db == /usr/local/pitweb/ptagis-1.0/web
A 2007-11-27 11:04 PST rday 1.1 log4j.properties         ptagis-1.0/web/sree/WEB-INF              == /usr/local/pitweb/ptagis-1.0/web
A 2007-11-27 11:04 PST rday 1.1 NotificationAction.class ptagis-1.0/web/sree/WEB-INF/classes/util == /usr/local/pitweb/ptagis-1.0/web
A 2007-11-27 11:06 PST rday 1.1 action.jsp               ptagis-1.0/web/ptagis/home/mymessages    == /usr/local/pitweb/ptagis-1.0/web
A 2007-11-27 11:06 PST rday 1.1 display.jsp              ptagis-1.0/web/ptagis/home/mymessages    == /usr/local/pitweb/ptagis-1.0/web
A 2007-11-27 11:06 PST rday 1.1 list.jsp                 ptagis-1.0/web/ptagis/home/mymessages    == /usr/local/pitweb/ptagis-1.0/web
A 2007-11-27 11:06 PST rday 1.1 status.jsp               ptagis-1.0/web/ptagis/home/mymessages    == /usr/local/pitweb/ptagis-1.0/web
A 2007-11-27 11:06 PST rday 1.1 log4j-1.2.14.jar         ptagis-1.0/web/sree/WEB-INF/lib          == /usr/local/pitweb/ptagis-1.0/web
A 2007-11-27 11:06 PST rday 1.1 log4j.jar                ptagis-1.0/web/sree/WEB-INF/lib          == /usr/local/pitweb/ptagis-1.0/web
A 2007-11-27 11:06 PST rday 1.1 convertTasks.jsp         ptagis-1.0/web/sree/test                 == /usr/local/pitweb/ptagis-1.0/web
A 2007-11-27 11:06 PST rday 1.1 MyMessages.java          ptagis-1.0/src/db                        == /usr/local/pitweb/ptagis-1.0/src
A 2007-11-27 11:06 PST rday 1.1 Log4jInit.java           ptagis-1.0/src/util                      == /usr/local/pitweb/ptagis-1.0/src
A 2007-11-27 11:06 PST rday 1.1 MyFilesUploadAction.java ptagis-1.0/src/util                      == /usr/local/pitweb/ptagis-1.0/src
A 2007-11-27 11:06 PST rday 1.1 MyFilesUploadForm.java   ptagis-1.0/src/util                      == /usr/local/pitweb/ptagis-1.0/src
A 2007-11-27 11:06 PST rday 1.1 NotificationAction.java  ptagis-1.0/src/util                      == /usr/local/pitweb/ptagis-1.0/src
A 2007-11-27 11:06 PST rday 1.1 RegTagScanner.java       ptagis-1.0/src/util                      == /usr/local/pitweb/ptagis-1.0/src
M 2007-11-27 11:06 PST rday 1.2 ReportHelper.java        ptagis-1.0/src/util                      == /usr/local/pitweb/ptagis-1.0/src
M 2007-11-27 11:30 PST rday 1.7 nav.jsp                  ptagis-1.0/web/ptagis                    == /usr/local/pitweb/ptagis-1.0/web/ptagis
M 2007-11-27 11:32 PST rday 1.5 ptagis.properties        ptagis-1.0/web/ptagis/WEB-INF            == /usr/local/pitweb/ptagis-1.0/web/ptagis/WEB-INF
R 2007-11-27 11:59 PST rday 1.2 SendMyMessage.java       ptagis-1.0/src/util                      == /usr/local/pitweb/ptagis-1.0/src/util
M 2007-11-27 15:57 PST rday 1.8 nav.jsp                  ptagis-1.0/web/ptagis                    == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 15:57 PST rday 1.4 sr.properties            ptagis-1.0/web/sree/WEB-INF              == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 15:57 PST rday 1.3 web.xml                  ptagis-1.0/web/sree/WEB-INF              == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 15:57 PST rday 1.3 audit.log                ptagis-1.0/web/sree/WEB-INF/classes      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 15:57 PST rday 1.3 repository.bak           ptagis-1.0/web/sree/WEB-INF/classes      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 15:57 PST rday 1.3 repository.xml           ptagis-1.0/web/sree/WEB-INF/classes      == /usr/local/pitweb/ptagis-1.0/web
M 2007-11-27 15:57 PST rday 1.4 sree.log                 ptagis-1.0/web/sree/WEB-INF/classes      == /usr/local/pitweb/ptagis-1.0/web

Changes to *.properties files and web.xml are particularly dangerous.

• web/ptagis/WEB-INF/web.xml was not changed recently
• web/sree/WEB-INF/web.xml was corrupted by pitblade paths, but was corrected so that it has the following new stanza:

bash-3.00# cvs diff -r1.1 web.xml
Index: web.xml
===================================================================
RCS file: /home/ptagdev/CVScanary/ptagis-1.0/web/sree/WEB-INF/web.xml,v
retrieving revision 1.1
retrieving revision 1.3
diff -r1.1 -r1.3
26a27,31
>     <param-name>property.path.log4j</param-name>
>     <param-value>/usr/local/pitweb/ptagis-1.0/web/sree/WEB-INF/log4j.properties</param-value>
>   </context-param>
> 
>   <context-param>

• web/sree/WEB-INF/sr.properties was corrupted with pitblade paths, but corrected so that it includes just these new lines:

bash-3.00# cvs diff -r1.2 sr.properties
Index: sr.properties
===================================================================
RCS file: /home/ptagdev/CVScanary/ptagis-1.0/web/sree/WEB-INF/sr.properties,v
retrieving revision 1.2
retrieving revision 1.4
diff -r1.2 -r1.4
10a11
> 
18a20,27
> ### properties for the scheduler that runs/logs/reports on the queries
> property.file.log4j=log4j.properties
> property.file.db=db.properties
> #arguments: 1:reportname, 2:savename, 3:url_for_saved_report
> scheduler.mymessage.sender=Query Notification
> scheduler.mymessage.title=Query finished: {1}
> scheduler.mymessage.body=Your report {1} has been executed. Please visit the My Reports area to see it. Alternatively, the following is a direct link to <a href="{3}" target=\"_blank\">{2}</a>.
> 

• web/ptagis/WEB-INF/ptagis.properties was corrupted with pitblade paths, but corrected so that it contains these additions:

bash-3.00# cvs diff -r 1.3 ptagis.properties 
Index: ptagis.properties
===================================================================
RCS file: /home/ptagdev/CVScanary/ptagis-1.0/web/ptagis/WEB-INF/ptagis.properties,v
retrieving revision 1.3
retrieving revision 1.5
diff -r1.3 -r1.5
6a7,9
> ### MyMessages properties
> mymessages.page.description=The following are your PTAGIS messages.
> 
11,13d13
< services.tagproj.list.instructions=Please select one of your tag projects to see its details!
< services.tagproj.sponsorship.status=The list of tag coordinators have been updated.
< 
18a19,21
> services.tagproj.list.instructions=Please select one of your tag projects to see its details!
> services.tagproj.sponsorship.status=The list of tag coordinators have been updated.
> 
43c46
< services.testtags.upload.dir=/usr/local/pitweb/tmp
---
> services.testtags.upload.dir=/tmp

That should be corrected to return the testtags.upload.dir to /usr/local/pitweb/tmp DONE.

• web/ptagis/WEB-INF/log4j.properties was corrupted with pitblade paths and still needs to be corrected:

bash-3.00# cvs diff -r1.1 log4j.properties 
Index: log4j.properties
===================================================================
RCS file: /home/ptagdev/CVScanary/ptagis-1.0/web/ptagis/WEB-INF/log4j.properties,v
retrieving revision 1.1
retrieving revision 1.2
diff -r1.1 -r1.2
4a5
> log4j.logger.mymessages=TRACE, A4
15c16
< log4j.appender.A2.File=/usr/local/pitweb/ptagis-1.0/logs/TestLogging.log
---
> log4j.appender.A2.File=/dsk2/ptagis-1.0/logs/TestLogging.log
22,23c23,24
< #log4j.appender.A2.Threshold=INFO
< log4j.appender.A3.File=/usr/local/pitweb/ptagis-1.0/logs/myfiles.log
---
> #log4j.appender.A3.Threshold=INFO
> log4j.appender.A3.File=/dsk2/ptagis-1.0/logs/myfiles.log
27a29,37
> 
> # Appender A4 writes to a file
> log4j.appender.A4=org.apache.log4j.FileAppender
> #log4j.appender.A4.Threshold=INFO
> log4j.appender.A4.File=/dsk2/ptagis-1.0/logs/mymessages.log
> log4j.appender.A4.Append=true
> log4j.appender.A4.layout=org.apache.log4j.PatternLayout
> log4j.appender.A4.layout.ConversionPattern=%d %5p %c{2} (%F:%L) - %m%n
> 

known bug with CVS and jars

Long ago, we had a bug with CVS and the SREE jars where they would be corrupted by the act of checking them into the repository. That was resolved in production by putting the jars on the CVS ignore list. It is definitely worth checking the integrity of the jars on bay and jack.

Doing a side-by-side diff of the sree/WEB-INF/lib directories on bay and reedi shows no differences in the jars.

Email from Marton

I have done a little more debugging on bay, but did not have too much success yet. I will be able to spend more time on this on Friday instead of just an hour here and there. FYI, here is what I have tried

• I looked at the classes weblogic actually loads (enabled verbose java logging), but that did not help. Now verbose loggin is off again.
• I have decided to take out the offending line from scheduleHelper.jsp (temporarily) to see if the rest of the process works. Now running a report produces the two tasks, but the tasks still do not send the mymessage, nor write to the log4j. I left the changes in for now, but will probably restore scheduleHelper.jsp, because this workaround did not fix the underlying problem...
• Edited web/sree/WEB-INF/log4j.properties , which had the wrong path from pitblade and fixed it (you have done the same for the file under /ptagis, but not under /sree)

Please restart weblogic on jack. I had to stop it to test the classloading, but could not restart it because I am not a sudoer there. Perhaps you can make me one, so I can do this myself in the future.

Another thing related to a section in wiki: trying http://bay.psmfc.org:7003/sree/sr will not work properly, because the url you are using in the browser must match the url specified in sree.properties. Otherwise sree won't log you in properly (as shown by the error messages in appserver_*.log). Not being logged on is then the reason why you don't see your the queries.

Ryan's response

You should be able to use the "SU" wrapper script to become root. I haven't tried "sudo". It may not be enabled on bay and jack. But user nagym is in the list of admins that have access to root on jack.

• I restarted weblogic on jack
• It looks like the URL I used to access SREE directly is the one listed in sree.properties:

bash-3.00# grep bay sree.properties 
replet.repository.servlet=http\://bay.psmfc.org\:7003/sree/sr
replet.repository.url=http\://bay.psmfc.org\:7003/sree/sr
cluster.adm.url=http\://bay.psmfc.org\:7003/enterprisemanager/EnterpriseManager
bash-3.00# pwd
/usr/local/pitweb/ptagis-1.0/web/sree/WEB-INF/classes

• I was just able to access SREE directly through the url http://bay.psmfc.org:7003/sree/sr. It showed me the public queries when I had not authenticated and my personal queries when I was authenticated. Did you update the sree.properties to match the URL I was using?
• Furthermore, I was able to run a query successfully on bay just now (Thursday 10:29 PST). Oops, it wasn't successful. But it did pop up the scheduler window to say it was running. And it did show up with start and stop times in My Queries, but it shows a status of Failed.
• I tried the standard response to a failed query: I opened the query for editing and saved it without making any changes. Then I ran the same query again and it completed successfully. It created a message in My Messages, and I was able to download the resulting report.
• I couldn't find the scheduler.log. It doesn't appear that log is being generated.
• The first time each of my queries is run, it fails. But a message appears in My Messages saying that the query completed. The message includes a link to a non-existent report.
• However, if I edit and save each query, it appears to then work.
• I ran a query that returned over 65,000 rows and the My Messages trigger was correctly sent at the completion of the report:

[INFO] Nov 29, 2007 11:08:52 AM: rday_qb_YINN_2A;Task next run status:Pending;scheduled at:Thu Nov 29 11:09:01 PST 2007
[INFO] Nov 29, 2007 11:08:52 AM: rday_qb_YINN_2B;Task next run status:Wait for trigger;New task, Completion Condition pending, no self schedule
[INFO] Nov 29, 2007 11:09:01 AM: rday_qb_YINN_2A;Task next run status:Ready;Ready for execution
[INFO] Nov 29, 2007 11:09:01 AM: rday_qb_YINN_2A;Task last run status:Running;start at:Thu Nov 29 11:09:01 PST 2007;Execution Started
[INFO] Nov 29, 2007 11:09:01 AM: rday_qb_YINN_2A;Task next run status:Ignored;Condition requires no reschedule
[INFO] Nov 29, 2007 11:09:02 AM: Start query:$(@sql)
[INFO] Nov 29, 2007 11:09:02 AM: Task {Finish query:SELECT tag_id AS "Tag Id",tag_file_id AS "Tag File Id",tag_date AS "Tag Date",tag_site AS "Tag Site",release_date AS "Release Date",release_site AS "Release Site",migr_year AS "Migration Year",organization AS "Organization",coord_id AS "Coordinator Id",huc_code AS "HUC Code",river_km AS "River KM",species AS "Species",run AS "Run",rear_type AS "Rear Type",obs_site AS "Obs Site",first_obs_date AS "First Observation Date",last_obs_date AS "Last Observation Date",length AS "Length",weight AS "Weight",flags AS "Flags",tag_comments AS "Tag Comments",obs_count AS "Obs Count",first_obs_file AS "First Obs File",last_obs_file AS "Last Obs File",first_coil AS "First Coil",last_coil AS "Last Coil",first_monitor_name AS "First Monitor Name",last_monitor_name AS "Last Monitor Name",travel_time AS "Travel Time" FROM wqb_obs_site_detail WHERE (organization = 'YINN') AND (first_obs_date >= DATE('01/01/2007 00:00:00 PST'))} completed in 201 ms
[INFO] Nov 29, 2007 11:22:19 AM: Commit rday/YINN_2.csv with 1.1 to rday/YINN_2.csv.zip
[INFO] Nov 29, 2007 11:22:19 AM: Destroy report: rday_qb/YINN_2.11963633420911@local total 0
[INFO] Nov 29, 2007 11:22:19 AM: rday_qb_YINN_2A;Task last run status:Finished;start at:Thu Nov 29 11:09:01 PST 2007;end at:Thu Nov 29 11:22:19 PST 2007;Task Completed
[INFO] Nov 29, 2007 11:22:19 AM: rday_qb_YINN_2B;Task next run status:Pending;scheduled at:Thu Nov 29 11:22:19 PST 2007;triggered by rday_qb_YINN_2A
[INFO] Nov 29, 2007 11:22:19 AM: rday_qb_YINN_2B;Task next run status:Ready;Ready for execution
[INFO] Nov 29, 2007 11:22:19 AM: rday_qb_YINN_2B;Task last run status:Running;start at:Thu Nov 29 11:22:19 PST 2007;Execution Started
[INFO] Nov 29, 2007 11:22:19 AM: rday_qb_YINN_2B;Task next run status:Wait for trigger;Completion Condition pending, no self reschedule
[INFO] Nov 29, 2007 11:22:19 AM: rday_qb_YINN_2B;Task last run status:Finished;start at:Thu Nov 29 11:22:19 PST 2007;end at:Thu Nov 29 11:22:19 PST 2007;Task Completed

What is a task?

I'm confused about what is getting converted in the convertTask.jsp. I thought that a query was a task. But you verified that I have no tasks to be converted on bay.

I *do* have a lot of queries on bay that were copied over from reedi. And it seems that there is some conversion necessary for them to run on bay (I have to edit and resave them). I suppose this could be due to a problem in the process of copying them from reedi.

Answer

I guess a task in anything that is in schedule.xml. The problem I'm seeing is that I have queries defined in repository.xml and so they show up as visible in the My Queries page, but when I run them they fail because they are not defined in schedule.xml. I'm not sure how this happens, but it is a pretty regular occurance, so we've had to advise users to generally try re-saving a query if it fails to run (http://php.ptagis.org/wiki/index.php/FAQ#Why_does_my_query_fail_every_time_I_run_it.2C_even_though_it_used_to_work.3F).

The real question is if there is some way to protect users from this problem by preventing it or automatically recovering from it.

replicating this problem

• open a browser to http://bay.psmfc.org/ptagis
• navigate to My Queries
• open a second browser on the same machine to http://pitblade.psfmc.org/ptagis
• navigate to My Queries
• run a known good query on bay
• verify that you can get the output
• edit that query on pitblade and re-save it

it isn't necessary to make any changes, just increment the version number of the query

• return the browser on bay and attempt to run the same query again

At this point, the pitblade session data will have leaked to the browser pointed at bay. So when you try to run the query, it will try to run the version number that was generated on pitblade. It won't find this version in repository.xml and generate an error. Further attempts to run this query will fail until you edit and re-save it on bay.

Although this procedure uses two browsers to trigger the problem, based on the frequency that this is reported I believe it can be caused with a single browser in some border conditions. Perhaps partially expired sessions will cause this.

description of the steps above

The following is the explanation of why the above steps produce the issue. Note however that this does not describe why we would see the issue during normal use. Nevertheless this explains the design behind the queries.

• PTAGIS queries have two main components:

1. the definition (data source, fields, filter criteria, etc.) is stored in the database 2. the report template (the report's formatting and the sql query to execute) is stored by StyleReport (in sree/WEB-INF/classes/templates/ and as an index record in repository.xml)

• When queries are created both of these components are created, when they are deleted both of these components are deleted, and when they are modified the definition is modified and the report template is deleted and recreated.

• The problem with the steps above is that the two servers point to the same database, but the underlying file system and StyleReport instance is different. In particular
• creating a report on bay causes the database record and the StyleReport report templates on bay to be created; the database record is associated with this report template: e.g. myquery v.1 -> nagym_myquery_1
• modifying the query on pitblade causes the database record to be updated and the StyleReport report templates on pitblade to be updated (deleted/recreated); the database record is now associated with the report template on pitblade: e.g. myquery v.2 -> nagym_myquery_2
• running the query on bay now causes the problem, because the query points to a report template that does not exist on bay's file system: nagym_myquery_2 was created on pitblade's file system, bay only has the old nagym_myquery_1 which is now a 'zombie' (no queries point to it)

Finally, note that these problems happen because we are using two systems whose StyleReport instances/file systems are different, but point to the same database. This is not the situation in the real life cases. Even if a clustered system is in use, both of their machines point to the same shared file system for StyleReport and the same database. --Nagym 07:47, 17 December 2007 (PST)

Still some problems with queries

I just tried to execute a query on bay that had worked after the My Messages deployment. It ran to completion and I got a success message, but the link in the message took me to a SREE error page like this:

Internal error:

rday/ICH_March_2006_7.csv Error in retrieving the specified archive report. Please check if the report has been deleted.


Back

And if I look at the status on the My Queries page, it says the query failed. My hunch is that if I re-save the query it will work.

Rday 16:37, 4 December 2007 (PST)

I found that the query ICH_March_2006_7 does not exist in repository.xml on bay. It does have entries in schedule.xml and schedule-activity.xml. That query doesn't exist in sree/WEB-INF/classes/templates either. Why is there a version mismatch? repository.xml and classes/templates only have version 6 of that query.

After I did the edit/re-save on the query:

• ICH_March_2006_8 is present in:

classes/templates

repository.xml

After I ran the query, it completed successfully and the message had a good link to the output. Also the query was present in:

repository.xml

schedule.xml

schedule-activity.xml

classes/templates

I'm not sure what is causing the query to be dropped from repository.xml and templates.

scheduling queries fails

Pressing the "schedule" button on the My Queries page causes a java stack dump like this:

Compilation of '/usr/local/bea81/user_projects/domains/mydomain/./appserver_bay/.wlnotdelete/extract/appserver_bay_sree_sree/jsp_servlet/__schedule.java' failed: /usr/local/bea81/user_projects/domains/mydomain/./appserver_bay/.wlnotdelete/extract/appserver_bay_sree_sree/jsp_servlet/__schedule.java:412: cannot resolve symbol
probably occurred due to an error in /schedule.jsp line 29:
int nSaveFormat= Builder.GENERATED_REPORT;

/usr/local/bea81/user_projects/domains/mydomain/./appserver_bay/.wlnotdelete/extract/appserver_bay_sree_sree/jsp_servlet/__schedule.java:574: cannot resolve symbol
probably occurred due to an error in /schedule.jsp line 341:
<option value="<%=Builder.GENERATED_REPORT%>"<%if (nSaveFormat == Builder.GENERATED_REPORT) out.print(" SELECTED");%>>Standard Report

/usr/local/bea81/user_projects/domains/mydomain/./appserver_bay/.wlnotdelete/extract/appserver_bay_sree_sree/jsp_servlet/__schedule.java:576: cannot resolve symbol
probably occurred due to an error in /schedule.jsp line 341:
<option value="<%=Builder.GENERATED_REPORT%>"<%if (nSaveFormat == Builder.GENERATED_REPORT) out.print(" SELECTED");%>>Standard Report

Full compiler error(s):

/usr/local/bea81/user_projects/domains/mydomain/./appserver_bay/.wlnotdelete/extract/appserver_bay_sree_sree/jsp_servlet/__schedule.java:412: cannot resolve symbol
symbol  : variable GENERATED_REPORT 
location: class inetsoft.report.io.Builder
        int nSaveFormat= Builder.GENERATED_REPORT; //[ /schedule.jsp; Line: 29]
                                ^
/usr/local/bea81/user_projects/domains/mydomain/./appserver_bay/.wlnotdelete/extract/appserver_bay_sree_sree/jsp_servlet/__schedule.java:574: cannot resolve symbol
symbol  : variable GENERATED_REPORT 
location: class inetsoft.report.io.Builder
        out.print(String.valueOf(Builder.GENERATED_REPORT));  //[ /schedule.jsp; Line: 341]
                                        ^
/usr/local/bea81/user_projects/domains/mydomain/./appserver_bay/.wlnotdelete/extract/appserver_bay_sree_sree/jsp_servlet/__schedule.java:576: cannot resolve symbol
symbol  : variable GENERATED_REPORT 
location: class inetsoft.report.io.Builder
        if (nSaveFormat == Builder.GENERATED_REPORT) out.print(" SELECTED"); //[ /schedule.jsp; Line: 341]
                                  ^
3 errors

Wed Dec 05 13:08:03 PST 2007

classpath on bay

Here is the classpath used by weblogic on bay as reported by the startManagedWebLogic.sh script

CLASSPATH=/usr/local/bea81/jdk142_11/lib/tools.jar:
/usr/local/bea81/weblogic81/server/lib/weblogic_sp.jar:
/usr/local/bea81/weblogic81/server/lib/weblogic.jar::
/usr/local/bea81/weblogic81/common/eval/pointbase/lib/pbserver44.jar:
/usr/local/bea81/weblogic81/common/eval/pointbase/lib/pbclient44.jar:
/usr/local/bea81/jdk142_11/jre/lib/rt.jar:
/usr/local/bea81/weblogic81/server/lib/webservices.jar:
/usr/local/pitweb/ptagis-1.0/web/ptagis/WEB-INF/lib/log4j-1.2.14.jar:
/usr/local/pitweb/ptagis-1.0/web/ptagis/WEB-INF/classes:
/usr/local/pitweb/ptagis-1.0/web/ptagis/WEB-INF/lib/commons-httpclient.jar:
/usr/local/pitweb/ptagis-1.0/web/ptagis/WEB-INF/lib/commons-logging.jar:
/usr/local/pitweb/ptagis-1.0/web/ptagis/WEB-INF/lib/struts.jar:
/usr/local/pitweb/ptagis-1.0/web/sree/WEB-INF/classes:
/usr/local/pitweb/ptagis-1.0/web/sree/WEB-INF/classes/ptagis:
/usr/local/StyleReportEE/lib/sree_pro.jar:
/usr/local/StyleReportEE/lib/etools.jar:
/usr/local/StyleReportEE/lib/design_pro.jar:
/usr/ingres/ing26/ingres/lib/edbc.jar:

classpath on pitblade

Here is the classpath on pitblade as reported by the startWebLogic.sh script

CLASSPATH=/dsk2/local/bea81/jdk141_02/lib/tools.jar:
/dsk2/local/bea81/weblogic81/server/lib/weblogic_sp.jar:
/dsk2/local/bea81/weblogic81/server/lib/weblogic.jar:
/dsk2/local/bea81/weblogic81/server/lib/ojdbc14.jar::
/dsk2/local/bea81/weblogic81/common/eval/pointbase/lib/pbserver44.jar:
/dsk2/local/bea81/weblogic81/common/eval/pointbase/lib/pbclient44.jar:
/dsk2/local/bea81/jdk141_02/jre/lib/rt.jar:
/dsk2/local/bea81/weblogic81/server/lib/webservices.jar:
/dsk2/ptagis-1.0/web/ptagis/WEB-INF/lib/log4j-1.2.14.jar:
/dsk2/ptagis-1.0/web/ptagis/WEB-INF/classes:
/dsk2/ptagis-1.0/web/ptagis/WEB-INF/lib/commons-httpclient.jar:
/dsk2/ptagis-1.0/web/ptagis/WEB-INF/lib/commons-logging.jar:
/dsk2/ptagis-1.0/web/sree/WEB-INF/classes:
/dsk2/ptagis-1.0/web/sree/WEB-INF/classes/ptagis:
/dsk2/ptagis-1.0/web/sree/WEB-INF/lib/sree_pro.jar:
/dsk2/ptagis-1.0/web/sree/WEB-INF/lib/etools.jar:
/dsk2/ptagis-1.0/web/sree/WEB-INF/lib/design_pro.jar:
/dsk1/AdvIngres/ing26/ingres/lib/edbc.jar: 

diff reports etools.jar is different

pitblade:/dsk2/ptagis-1.0/web/sree/WEB-INF/lib/etools.jar is not the same as bay:/usr/local/StyleReportEE/lib/etools.jar

design_pro.jar and sree_pro.jar are the same.

-r--r--r--+ 1 root root 12149866 Jan 23  2007 /net/bay/usr/local/StyleReportEE/lib/etools.jar
lrwxrwxrwx  1 root bin        23 Feb 24  2006 /net/pitblade/dsk2/ptagis-1.0/web/sree/WEB-INF/lib/etools.jar -> etools.jar.v70.20060114
-rw-r--r-- 1 root bin 20457121 Apr 22  2005 /net/pitblade/dsk2/ptagis-1.0/web/sree/WEB-INF/lib/etools.jar.v70.20060114

However, the etools.jar on reedi matches the one on bay:

[rday@snapper ~]$ diff /net/reedi/global/ds1/pitweb/ptagis-1.0/web/sree/WEB-INF/lib/etools.jar /net/bay/usr/local/StyleReportEE/lib/etools.jar

The classpath on reedi points to /usr/local/StyleReportEE/lib/etools.jar which is a symlink that points to /global/ds1/pitweb/ptagis-1.0/web/sree/WEB-INF/lib/etools.jar

So which version of etools.jar is more recent, the pitblade version or the reedi/bay version?

the files in the reedi etools.jar all have a date of 11-11-2004 10:35

the files in the pitblade etools.jar generally have a date of 4-6-2005 13:23

classpath on reedi

CLASSPATH=/usr/local/bea81/weblogic81/server/lib/CR105198_810.jar:
/usr/local/bea81/jdk141_02/lib/tools.jar:
/usr/local/bea81/weblogic81/server/lib/weblogic_sp.jar:
/usr/local/bea81/weblogic81/server/lib/weblogic.jar:
/usr/local/bea81/weblogic81/server/lib/ojdbc14.jar::
/usr/local/bea81/weblogic81/common/eval/pointbase/lib/pbserver44.jar:
/usr/local/bea81/weblogic81/common/eval/pointbase/lib/pbclient44.jar:
/usr/local/bea81/jdk141_02/jre/lib/rt.jar:
/usr/local/bea81/weblogic81/server/lib/webservices.jar:
/global/ds1/pitweb/ptagis-1.0/web/ptagis/WEB-INF/classes:
/global/ds1/pitweb/ptagis-1.0/web/ptagis/WEB-INF/lib/commons-httpclient.jar:
/global/ds1/pitweb/ptagis-1.0/web/ptagis/WEB-INF/lib/commons-logging.jar:
/global/ds1/pitweb/ptagis-1.0/web/ptagis/WEB-INF/lib/struts.jar:
/global/ds1/pitweb/ptagis-1.0/web/sree/WEB-INF/classes:
/global/ds1/pitweb/ptagis-1.0/web/sree/WEB-INF/classes/ptagis:
/usr/local/StyleReportEE/lib/sree_pro.jar:
/usr/local/StyleReportEE/lib/etools.jar:
/usr/local/StyleReportEE/lib/design_pro.jar:
/usr/ingres/ing26/ingres/lib/edbc.jar:
/usr/ingres/ing26/ingres/lib/edbc.jar:

Next steps

• Ryan will look through repository.xml and schedule.xml for discrepancies for production users
• Marton will put in workaround for scheduler stack trace
• Marton will look at code for version numbers of queries

Recent "replet not found" errors in production

bash-2.05# grep "Replet not found" schedule.log
[ERROR] Nov 27, 2007 11:49:47 AM: Replet not found: cameronap_qb/qAcoustic_2
inetsoft.sree.RepletException: Replet not found: cameronap_qb/qAcoustic_2
[INFO] Nov 27, 2007 11:49:50 AM: cameronap_qb_qAcoustic_2;Task last run status:Failed;start at:Tue Nov 27 11:49:47 PST 2007;end at:Tue Nov 27 11:49:50 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: cameronap_qb/qAcoustic_2
[ERROR] Nov 27, 2007 12:07:07 PM: Replet not found: jfisher_qb/SalmonObsB2JTWX_4
inetsoft.sree.RepletException: Replet not found: jfisher_qb/SalmonObsB2JTWX_4
[INFO] Nov 27, 2007 12:07:12 PM: jfisher_qb_SalmonObsB2JTWX_4;Task last run status:Failed;start at:Tue Nov 27 12:07:07 PST 2007;end at:Tue Nov 27 12:07:12 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: jfisher_qb/SalmonObsB2JTWX_4
[ERROR] Nov 27, 2007 1:30:29 PM: Replet not found: lockhart_qb/96tag_1
inetsoft.sree.RepletException: Replet not found: lockhart_qb/96tag_1
[INFO] Nov 27, 2007 1:30:34 PM: lockhart_qb_96tag_1;Task last run status:Failed;start at:Tue Nov 27 13:30:28 PST 2007;end at:Tue Nov 27 13:30:34 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: lockhart_qb/96tag_1
[ERROR] Nov 27, 2007 2:55:37 PM: Replet not found: kinzer_qb/by00type1_3
inetsoft.sree.RepletException: Replet not found: kinzer_qb/by00type1_3
[INFO] Nov 27, 2007 2:55:39 PM: kinzer_qb_by00type1_3;Task last run status:Failed;start at:Tue Nov 27 14:55:37 PST 2007;end at:Tue Nov 27 14:55:39 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: kinzer_qb/by00type1_3
[ERROR] Nov 28, 2007 9:15:35 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at B1J B2A B2J BCC BO1_3
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at B1J B2A B2J BCC BO1_3
[INFO] Nov 28, 2007 9:15:45 AM: Gene_Ploskey_qb_2007 PIT DET at B1J B2A B2J BCC BO1_3;Task last run status:Failed;start at:Wed Nov 28 09:15:34 PST 2007;end at:Wed Nov 28 09:15:45 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at B1J B2A B2J BCC BO1_3
[ERROR] Nov 28, 2007 9:15:47 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
[INFO] Nov 28, 2007 9:15:53 AM: Gene_Ploskey_qb_2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3;Task last run status:Failed;start at:Wed Nov 28 09:15:47 PST 2007;end at:Wed Nov 28 09:15:53 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
[ERROR] Nov 28, 2007 9:15:56 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
[INFO] Nov 28, 2007 9:16:02 AM: Gene_Ploskey_qb_2007 PIT DET at BVX_2;Task last run status:Failed;start at:Wed Nov 28 09:15:56 PST 2007;end at:Wed Nov 28 09:16:02 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
[ERROR] Nov 28, 2007 9:20:28 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at B1J B2A B2J BCC BO1_3
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at B1J B2A B2J BCC BO1_3
[INFO] Nov 28, 2007 9:20:32 AM: Gene_Ploskey_qb_2007 PIT DET at B1J B2A B2J BCC BO1_3;Task last run status:Failed;start at:Wed Nov 28 09:20:28 PST 2007;end at:Wed Nov 28 09:20:32 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at B1J B2A B2J BCC BO1_3
[ERROR] Nov 28, 2007 9:30:37 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at B1J B2A B2J BCC BO1_3
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at B1J B2A B2J BCC BO1_3
[INFO] Nov 28, 2007 9:30:37 AM: Gene_Ploskey_qb_2007 PIT DET at B1J B2A B2J BCC BO1_3;Task last run status:Failed;start at:Wed Nov 28 09:30:26 PST 2007;end at:Wed Nov 28 09:30:37 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at B1J B2A B2J BCC BO1_3
[ERROR] Nov 28, 2007 9:36:21 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
[INFO] Nov 28, 2007 9:36:26 AM: Gene_Ploskey_qb_2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3;Task last run status:Failed;start at:Wed Nov 28 09:36:21 PST 2007;end at:Wed Nov 28 09:36:26 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
[ERROR] Nov 28, 2007 9:37:04 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
[INFO] Nov 28, 2007 9:37:08 AM: Gene_Ploskey_qb_2007 PIT DET at BVX_2;Task last run status:Failed;start at:Wed Nov 28 09:37:04 PST 2007;end at:Wed Nov 28 09:37:08 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
[ERROR] Nov 28, 2007 9:40:24 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
[INFO] Nov 28, 2007 9:40:27 AM: Gene_Ploskey_qb_2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3;Task last run status:Failed;start at:Wed Nov 28 09:40:24 PST 2007;end at:Wed Nov 28 09:40:27 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
[ERROR] Nov 28, 2007 9:40:49 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
[INFO] Nov 28, 2007 9:40:52 AM: Gene_Ploskey_qb_2007 PIT DET at BVX_2;Task last run status:Failed;start at:Wed Nov 28 09:40:49 PST 2007;end at:Wed Nov 28 09:40:52 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
[ERROR] Nov 28, 2007 9:57:31 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
[INFO] Nov 28, 2007 9:57:36 AM: Gene_Ploskey_qb_2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3;Task last run status:Failed;start at:Wed Nov 28 09:57:31 PST 2007;end at:Wed Nov 28 09:57:36 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
[ERROR] Nov 28, 2007 10:02:07 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
[INFO] Nov 28, 2007 10:02:11 AM: Gene_Ploskey_qb_2007 PIT DET at BVX_2;Task last run status:Failed;start at:Wed Nov 28 10:02:07 PST 2007;end at:Wed Nov 28 10:02:11 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BVX_2
[ERROR] Nov 28, 2007 10:09:20 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
[INFO] Nov 28, 2007 10:09:24 AM: Gene_Ploskey_qb_2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3;Task last run status:Failed;start at:Wed Nov 28 10:09:20 PST 2007;end at:Wed Nov 28 10:09:24 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BO2 BO3 BO4 BVJ BWL_3
[ERROR] Nov 28, 2007 10:18:52 AM: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BCC BO2 BO3 BO4 BVJ_4
inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BCC BO2 BO3 BO4 BVJ_4
[INFO] Nov 28, 2007 10:18:56 AM: Gene_Ploskey_qb_2007 PIT DET at BCC BO2 BO3 BO4 BVJ_4;Task last run status:Failed;start at:Wed Nov 28 10:18:52 PST 2007;end at:Wed Nov 28 10:18:56 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: Gene_Ploskey_qb/2007 PIT DET at BCC BO2 BO3 BO4 BVJ_4
[ERROR] Nov 29, 2007 8:30:21 AM: Replet not found: carriegass_qb/bt cy05_2
inetsoft.sree.RepletException: Replet not found: carriegass_qb/bt cy05_2
[INFO] Nov 29, 2007 8:30:25 AM: carriegass_qb_bt cy05_2;Task last run status:Failed;start at:Thu Nov 29 08:30:21 PST 2007;end at:Thu Nov 29 08:30:25 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: carriegass_qb/bt cy05_2
[ERROR] Nov 29, 2007 1:07:07 PM: Replet not found: lockhart_qb/YPRdet_5
inetsoft.sree.RepletException: Replet not found: lockhart_qb/YPRdet_5
[INFO] Nov 29, 2007 1:07:07 PM: lockhart_qb_YPRdet_5;Task last run status:Failed;start at:Thu Nov 29 13:07:01 PST 2007;end at:Thu Nov 29 13:07:07 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: lockhart_qb/YPRdet_5
[ERROR] Nov 30, 2007 9:22:50 AM: Replet not found: steveb_qb/LOOHall_1
inetsoft.sree.RepletException: Replet not found: steveb_qb/LOOHall_1
[INFO] Nov 30, 2007 9:22:50 AM: steveb_qb_LOOHall_1;Task last run status:Failed;start at:Fri Nov 30 09:22:45 PST 2007;end at:Fri Nov 30 09:22:50 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: steveb_qb/LOOHall_1
[ERROR] Nov 30, 2007 9:24:16 AM: Replet not found: steveb_qb/LOOHall_1
inetsoft.sree.RepletException: Replet not found: steveb_qb/LOOHall_1
[INFO] Nov 30, 2007 9:24:20 AM: steveb_qb_LOOHall_1;Task last run status:Failed;start at:Fri Nov 30 09:24:16 PST 2007;end at:Fri Nov 30 09:24:20 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: steveb_qb/LOOHall_1
[ERROR] Nov 30, 2007 3:13:25 PM: Replet not found: lockhart_qb/by02smolt_6
inetsoft.sree.RepletException: Replet not found: lockhart_qb/by02smolt_6
[INFO] Nov 30, 2007 3:13:31 PM: lockhart_qb_by02smolt_6;Task last run status:Failed;start at:Fri Nov 30 15:13:25 PST 2007;end at:Fri Nov 30 15:13:31 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: lockhart_qb/by02smolt_6
[ERROR] Nov 30, 2007 11:07:55 PM: Replet not found: rday_qb/ICH March 2006_6
inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_6
[INFO] Nov 30, 2007 11:08:00 PM: rday_qb_ICH March 2006_6;Task last run status:Failed;start at:Fri Nov 30 23:07:55 PST 2007;end at:Fri Nov 30 23:08:00 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_6
[ERROR] Dec 3, 2007 9:02:48 AM: Replet not found: lockhart_qb/by97smolt_8
inetsoft.sree.RepletException: Replet not found: lockhart_qb/by97smolt_8
[INFO] Dec 3, 2007 9:02:53 AM: lockhart_qb_by97smolt_8;Task last run status:Failed;start at:Mon Dec 03 09:02:47 PST 2007;end at:Mon Dec 03 09:02:53 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: lockhart_qb/by97smolt_8
[ERROR] Dec 3, 2007 9:10:29 AM: Replet not found: lockhart_qb/by97smolt_8
inetsoft.sree.RepletException: Replet not found: lockhart_qb/by97smolt_8
[INFO] Dec 3, 2007 9:10:34 AM: lockhart_qb_by97smolt_8;Task last run status:Failed;start at:Mon Dec 03 09:10:29 PST 2007;end at:Mon Dec 03 09:10:34 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: lockhart_qb/by97smolt_8
[ERROR] Dec 3, 2007 3:08:24 PM: Replet not found: TEANAWAYSTEEL_qb/TEAN STEEL_8
inetsoft.sree.RepletException: Replet not found: TEANAWAYSTEEL_qb/TEAN STEEL_8
[INFO] Dec 3, 2007 3:08:27 PM: TEANAWAYSTEEL_qb_TEAN STEEL_8;Task last run status:Failed;start at:Mon Dec 03 15:08:24 PST 2007;end at:Mon Dec 03 15:08:27 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: TEANAWAYSTEEL_qb/TEAN STEEL_8
[ERROR] Dec 4, 2007 8:51:47 AM: Replet not found: rweldert_qb/ugrallhat2007_1
inetsoft.sree.RepletException: Replet not found: rweldert_qb/ugrallhat2007_1
[INFO] Dec 4, 2007 8:51:52 AM: rweldert_qb_ugrallhat2007_1;Task last run status:Failed;start at:Tue Dec 04 08:51:47 PST 2007;end at:Tue Dec 04 08:51:52 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: rweldert_qb/ugrallhat2007_1
[ERROR] Dec 4, 2007 10:41:10 AM: Replet not found: lyle_qb/bi-op06_2
inetsoft.sree.RepletException: Replet not found: lyle_qb/bi-op06_2
[INFO] Dec 4, 2007 10:41:16 AM: lyle_qb_bi-op06_2;Task last run status:Failed;start at:Tue Dec 04 10:41:10 PST 2007;end at:Tue Dec 04 10:41:16 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: lyle_qb/bi-op06_2
[ERROR] Dec 4, 2007 10:45:37 AM: Replet not found: lyle_qb/bi-op06_2
inetsoft.sree.RepletException: Replet not found: lyle_qb/bi-op06_2
[INFO] Dec 4, 2007 10:45:41 AM: lyle_qb_bi-op06_2;Task last run status:Failed;start at:Tue Dec 04 10:45:37 PST 2007;end at:Tue Dec 04 10:45:41 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: lyle_qb/bi-op06_2
[ERROR] Dec 5, 2007 10:44:30 AM: Replet not found: plasterk_qb/2006 tag recap_1
inetsoft.sree.RepletException: Replet not found: plasterk_qb/2006 tag recap_1
[INFO] Dec 5, 2007 10:44:35 AM: plasterk_qb_2006 tag recap_1;Task last run status:Failed;start at:Wed Dec 05 10:44:30 PST 2007;end at:Wed Dec 05 10:44:35 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: plasterk_qb/2006 tag recap_1
[ERROR] Dec 5, 2007 10:51:44 AM: Replet not found: plasterk_qb/2006 tag recap_1
inetsoft.sree.RepletException: Replet not found: plasterk_qb/2006 tag recap_1
[INFO] Dec 5, 2007 10:51:44 AM: plasterk_qb_2006 tag recap_1;Task last run status:Failed;start at:Wed Dec 05 10:51:37 PST 2007;end at:Wed Dec 05 10:51:44 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: plasterk_qb/2006 tag recap_1
[ERROR] Dec 7, 2007 6:15:01 AM: Replet not found: rday_qb/ICH March 2006_8
inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_8
[INFO] Dec 7, 2007 6:15:07 AM: rday_qb_ICH March 2006_8;Task last run status:Failed;start at:Fri Dec 07 06:15:00 PST 2007;end at:Fri Dec 07 06:15:07 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_8
[ERROR] Dec 7, 2007 10:07:25 AM: Replet not found: rday_qb/ICH March 2006_8
inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_8
[INFO] Dec 7, 2007 10:07:30 AM: rday_qb_ICH March 2006_8;Task last run status:Failed;start at:Fri Dec 07 10:07:25 PST 2007;end at:Fri Dec 07 10:07:30 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_8
[ERROR] Dec 7, 2007 10:17:54 AM: Replet not found: rday_qb/ICH March 2006_9
inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_9
[INFO] Dec 7, 2007 10:17:58 AM: rday_qb_ICH March 2006_9;Task last run status:Failed;start at:Fri Dec 07 10:17:54 PST 2007;end at:Fri Dec 07 10:17:58 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_9
[ERROR] Dec 7, 2007 12:50:27 PM: Replet not found: rday_qb/ICH March 2006_11
inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_11
[INFO] Dec 7, 2007 12:50:31 PM: rday_qb_ICH March 2006_11;Task last run status:Failed;start at:Fri Dec 07 12:50:27 PST 2007;end at:Fri Dec 07 12:50:31 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_11
[ERROR] Dec 7, 2007 12:52:33 PM: Replet not found: rday_qb/ICH March 2006_11
inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_11
[INFO] Dec 7, 2007 12:52:37 PM: rday_qb_ICH March 2006_11;Task last run status:Failed;start at:Fri Dec 07 12:52:33 PST 2007;end at:Fri Dec 07 12:52:37 PST 2007;Execution failed: inetsoft.sree.RepletException: Replet not found: rday_qb/ICH March 2006_11
bash-2.05# 

For the errors above, I began to verify that the query in question does not exist in repository.xml, but does exist in sree/WEB-INF/classes/templates.

-rw-r--r--   1 root     other       9689 Nov 26 13:12 cameronap_qAcoustic_2.srt
-rw-r--r--   1 root     other      10008 Nov 27 12:00 jfisher_SalmonObsB2JTWX_4.srt
lockhart is a different problem.  He has a space at the end of his user name.