Process Management and Monitoring Notebook - page 11 of 74

EditDeleteAnnotateNotarize
First PagePrevious PageNext PageLast PageTable of ContentsSearch

Date and Author(s)

Minutes of 2001.09.20 conf call

Scalable Systems Software - Process Management Working Group 
Teleconference 9/20/01 10:00 AM (PDT)  
------------------------------------ 
 
Attendees 
--------- 
Paul Hargrove       LBNL 
Jason Duell         LBNL 
Eric Roman          LBNL 
Rusty Lusk          ANL 
Narayan Desai       ANL 
Al Geist            ORNL 
Stephen Scott       ORNL 
Scott Jackson       PNNL 
 
Brett Bode, Mike Welcome, and Katie Vargo were unable to attend. 
 
Agenda 
------ 
  *  Reading material for XML 
  *  XML Schema for process startup request 
     See pages 9 and 10 of PMWG e-notebook 
  *  System monitoring 
  *  Set time/date for next meeting 
  *  System wide issues (common socket behavior and interoperability) 
 
Action Items 
------------ 
  *  Read description of "global concerns" on XML delimiting 
     (posted on build/configure website) 
  *  Post links to XML reading material on project notebook.   [All] 
  *  Look at how other systems (XMLRPC, SOAP, Slide) handle passing 
     parsing multiple XML request through a single socket      [All] 
  *  Place 1 sentence reviews of known XML reading material  
     on PMWG/XML page (page 7 of PMWG notebook).               [Rusty] 
  *  Type up a more detailed, thoroughly commented description of 
     process manager requests.                                 [Narayan] 
  *  Discuss conflicts with OSCAR system model                 [Stephen] 
 
XML Reading Material 
-------------------- 
Rusty's group is learning XML in detail.  They believe that DTD's are 
now obsolete, and that schema's are more popular.   They are reading 
_Essential XML_.  Rusty's group is trying to find a validating XML 
parser that can validate against a schema.  Most current validating 
parsers validate against a DTD, not a schema. 
 
Paul asked whether any of the group had looked at SOAP or at XMLRPC. 
None of the group has had a chance to look at these protocols in detail. 
Narayan Desai believes that SOAP and XMLRPC standards may not be useful 
to us at this point, and that the standards may not be frozen.  Al Geist 
suggested looking at Slide. 
 
XML Schema for Process Startup Request 
-------------------------------------- 
Paul asked whether the standard input, standard output, or 
standard error of a job could be specified in the process startup request. 
Paul asked whether anyone could be notified when a job completes. 
Rusty had not looked at this issue. 
 
Rusty said that one missing attribute from the process startup request 
is the current working directory to use when the process starts. 
 
Rusty is looking for a way to say: "Process 8 writes it output to the 
file stdout.8"  Paul suggested a mechanism similar to printf.  Rusty agreed 
to look at this in the next version of the request.  Rusty stated that 
he would like to learn whether XML has a way to turn string arguments into 
array arguments. 
 
Narayan suggested the group try to take advantage of "well-formed" values 
for schemas.  He suggested using well-formed schemas to perform input 
validation for every software component.  If it parses, then it's valid. 
Elements of a request that were not meaningful at a particular site would 
not even parse.  Al Geist asked whether site administrators should have 
to configure the request format for every part of the system software. 
Narayan says that this could be automated as part of the system build 
process. 
 
There was a brief discussion about host ranges.  How complicated should 
host ranges be?  How do you specify ranges? 
 
Jason asked whether there was a way to specify environment variables 
globally ( across a range of nodes).  Rusty said that there were no 
complicated rules to handle range overlap, so that the request processing 
could be kept very concise. 
 
Stephen suggested adding a field to specify a particular cluster within the 
process startup request.  Rusty stated that there would be only one process 
manager per cluster.  Stephen contested that a process manager should be 
able to forward process startup requests to another process manager.  Paul 
stated that a request to start a process should only be sent to the process 
manager responsible that will start the process.  If the user does not 
know in advance where a process should be started, the decision should be 
left to the scheduler.  Stephen agreed that this decision would be more 
appropriate for the scheduler. 
 
Al Geist asked whether there was a distinction between a required and  
an unrequired field of an XML request.  Narayan stated that by default, all 
fields were optional; "required" is necessary to make a tag required.  
"optional" may be used to make an element optional, and "prohibited" may 
be used to forbid a particular element. 
 
Paul asked how to refer to the process started by a process startup 
request?  Rusty has not thought about this. 
 
Al stated that this was part of a more general problem.  Every Scalable 
Systems component will expect some response.  What should this 
response be?  Paul suggested a tuple containing an opaque response and a 
location identifier (opaque, locationId).  The location identifier would 
specify who to contact to operate on the job.  There was some discussion. 
This format might only be appropriate for instantiation or query requests. 
It would not be appropriate for a "stop job" request to return a tuple 
of this form, but only a status code specifying whether the request 
was successful. 
 
Al proposed that schema's be developed for both responses, and 
for requests.  We want some consistency in the response schemas. 
Two responses that would be appropriate for all components are an "I don't 
understand your request" response, and a "No, I won't do that, 
but the following component can" response. 
 
Monitoring 
---------- 
Paul opened the monitoring discussion by asking which requests or objects 
do we need to monitor?  Paul also asked about how monitoring works through 
this system in general. 
 
Is there a way to "stream" XML information?  Rusty stated that XML has a way 
to stream XML information through a special model.  For requests, he suggested 
only using a document model. 
 
Scott suggested using a subscription model for monitoring. 
The subscription model would bring the system closer to an event model. 
 
Paul summarized the discussion by identifying 3 event models.  First, a 
streaming model for continuous or interactive monitoring.  Second, a request/ 
response model.  Third, a subscription model for event processing. 
 
Next meeting 
------------ 
Rusty is unable to attend the October 4th meeting due to a schedule 
conflict. 
 
The next meeting is scheduled for September 27th. 
 
System Wide Issues 
------------------ 
Narayan opened this discussion by asking whether multiple requests 
could be sent down a socket to another component.  Can sockets be held 
open?  This raised the issue of how to delimit XML requests. 
 
Jason suggested encapsulating requests inside another format, possibly 
HTTP.  If the length of a request is sent before the request, there is 
no need to delimit the end with a special marker. 
 
How are different types of request sent through a stream of requests? 
Is a higher level protocol or convention to encapsulate requests in a 
stream needed? 
 
There may be issues with lingering sockets.  The group agreed that we 
should start simple, by assuming a single request per socket.  All the 
working groups will use this convention. 
 
The group discussed issues that arise when sockets close unexpectedly. 
They agreed that the parser needs to be smart about what happens in 
this case. 
 
 
 


Addendum:   Al Geist   Date: Wed Sep 26 21:22:08 2001 (GMT)
There are some comments on using HTTP format between components in the main notebook. (Addendums of "General Ideas" page 8)

-Al