Resource Management and Accounting Notebook - page 84 of 150

First PagePrevious PageNext PageLast PageTable of ContentsSearch

Date and Author(s)

QBank Deployment Guide

QBank Deployment Guide

Dynamic versus Delayed Accounting

Delayed Accounting:

In the absence of a dynamic system, some sites enforce allocations by periodically (weekly or nightly) parsing resource manager job logs and then applying debits against the appropriate user accounts. Although QBank can easily support this type of system by the use of the qwithdraw command in post-processing scripts, this approach will allow a user or project to use resources significantly beyond their designated allocation and generally suffers from stale accounting information.

Dynamic Accounting:

QBank's design allows it to interact dynamically with your resource management system. Withdrawals for resource utilization can be made immediately when the job finishes (or even incrementally throughout the job). Additionally, reservations can be issued at the start of a job to place a hold against the user's account, thereby ensuring that a job will only start if it has sufficient reserves to complete. The remainder of this document will describe the interactions for dynamic accounting.

Methods of interacting with QBank

There are essentially five ways of interacting with QBank. Let's consider a simple withdrawal in each of the different ways.

  1. Configure an application which already has hooks for QBank
  2. The easiest way to use QBank is to use a resource management system with built-in support for QBank. For example, the Maui Scheduler can be configured to directly interact with QBank to perform the withdrawals, reservations, etc by setting the appropriate parameters in the Maui config file.

  3. Using the appropriate command-line client
  4. From inside a script, or by invoking a system command, you can use a command line client (one of the "q" commands in qbank's bin directory).
    To issue a withdrawal at the completion of a job, you would use qwithdraw:
    /usr/local/qbank/bin/qwithdraw -a mscfops -u scottmo -M MPP1 -p 8 -w 3600 -Q 3 -j PBS.1234

  5. Invoking the subroutine directly from the 'qbank' client
  6. If you need options only available through the subroutine interface or prefer the expressivity of key-value pairs, you can use qbank.
    To issue a withdrawal you must invoke the make_withdrawal subroutine:
    /usr/local/qbank/bin/qbank make_withdrawal ACCOUNT=mscfops USER=scottmo MACHINE=MPP1 PROCS=8 WCTIME=3600 QOS=3 JOBID=PBS.1234

  7. Use the C++ API
  8. If your resource manager is written in C or C++ and you have access to the source code, you can link in and access the QBankAPI. See APItest.c in the samples directory for a simple example. Note, however, that this interface has not been updated recently and has received very little testing.
    To make a withdrawal via this interface you might do something like:
      request=qb_setrequest(qhandle,"withdrawal", "make"); 
      qb_addconstraints(request,"ACCOUNT", "mscfops"); 
      qb_addconstraints(request,"USER", "scottmo"); 
      qb_addconstraints(request,"MACHINE", "MPP1"); 
      qb_addconstraints(request,"PROCS", "8"); 
      qb_addconstraints(request,"WCTIME", "3600"); 
      qb_addconstraints(request,"QOS", "3"); 
      qb_addconstraints(request,"JOBID", "PBS.1234"); 
      if ((response=qb_getresponse(qhandle))==NULL) 

  9. Communicate via the QBank Wire Protocol
  10. Finally, your program can directly use the protocol for communicating over the wire to the QBank server. This will entail building the request command, appending an HMAC digest to authenticate the requestor, sending the request along with the bytecount to the server and parsing the response. Although the protocol is not documented, it is a simple protocol and easily learned by looking at the source code. The Maui Scheduler communicates with QBank via this method.
    The command sent via the protocol to the QBank server might look something like:
    00000166 COMMAND=make_withdrawal ACCOUNT=mscfops JOBID=PBS.1234 MACHINE=MPP1 QOS=3 PROCS=8 USER=scottmo WCTIME=3600 AUTH=root CHECKSUM=d6dd9e1c500f92e6b0bd8d801f4c598fcb4833d2

Interaction Points

There are three potential interaction points during the lifetime of a job in which QBank might interact with your resource managment system.

  1. Job submission time [optional -- recommended]
  2. It is useful to check a user's balance at the time of submission to ensure the user's account has enough cycles to run the job. A warning can be issued if funds are low or the job might be rejected with an informative message in the case of insufficient funds or any other problems with the account. Otherwise, the job might wait in the queue for days only to fail at start time where it could be potentially harder to route the message back to the user.

    When jobs are submitted to a meta-scheduler or resource broker, it may be useful to obtain a job quote from the various resource providers so that the user may make an intelligent choice about resource tradeoffs, turnaround time and how much it is going to "cost" him.

    This step is most aptly performed by using the quotation feature of QBank. A quote request will return the amount that the job would be charged if it runs for its full time allotment along with a unique quote id which can be used to secure that rate at the reservation and withdrawal steps. It will fail with a qualifying message if the user does not have sufficient funds.

    In addition to verifying sufficient funds, this step is useful to enforce that the user specify an account (project to charge to) and a wallclock limit. Other policy screens can be employed at this time as well (processor-second ceiling for the job, etc.).

    To interface with QBank at this phase requires that

  3. Job start time [optional -- highly recommended]
  4. At the start of a job QBank creates a hold on the users account (reservation) based on the resources requested and the wallclock limit. This step does not have to be performed but it is highly recommended because this mechanism prevents jobs from using more credits than have been allocated.

    This step makes use of the reservation feature of QBank. If the reservation succeeds, it will return a message indicating the amount reserved for the job. In the case of insufficient resources to run the job or some other problem with the reservation, the command will fail with an informative message. Depending on site policy, this may or may not prevent the job from starting.

    In order to make a QBank reservation at the start of a job the, the resource management system must:

  5. Job completion [required]
  6. When a job completes the final withdrawal is made (which also removes the reservation). Ideally, this step will occur immediately after the job completes for dynamic accounting. This has the added benefit that job run times can often be reconstructed from QBank reservation and withdrawal timestamps in case the resource management accounting data becomes corrupt.

    This step makes use of the withdrawal feature of QBank. If the withdrawal succeeds, it will return a message indicating the amount charged for the job. In the case of insufficient resources to run the job or some other problem with the reservation, the command will fail with an informative message. Depending on site policy, the job may or may not be charged if the job did not complete "successfully".

    This step requires that the resource management system: