Load Patrons

From Staffwiki

Jump to: navigation, search

Contents

[edit] Loading Patron Records

[edit] Reference Documents

  1. Batch Loading Patron Data into I-Share Voyager Databases.
  2. Patron Record Standard Interface File (Annotated)

[edit] Introduction

The "patron load" is a recurring process which must take

expiration of records
expiration of patrons
change of patron status
avoidance of duplication

into account.

[edit] To extract data from Banner® and run the loading script

[edit] Using the library UNIX system
  1. Connect via SSH to the library server
    1. To display the current system location use "uname -a"
    2. To display the login id, type "whoami"
  2. Change Directory (cd) to the /home/webs/cgi-bin/test/voyager
  3. List (ls -l) the directory and find the PatronLoad.sh file
  4. Change variable "datadate" to the appropriate date, usually the date of extraction. The data subdirectory will have another subdirectory named "datadate" into which the input files should be placed from the Banner extraction and transfer from prod.csu.edu
    1. The data subdirectory at /home/webs/cgi-bin/test/voyager/data can be created before the use of sftp (see below) by using the mkdir yyyymmdd command while in /home/webs/cgi-bin/test/voyager/data
  5. Review
    1. Review variables S (students), E (employees), C (data from CBORD), and P (patron output). Make sure the file names from the extraction match the files in the "datadate" subdirectory used in PatronLoad.sh
    2. Review the various dates. They can override the current calculated values used in the Python script.
    3. NB: The prepExtraction.py program creates a comma/tab delimited file as input to makeFullRecord.py
  6. Data files for input and output will be left in /home/webs/cgi-bin/test/voyager/data/"datadate"
  7. Run the shell by typing "./PatronLoad.sh"
  8. Review the .err and .log files
  9. Examine the data for correctness and formatting issues
[edit] The Banner extraction process

You must have a Banner account appropriate to this task. Discuss the requirements with Helen Jackson of ITD.

  1. UNIX access username and password
  2. SQL access username and password. The sql_username may be the same as the UNIX login username.

User supplied data is indicated in bold.

  1. Log in to the UNIX system
    1. Via a Windows Secure Shell Application, choose the proper profile or use Quick Connect in the windows ssh interface.
    2. Via a UNIX or Linux system, type “ssh <username>@prod.csu.edu”. You will be prompted for the UNIX access <password>
  2. Log in to Banner

NB: Check with the ITD programmer for the correct scripts to use. Likely you will have been working with her/him during the process. The following reflects the initial setup via Thomas Ware of ITD.

    1. Type PROD
    2. Optionally use ls to list files
    3. sqlplus <sql_username>
    4. Login: <username>
    5. Password: <password>
  1. Extract the data
    1. start enrolled_students.sql
    2. Enter value for file name: <student_output_file_name>
    3. Enter value for term: yyyymm (term codes are YYYY01, YYYY05, or YYYY09 only)
    4. Wait until the extraction and creation of the SIF file is completed. This may produce an echo of the file to the terminal with many data lines displayed.
    5. ! Type a single exclamation point and press Enter (or optionally retrieve the employee data now also)
  2. Retrieve the student data
    1. Use a Secure Shell® program on Windows® or Linux to move data with privacy. The support staff can install this program for you. You will use this to transmit files over a secure stream of data so that third parties cannot retrieve the information. The file will be placed at your current local attach point.
    2. Example if on Linux or UNIX
      1. sftp <username>@prod.csu.edu
      2. Supply <password> when prompted
      3. get <student_output_file_name>
      4. exit
  3. Log in to the UNIX system as before
    1. Via a Windows Secure Shell Application, choose the proper profile.
    2. Via a UNIX or Linux system, type “ssh <username>@prod.csu.edu”. You will be prompted for the UNIX access <password>
  4. Log in to Banner
    1. Type PROD
    2. sqlplus <sql_username>
    3. Login: <username>
    4. Password: <password>
    5. Optionally use ls to list files
  5. Extract the non-student employee data
    1. start employees_master.sql
    2. Enter value for file name: <employee_output_file_name>
    3. Wait until the extraction and creation of the SIF file is completed
    4. ! Type a single exclamation point and press Enter
  6. Retrieve the non-student data. These are records for staff and faculty employees of the university
    1. sftp <username>@prod.csu.edu
    2. Supply <password> when prompted
    3. get <employee_output_file_name>
    4. exit

[edit] Cleanup and Post Process the Records

The current September 15, 2008 extraction scripts produce an output file with errors, at least for the student records.

The first 1 to 3 lines will need to be removed because they are not SIF records and should not be in the file.

The extraction also produces records with Social Security numbers. These few records must be removed or tolerated. The post process script reports the errors with record numbers and fields in error.

  1. Post-process the records
    1. Adjust patron codes as needed. As of August 27, 2008, the PR, DR, and CP codes must be changed on each record to GR. Use vi or EMACS (preferable) to do this
  2. Re-examine the file for proper structure
    1. Remove leading "garbage" record(s)
    2. Check the alignment of fields
      1. emacs <filename>
      2. M-x line-number-mode
      3. M-x column-number-mode
      4. Examine several records at the beginning, middle, and end of the file by placing the cursor on each field and making certain that the offset matches the EMACS offset as indicated in the SIF layout document on the right margin.
    3. It is useful to sort the file based on patron identification
      1. sort --key=21,45 <filename> > <filename.sorted>
    4. Remove duplicated records within the file
      1. uniq < <filename.sorted> > <filename.uniq>
    5. Use diff to compare the two (optional)
      1. diff <filename.uniq> <filename.sorted>
    6. Delete records with SSN data rather than UID data
      1. The sort will have placed SSNs in distinct places within the file and clustered according to the SSN value. This will be at the beginning of the file in most cases.
  3. Re-examine file <filename.uniq> for field-level issues using EMACS in column-number mode and checking http://libstaff.csu.edu/lists/web/howto/PRSIF.pdf
    1. Clear SSN's as needed
    2. Translate or re-map Patron Group Codes
      1. Use EMACS M-% to locate (without quotes) " DR ", " PR ", and " CP " and alter them to " GR ". Use ! to complete the replacements without reprompting.
  4. Transmit the file to CARLI, noting the file name for entry in the WRO.
      1. sftp csuftp@reports.carli.illinois.edu
      2. Enter <password> when prompted
      3. put <filename.csu.students.yyyymmdd.uniq>
      4. exit
  5. Generate a Patron Load Work Request Order (WRO)at the site http://auth.carli.illinois.edu/wro/cgi/wro.cgi This is done according to CARLI expectations
  6. Examine the WRO diagnostics on files placed on reports.carli.illinois.edu. Retrieve the files by using the GET command in sftp. The filenames involved will have been sent to the email addresses associated with the Work Request Order (WRO).
    1. sftp csuftp@reports.carli.illinois.edu
      1. Enter <password> when prompted
    2. get <filename>
    3. exit
  7. Repair the file as needed by repeating the above
  8. Verify WRO submission for full records by replying to the WRO. You may allow it to elapse or you may choose to withdraw it, depending upon an analysis of the information in the diagnostics and log files. CARLI supplies documentation of the procedures at a link embedded within the WRO report pages.
  9. Notify appropriate personnel of further processing needed
Personal tools