OPERATION AND MAINTENANCE PLAN

Copyright 2000

Revised 2004 without permission


A Word on the copyright of this document iv

1.  Scope. 5

1.1    Operation and Maintenance Plan Identification 5

1.2    Objectives 5

1.3    Internet Website Overview. 5

1.3.1   Application Software. 6

1.3.2   Host Machines 6

1.3.3   Network. 6

1.3.4   Internet Gateway. 6

1.3.5   Communication Channels 6

1.4    Document Overview. 6

2.  Definitions 7

3.  System Design. 7

3.1    Overview. 7

3.1.1   Network Description. 8

3.1.2   Capacity and Volume. 9

3.2    Business Offices: Locations, Telephone Numbers, Primary Contacts 9

3.3    Co-location Site. 10

3.3.1   Hardware. 10

3.3.2   Software. 15

3.4    Business Site. 23

3.4.1   Telephone System. 24

3.4.2   Hardware. 24

3.4.3   Software. 28

3.5    Outsource Site. 31

3.5.1   Hardware. 31

3.5.2   Software. 31

4.  Policies 31

4.1    Responsibilities and Organization 31

4.1.1   Responsibilities 31

4.1.2   Required Commitment and Support of Management 31

4.1.3   Organization Staffing Criteria and Guidelines 32

4.2    Philosophy 33

4.2.1   Overview. 33

4.2.2   Adherence to <defunct>-Established Policies and Procedures 33

5.  Procedures 33

5.1    Interfacing with Other Groups and Organizations 33

5.1.1   Software Development Interface. 33

5.1.2   SCM Interface. 34

5.1.3   Support Staff Interface (Documentation/System Administration/Training) 34

5.1.4   Project Management Interface. 35

5.2    Fault Management 35

5.2.1   Surveillance and Control 35

5.2.2   Alarm Handling. 35

5.2.3   Trouble Detection. 36

5.2.4   Trouble Correction. 36

5.2.5   Test and Acceptance. 37

5.2.6   Network Recovery. 37

5.2.7   Work Order Handling. 37

5.2.8   Escalation Procedures 37

5.3    Configuration Management 37

5.3.1   System Turn-up. 37

5.3.2   Network Provisioning. 37

5.3.3   Auto-discovery. 52

5.3.4   Backup and Restore. 52

5.3.5   Database Handling. 57

5.3.6   Availability Of Spare Parts 61

5.3.7   Corrective Maintenance. 61

5.3.8   Preventative Maintenance. 62

5.3.9   Change Requests 62

5.3.10 Change Control 62

5.4    Accounting Management 62

5.4.1   Track Service Usage. 62

5.4.2   Monitor Service Quality Trends 62

5.4.3   Supply of Statistical Information as a Basis for Expansion. 62

5.5    Performance Management 62

5.5.1   Data Collection. 62

5.5.2   Report Generation. 62

5.5.3   Data Analysis 62

5.6    Security Management 62

5.6.1   Control Network Element Access 63

5.6.2   Enable Network Element Functions 63

5.6.3   Access Logs 63

6.  References 63

Appendix A - Inventory, Parts, Identifications 64

Appendix B – RAID Implementation. 69

Appendix C – Network Diagrams 76

Appendix D – Steps to Rebuild the Accrue Oracle Database Schema. 77

Appendix E – Production Oracle Auto Start and Shutdown. 80

Appendix F – Firewall Erata – Redirect www traffic. 81

 

 

List of Figures and Tables

Figure 1 – Business Sites 8

Figure 2 – Network and Phone Topology. 8

Figure 3 – Failure Restoration Flow Chart 37

Figure 4 – SCSI Connections between 250, 450, and A1000. 39

 

A Word on the copyright of this document

 

All of the material in the original document was created in 2000 for a now defunct dot.com.

 

This original document was written by the team of Terry Mohn, Sri N…, John Edwards, and David Russell.  Two managers, Doug Hegebarth and Clifton Mclellan, both pushed for and reviewed each of three releases of this document.

 

All proprietary references such as names, phone number, logos, icons, serial numbers and IP addresses have been removed where it makes sense.  Phone numbers for customer support of various vendors have been left.  Some fictitious information has been added for internet publication.  You will encounter frequent <snip> references; but the format will allow you to understand the intent of the paragraph or entries.

 

I offer my apologies to Sri for not knowing the spelling of his last name.  To be corrected…

 

Some people might relate a host name in a script to the project where this document was created.  Others might compare the date to my resume.  If you have a burning desire to know who this document was created for you will have to ask yourself this question “Are you equipped to know”?

 

David Russell


1.                    Scope

1.1                 Operation and Maintenance Plan Identification

The Operation and Maintenance (O&M) Plan provides <defunct>.com with the policies and procedures for the managing company equipment and systems.  The control, maintenance and execution of activities documented herein are under the responsibility of <defunct>’s Operations Group (OG).  Identification of responsibilities and sequences of major activities related to the managing company assets are established by this document.

1.2                 Objectives

Through policy declaration, this document specifies the commitment to managing O&M processes for all <defunct> projects.   OG personnel shall execute the procedures defined in this document to maintain consistent and approved actions.

1.3                 Internet Website Overview

The <defunct>.com system is designed to be an Internet Trading HUB.  This Trading HUB will provide a forum for buyers and sellers of metal working machine tools to trade used equipment, order third party ancillary support, and learn about the Industry in general.

The components that make up the <defunct>.com system will be designed for:

Ø      High information throughput, rapid network response,

Ø      Secure transactions,

Ø      Ease of use and functionality using a browser based interface

Ø      Integration with third party software and communication interfaces,

Ø      A high degree of flexibility in administration of configuration of the system.

The website application is predominantly comprised using Commercial off the Shelf Software (COTS) developed by and Independent Software Vendor (ISV).  Object oriented middleware (both custom and commercial) and fast network response times will be instituted.  Scalability, in the form of component TCP/IP (WWW) services installed into a UNIX server environment, will be practiced.

The website is the primary mode of commerce and revenue.  It is co-located at a remote site with connections to our business office.  The co-location site hosts our hardware, network, firewalls, and application software.  The following areas shall fall under O&M management.

1.3.1           Application Software

The auction application, website monitor, Enterprise Resource Planning (ERP) software and Customer Relationship Management (CRM) software shall be managed. 

1.3.2           Host Machines

Hardware servers are comprised of Sun, Dell and Compaq servers.  These are networked to the Internet to form our website. We shall manage backup and recovery systems within this procedure.

1.3.3           Network 

The Ethernet LAN, routers, switches and hubs comprising the physical connection between host machines, firewalls and private networks shall be managed.  We shall assess our security measures against attack, monitor authorization and authentication, and manage failure recovery.

1.3.4           Internet Gateway

This is comprised of our Internet Service Provider’s (ISP) domain service, access to tier-one Internet and both availability and performance of our services to the North American Internet users.

1.3.5           Communication Channels

Our communications channels consist of help and FAQ pages, customer self-service pages, email, chat and telephone communications between our website, our outsourced customer service site (operated by service.com) and our business site.  We shall monitor, control and manage each channel.

1.4                 Document Overview

This document will define management policies and procedures relevant to operating all company assets.  This information will be broken into the following areas.

Ø      System Design – hardware and software components

Ø      Policies – responsibilities, commitment to perform and organizational philosophy

Ø      Fault management – surveillance and control, trouble report handling, work order handling and escalation procedures

Ø      Configuration management – change requests and configuration control

Ø      Performance management – performance measurement, monitor service quality trends, supply of statistical information as a basis for expansion

Ø      Spare parts maintenance – availability of spare parts

Ø      Field maintenance – corrective and preventative maintenance

Ø      Diagrams – network details, tables and report templates

2.                    Definitions

The following definitions apply to this document:

TBD

 

The following acronyms apply to this document:

 

Acronym

Meaning

CPU

Central Processing Unit

CRM

Customer Relationship Management

CSC

Computer Software Component

CSU

Computer Software Unit

ERP

Enterprise Resource Planning

HW

Hardware

I/F

Interface

ISP

Internet Service Provider

OG

Operations Group

O&M

Operation and Management

OMP

Operation and Management Plan

RMA

Reliability/Maintainability/Availability

SW

Software

3.                    System Design

3.1                 Overview

Figure 1 – Business Sites

Our business computing machinery are grouped into three networks: a co-location center housing the production equipment, which is located at a network operations center (NOC) hosted by SimpleNet, network equipment supporting business activities at our business site and communication equipment supporting outsourced customer service activities provided by service.com.  <defunct>.com is company domain name.

3.1.1     Network Description

 

Figure 2 – Network and Phone Topology

Web servers and application servers are networked to the Internet through a firewall.  A private connection exists between the NOC and the business site, which allows website administration, access to the business accounting software and web site record keeping.   Another private connection exists between the NOC and service.com, which guarantees private communication supporting customer service activities.  The business site uses a separate firewall for its Internet access. 

All computing equipment is supported by battery-backup power supply to cover the contingency of power loss.  The network is designed to provide 100 Mbit/sec data transfer rates.  Appropriate routers and switches are strategically placed to provide high security and network bandwidth. 

3.1.2     Capacity and Volume

The operational website and associated application software provides sufficient data storage and processing capabilities to support the following “go-live” web traffic requirements.  Scale and performance during specification of hardware shall use a multiplier of 10 against each metric.

Ø      Auction transactions (where the purchase is consummated): 40 per day

Ø      Catalog transactions (where the purchase is consummated): 400 per day

Ø      Unique visits: 2,000 per day

Ø      Web hits: 10,000 per day

Ø      When anticipating number of bids per auction transaction, use a multiplier of 10 (i.e. 400 bids per day).   Use the 10-multiplier for catalog item selection/deselecting as well (i.e. 4000 shopping cart additions or removals per day).

Ø      Anticipate a skewed activity log where Monday is the highest activity day, followed by Tuesday, then Thursday, Wednesday and Friday. Anticipate almost no activity over Saturday or Sunday.

Ø      We estimate weekly traffic load as follows: 35% Monday, 25% Tuesday, 15% Thursday, 10% Wednesday, 7% Friday, 7% Saturday & Sunday.

Ø      The telephone support system provides sufficient capacity and processing capabilities to support the following caller traffic requirements.

Ø      Web 1-800 help handled by service.com: 100 per day

Ø      General 1-800 information handled by service.com: 200 per day

Ø      Specialized 1-800 help by our staff:  100 per day

Ø      General 1-858 business communication by our staff: 1000 per day

3.2           Business Offices: Locations, Telephone Numbers, Primary Contacts

Primary business activities are carried out at <defunct>.com business office with address:

Ø      <defunct>.com business site

Ø      <snip>

Ø      <defunct>.com production (Internet) hardware and programmed software housed at co-location center

Ø      <defunct>.com website at SimpleNet

Ø      <snip>

3.3           Co-location Site

The co-location site is used to house the hardware for our firewalls, web monitors, web servers, application servers, database servers and supporting networking hardware.   A NOC is used to house the co-location center’s critical production computing devices since it provides around-the-clock security, earthquake proofing, and special fire retardant systems. 

All computing equipment co-located at the NOC is powered on a battery-backup power supply to cover the contingency of power loss at the NOC.  The network connecting these computing systems provides 100 Mbit/sec data transfer rates.  Appropriate routers and switches are strategically placed to provide high security and network bandwidth.  Static IP’s are used at the co-locate site to enable the firewall to control the highest degree of security and minimize the processing requirements of the servers.

Web servers and application servers are networked to the Internet through a firewall.  A private connection between NOC and business site provides sufficient bandwidth between application servers to prevent degradation.  Static IP’s are assigned to computers to enable the firewall to control the highest degree of security.

See Appendix C for Co-Location Site Network Diagram.

3.3.1     Hardware

3.3.1.1                  Switch

The required switch configuration is the following:

·        Cisco 2912XL

3.3.1.2                  Routers

The required router configuration is the following:

·        Cisco 2610

·        CSU/DSU

3.3.1.3                       Firewall

The required firewall configuration is the following:

·        Sun Ultra 10 Web Server

·        333 MHz processor

·        1GB memory

·        10 GB of disk space (highly dependent on user requirements)

·        SVGA Video card

·        Monitor (80 MHz, 1024 x 768 screen resolution)

·        Mouse

·        Keyboard

·        UPS (Uninterrupted Power Supply)

·        Ethernet network adapter (10/100 NIC card)

3.3.1.4                        Web Page Monitor

The Web Monitor supports the following functionality for our product to operate efficiently:

·        Support 10,000 web hits per day without impact to other network/Internet resources.

·        There shall be enough free disk space to contain the UNIX operating system, database and the web monitor software.

·        A monitor shall attach to the server to enable system control and administration.

·        The required web server configuration is the following:

·        Sun Ultra 10 Web Server

·        333 MHz processor

·        1GB memory

·        10 GB of disk space (highly dependent on user requirements)

·        SVGA Video card

·        Monitor (80 MHz, 1024 x 768 screen resolution)

·        Mouse

·        Keyboard

·        UPS (Uninterrupted Power Supply)

·        Ethernet network adapter (10/100 NIC card)

3.3.1.5                       Web Page Server (phase 2)

The Web Server shall support the following functionality for our product to operate efficiently:

·        Support 10,000 web hits per day without performance degradation to the background network processes.

·        There shall be enough free disk space to contain the UNIX operating system and the web server software.

·        A monitor shall attach to the server to enable system control and administration.

·        The required web server configuration is the following:

·        Sun Ultra 10 Web Server

·        333 MHz processor

·        1GB memory

·        10 GB of disk space (highly dependent on user requirements)

·        SVGA Video card

·        Monitor (80 MHz, 1024 x 768 screen resolution)

·        Mouse

·        Keyboard

·        UPS (Uninterrupted Power Supply)

·        Ethernet network adapter (10/100 NIC card)

3.3.1.6                    Disk Array

The required mirrored disk array configuration is the following:

 

3.3.1.7                  Tape Backup

The required tape configuration is the following:

·        Sun DLT 1000

3.3.1.8                       Application Servers

Application, middleware and database server software will execute on this high performance hardware.  The server will communicate to the web server(s) over a network.  The Auction Server shall support the following functionality for our product to operate efficiently:

·        There shall be enough free disk space to contain the application server software.

·        Support JSP, Java servlets and JDBC without performance degradation to the background network processes.

·        Support for SSL, HTTP through firewall

·        There shall be enough free disk space to contain the UNIX operating system, the application database software, and Oracle Server (8.1.5). 

·        The GUI portion of administrative tools distributed with the software requires a screen resolution of 1024 x 768 pixels (large fonts) with a minimum color depth of 256 colors. An appropriate SVGA monitor shall be installed.

The required production server configuration is the following:

·        Sun Enterprise 450 application and database server

·        400 MHz processor

·        256 MB memory

·        100 GB of disk space (highly dependent on user requirements)

·        SVGA Video card

·        Monitor (80 MHz, 1024 x 768 screen resolution)

·        Mouse

·        Keyboard

·        UPS (Uninterrupted Power Supply)

·        Ethernet network adapter (10/100 NIC card)

3.3.1.9                       ERP Server

Financial accounting software will execute on this high performance hardware.  The server will communicate between applications and the business site.  The ERP Financial Accounting Server shall support the following functionality for our product to operate efficiently:

·        There shall be enough free disk space to contain the financial application software.

·        Support ODBC and JDBC without performance degradation to the background network processes.

·        There shall be enough free disk space to contain the UNIX operating system, the financial accounting software, Oracle Server (8.0.5), and the financial accounting database. 

·        The GUI portion of administrative tools distributed with the software requires a screen resolution of 1024 x 768 pixels (large fonts) with a minimum color depth of 256 colors. An appropriate SVGA monitor shall be installed.

The required Financials Database server configuration is the following:

·        Sun Enterprise 250 application and database server

·        300 MHz processor

·        1GB memory

·        100 GB of disk space (highly dependent on user requirements)

·        SVGA Video card

·        Monitor (80 MHz, 1024 x 768 screen resolution)

·        Mouse

·        Keyboard

·        UPS (Uninterrupted Power Supply)

·         Ethernet network adapter (10/100 NIC card)

3.3.1.10             CRM Application Server

Customer Relationship Management software will execute on this high performance hardware.  The server will communicate between service.com outsource company and the co-location site.  The CRM Server shall support the following functionality for our product to operate efficiently:

·        There shall be enough free disk space to contain the CRM application software.

·        Support ODBC without performance degradation to the background network processes.

·        There shall be enough free disk space to contain the Windows NT server operating system, the CRM software, MS SQLServer7.0), and the CRM database. 

·        The GUI portion of administrative tools distributed with the software requires a screen resolution of 1024 x 768 pixels (large fonts) with a minimum color depth of 256 colors. An appropriate SVGA monitor shall be installed.

The required CRM server configuration is the following:

·        Dell xj500 server

·        Dual 300 MHz processor

·        1GB memory

·        100 GB of disk space (highly dependent on user requirements)

·        SVGA Video card

·        Monitor (80 MHz, 1024 x 768 screen resolution)

·        Mouse

·        Keyboard

·        UPS (Uninterrupted Power Supply)

·        Ethernet network adapter (10/100 NIC card)

3.3.1.11                   CRM E-Mail Server

Customer Relationship Management email software will execute on this high performance hardware.  The server will communicate between service.com outsource company and the co-location site.  The CRM Server shall support the following functionality for our product to operate efficiently:

·        There shall be enough free disk space to contain the CRM email application software.

·        Support ODBC without performance degradation to the background network processes.

·        There shall be enough free disk space to contain the Windows NT server operating system, the CRM email software, MS SQL (5.0), and the email database. 

·        The GUI portion of administrative tools distributed with the software requires a screen resolution of 1024 x 768 pixels (large fonts) with a minimum color depth of 256 colors. An appropriate SVGA monitor shall be installed.

The required CRM email server configuration is the following:

·        Dell xj500 server

·        300 MHz processor

·        256 MB memory

·        20 GB of disk space (highly dependent on user requirements)

·        SVGA Video card

·        Monitor (80 MHz, 1024 x 768 screen resolution)

·        Mouse

·        Keyboard

·        UPS (Uninterrupted Power Supply)

·        Ethernet network adapter (10/100 NIC card)

3.3.2      Software

3.3.2.1                  Database

3.3.2.1.1              Overview

There are five Oracle database instances and one Microsoft SQL Server at the co-location site.  Three of the Oracle instances support the Oracle Financial Applications, run on the E-250, and use Oracle8 software.  One instance is used by Moai Live Exchange, which runs on the E-450, and uses Oracle8i software, and the fifth instance is used by the Accrue Insight collection & reporting tool, currently runs on the E-450, and uses Oracle8i software. The single Microsoft SQL Server instance is used by the Octane Customer Relationship Management (CRM) software, runs on a dedicated NT dual processor machine, and uses the version 7 Microsoft products.

3.3.2.1.2              Installation

<xxx Documentation on issues will be added here>

3.3.2.1.3              Patches/Releases

We are currently running 8.1.5.1 on Leai and Tatooine.  Skywalker has not had the current patch installed.  There is already a .2 patch available for special circumstances, and a .3 promised, as well as a new release, 8.1.6 (also referred to as 8i Second Release).

The 8.1.5.1 patch fixed several Recovery Manager and interMedia context indexes (sic) problems.

3.3.2.1.4              RAID

See Appendix B for description of RAID.

3.3.2.2                  Firewall

Multiple servers handle web page creation via the connection to the Internet through a firewall.  The firewall application is the only application running on the firewall servers.  We have chosen Checkpoint’s Firewall-1 as the application of choice.  It is configured to allow only HTTP and HTTPS protocols passage to our web server(s).  The firewall is also configured to announce the IP of our web server to the Internet. 

·        Operating System: Sun Solaris version 2.6

·        <snip>

3.3.2.3                       Web Page Monitor

Our network analysis software, Accrue Insight, uses the Oracle8i database software.  Two extra cost Oracle8i options are required for this installation: the partitioning option, and the parallel query option.  Accrue states the latter as being the "bitmapped index" option; however, there is no such option.  Bitmapped indices are part of the parallel query option.

The files from the Accrue "collector" are stored in naboo::/insight/var/log.  There are two files here every hour of every day.  The files for the current hour and the previous hour are not compressed.  Files prior to these are compressed in the .gz format.  While these files could be uncompressed and edited, it's more likely that the database would be cleared and the "transformer" would be used to filter unwanted records.  Insight operates in a batch mode.  Files are loaded into the database at 9:30 PM and 1:30 AM.  The 9:30 load is referred to as the "mini load", and the 1:30 load as the "nightly load".  The nightly load also does reports.  These loads are done from two entries in the "goldmine" user account crontab file.

<xxx insert crontab –l output here>

After the nightly load, the files are moved from naboo::/insight/var/log to naboo::/insight/var/archive.  The /insight/var mount point will be moved onto another file system.

<xxx insert name of file system here>

Accrue documentation recommends that you run Oracle in "noarchivelogmode".  This recommendation was not in the "quick start" sheet that was  read prior to the installation, so we were initially running in archive log mode; however, the reasons for not running in archive log mode where only protective, and Accrue agrees that the recommendation was based on "keeping it simple".   We monitor our systems six times a day to monitor the disk utilization.

Without archive log mode we cannot do "hot backups", implement Oracle Recovery Manager (R*Man), or do point in time recovery.  There is a performance hit for running in this mode however.  The instance will run faster in noarchivelogmode, and will consume less disk space.  It will have to be down periodically for cold backups (i.e., scheduled maintenance), or be captured in a logical backup (schema export).  Both a little more trouble than hot backups, or using the R*man/Veritas libraries to operate the tape drive directly.  In the event we have a database failure, we may want to just rebuild the schema and reload the data from the collector files -- depending on size and duration of those stored collector files.

The steps required to rebuild are documented in Appendix D: Steps to Rebuild the Accrue Oracle Database Schema.

Because all data resides in the log and archive directories, the archive log mode in the Accrue Oracle instance has been turned off.

3.3.2.4                  Web Page Server (phase 2)

·        Operating System: Sun Solaris version 2.6

·        <snip>

3.3.2.5                       Application Server 1

Our auction software, Moai Live Exchange, uses the Oracle8i database software.  While the interMedia component (context indexes) is not an extra cost option, it is a product, which was developed by another team of developers, and as a result, it is not as “robust” as some of the other Oracle components.  There are tricks to the installation and operation, which are described in the following sections (4.3.2.5.1 through 4.3.2.5.4).

The Application Server 1 configuration follows:

·        Operating System: Sun Solaris version 2.6

·        <snip>

 

3.3.2.5.1                Installation

When doing a fresh install, including Oracle8i libraries, into a new “ORACLE_HOME” structure, if you specify the location for the Oracle “datafiles” for all components, as we do, there will be no location for the data file associated with the “context option” (8i “interMedia”), and the installation will fail.  The dbassist tool is not smart enough to prompt to relocate the context data file, and not smart enough to create a directory if it doesn’t exist.

3.3.2.5.2              Start & Stop Operations

With the exception of the Accrue instance, all production Oracle databases are run in the archive log mode.  This is the safest way to assure point-in-time recovery.  What happens here is that every transaction is processed in an Oracle “redo log”, when the active online redo log is filled it is switched, a new log becomes current, and the previously filled log is archived.  With these archived logs, it is possible to restore “hot backups” to a point-in-time before a failure occurred.  Hot backups are taken while the system is available to the user, as compared to cold backups of operating system files, or logical backups (exports) taken in Oracle’s “exclusive” (single user) mode.  Archived redo logs cannot be applied to restores of cold backups, or imports of export dump files.

The Accrue instance does not run in archive log mode for reasons discussed in section 4.3.2.3 Web Page Monitor.

The sequence recommended by Oracle to shutdown the system is as follows:

 

% su root

# shutdown -y -g0 -i6

 

If the Unix “boot”, “reboot”, or “init 6” commands are used the database does not shutdown with the automatic shutdown script “dbshut”.  These commands are not reading the shutdown script in the “/etc/rc0.d” directory.

 

Issuing the “shutdown” command executes the scripts in the above directory.

 

For the Moai and Accrue Oracle instance’s automatic startup and shutdown configurations please see Appendix E: Production Oracle Auto Start and Shutdown.

There is one Unix “crontab” file on the production server (Tatooine).

The Oracle user’s crontab file contains the following entries:

# Compress the archive redo logs every 15 minutes

0,15,30,45 * * * * /u02/app/oraprd/dba/comparch PRD > /dev/null 2>&1

# Perform database checks for OraLogs - D.Russell 01/2000

45 03,07,11 * * 0-6 /u01/app/oracle/scripts/database_checks.sh >/dev/null 2>&1

45 15,19,23 * * 0-6 /u01/app/oracle/scripts/database_checks.sh >/dev/null 2>&1

# Analyze Moai Tables Daily - D.Russell 01/2000

00 04 * * 0-6 /u01/app/oracle/scripts/analyze_tables.sh >/dev/null 2>&1

# Perform Hot Backup to Disk Daily - D.Russell 01/2000

00 02 * * 0-6 /u01/app/oracle/scripts/hot_disk.sh >/dev/null 2>&1

The entries in this facility cause several database actions to occur at preferred intervals.  Obviously, activities such as tape backups of file systems should occur after the completion of the Oracle hot backups.

There is one Unix “crontab” file on the production Web Page Monitor (Naboo).  This file is located in the “Goldmine” user’s account and contains the following information:

4 * * * * /insight/bin/gziplogs.pl /insight/var/logs >/dev/null 2>&1

30 1 * * * /insight/bin/nightly.pl > /insight/var/status/nightly.log 2>&1

30 21 * * *  /insight/bin/find_load > /insight/var/status/find_load_cron_log 2>&1

10 * * * *  /insight/bin/notify >/insight/var/status/notify_log 2>&1

The goldmine crontab may be discussed elsewhere; however, one of these entries, nightly.pl, also performs the periodic Oracle table analysis which is required for the Cost Based Optimizer (CBO) of Oracle.

3.3.2.5.3              Execution

The method to start the context server is not automated.  The files, which start the listener and database, do not start the context server.  This is currently a manual process, which is accomplished, as follows:

$ORACLE_BASE/scripts/start_context.sh

This process will be automated in the future.  The context server is not necessary in order to use context indexes (sic).  It is only necessary for background processes, which re-index the fields.

While it doesn’t seem to hurt to shutdown Oracle and/or reboot the Unix operating system without shutting down the context server, it could corrupt indexes (sic) if the background process was executing which required the context server.  With that in mind, we have created the follow script to shutdown the context server:

$ORACLE_BASE/scripts/stop_context.sh

If you neglect to stop the context server and restart the machine or the database, the context server process will stop because it is also a process within Oracle, and it will need to be manually restarted using the procedure above.

3.3.2.5.4              Maintenance

Context indexes (sic) rely on the Oracle cost based optimizer (CBO).  Queries written for the rules based optimizer (RBO) have a different structure.  Once a table has been “analyzed”, it is automatically queried using the CBO technique.  There is a daily crontab entry in the Unix Oracle account to analyze the Moai tables, as follows:

# Analyze Moai Tables Daily - D.Russell 12/1999

00 04 * * 0-6 /u01/app/oracle/scripts/analyze_tables.sh >/dev/null 2>&1

It is not a good idea to analyze all tables routinely, and absolutely not a good idea to analyze the “sys” and “system” data dictionary tables.  In the event that tables that should not be analyzed accidentally are analyzed, we have techniques, which will reverse the analysis and remove the statistics.

3.3.2.5.5              Verification

The easiest way to tell if the context server is running is to execute the following UNIX command:

ps –elf |grep –i ctx

A more complicated alternative from SQL*Plus follows:

select osuser, type, program from v$session

where program like ‘ctxsrv%’;

This shows you that the context server is not just a Unix process but is defined within Oracle as being an Oracle process.

3.3.2.6                  ERP Server

Oracle Enterprise Resource Planning (ERP) a.k.a., Oracle Financials

<defunct>.com implemented the following modules of Oracle Financials:  General Ledger, Accounts Receivable, Accounts Payable, Fixed Assets, Purchasing, Inventory, and Order Entry.  Three instances are maintained:  Production (PRD), Test (TST), and Vision (VIS). 

The Production instance contains the production financial records for the company.    The Test instance is for testing both code and operational changes and enhancements.  The Finance department uses the Test instance to verify, for example, that new procedures correctly update journals, before implementing the new procedures in Production.  It is imperative that both the PRD and TST instances be available at all times to the Finance department.  The TST and PRD setups must be the same at all times.   

The Vision database is fully configured to showplace virtually all of Oracle’s functionality.   It is available for investigating the impact of new transactions that are outside the current scope of the system.  As <defunct>.com grows and enters new markets, the Vision database will be used to see how complex transactions are configured and to understand process flows in an environment that may be different from our current system.  Vision only needs to be running at that time when such investigations take place.  At all other times, Vision should be shut down to preserve system resources.  

Each instance has its own binaries, so that both application and database patches can be applied first in the TST environment, and then, after successful testing of the changes, applied to PRD.  All changes, no matter small, must be applied first to TST. 

The Oracle Financials Implementation Technical Reference is a comprehensive guide that was written to cover the setup and configuration of the Oracle Financials at <defunct>.com.  The sections include Oracle Support, TARs, Patches, Scripts, Physical DB Design, Veritas and Oracle, RAID and Oracle, Technical Bulletins, Applications Sizing, an Engagement Summary of installation and setups of all three instances, custom printer configurations, Tuning of Oracle I/O, Documentation, Host Environments, Error Screen Shots, Read Me Section, Moai Integration with Oracle Financials, Tradesafe Specs, Correspondence, and Notes.  This book is always located in the bookshelf of the Database Architect.

A comprehensive two-volume set covers the functional setup.  Volume I covers Acceptance Certificates of ERP project manager Progress Reports, the Chart of Accounts configuration, the Flexfields setup for <defunct>.com, Oracle contracts, FastForward Standard Fuctionality, Training, Oracle consultants backgrounds, Schedules and Tasks, and Oracle Reports. 

Volume II covers FastForward Financials, FastForward Inventory and Order Entry, the Scope, Objectives and Approach of the Financials Implementation, Site Assessment, and Process Flows.

An additional multi-volume set of detailed setup screens for each module, including all month end closing procedures was also prepared.  These documents are located in the Finance department, in the Accounting Manager’s bookcase.

Each module has two types of documentation:  functional and technical, and is shared between the Finance Department and Database Architect.

The ERP configuration follows:

·        Operating System: Sun Solaris version 2.6

·        <snip>

3.3.2.7  CRM Application Server

As our out-source help facility, service.com answers first tier email, phones and live chat requests from our homepage.  In the process, they use the Octane software which is a database based application owned by <defunct>.com.  Octane's features include contact management, customer care, help desk, and product (defect) tracking.

Contact management is used by service.com to collect customer contact information when a customer chooses to not register as buyer or seller.  <defunct>.com has created an interface between the customer records in Octane and the customer data contained in Moai.  This modification was done in Octane's default database schema to add one column in b_entity called c_userOID, which service.net will use to identify if the customer is already registered in Live Exchange's database or Octane's database.  A Javascript servlet on the Oracle/Moai side was written to do the data sync.

In its’ next release, we will receive sales force automation and market campaign analysis.  The interfaces to other components will change with that release, as well.

Additional modifications have been made to the Octane database “b_octane” schema by service.com.  The script, which makes this modification, has been loaded into Visual Source Safe (VSS), which was also installed on the Octane server.

Because the changes were database "alters" they do not appear numerically, alphabetically, or logically, in sequence.  The changes are presented, as required.

The CRM configuration follows:

·        Operating System: Windows NT server version 4.0, SP 5

·        <snip>

3.3.2.8  CRM E-Mail Server

The CRM E-Mail Server configuration follows:

·        Operating System: Windows NT server version 4.0, SP 5

·        <snip>

3.4                Business Site

The business site is used to house the hardware for our firewalls, email, development and test servers, workstations, print servers and supporting networking hardware.  A locked enclosure shall house the firewall, email and print servers.  A separate locked area shall house the development and test servers.  A battery backup system shall provide 3 hours backup to all server equipment (except workstations) housed at the site.

The private connection between NOC and business site must provide sufficient bandwidth between application servers to prevent degradation.  For this reason, business Internet traffic travels through the separate connection.  Switches and routers are strategically placed to enable either 10 or 100 Mbit/sec connections between servers and clients.  Static IP’s are used for the servers to maximize security and minimize dynamic allocation.  Dynamic IP’s are used for all client workstations to minimize system administration overhead.

The business office houses the company intranet and PBX telephone switch. 

Multiple servers will handle printers, dynamic IP assignment to workstations, email and connection to the Internet through a firewall.  

All firewall software shall be configured and monitored by approved consultants or company personnel.

Application development software runs on high performance servers.  The applications include web page design, database design tools, middleware application development tools, middleware applications, web server and database server.  This development system should be considered a clean-room environment whereby the product under development has no chance to contaminate the production environment.  Connection between server and workstation is handled through a network connection between UNIX servers and PCs using samba. An Internet connection separated by firewall software on a server connects the business site to the Internet.

Email is provided by internal applications.  The firewall shall check the email addresses of all recipients for validity prior to granting the messages passage.

We will assign IP numbers using DHCP for workstations.  We will use Windows NT Server to provide DHCP, email, authentication and printer server functionality.  This server will reside at the business site.

See Appendix C for Business Site Network Diagram.

3.4.1        Telephone System

The business site houses our PBX telephone switches.  The telephone system is answered automatically by an automated software system supplied by Lucent.  A directory of personnel and telephone extensions is supplied to callers of the business telephone number (858)-638-0026.

Our outsource customer support vendor, service.com, provides <tbd 1-800-xxx-yyyy >  telephone support for a limited amount of customer support. 

3.4.2     Hardware

3.4.2.1                  Switch

The required switch configuration is the following:

·        Cisco 2912XL

3.4.2.2                       Routers

The required router configuration is the following:

·        Cisco 2610

·        CSU/DSU

3.4.2.3                       Hubs

The required hub configuration is the following:

Ø      Cisco Catalyst 2924

3.4.2.4                       Firewall

The firewall server shall be supported by a battery backup system capable of supplying power for 3 hours.  The required firewall configuration is the following:

·        Sun Ultra 10 Web Server

·        333 MHz processor

·        1GB memory

·