[Elluciansupport] Root Cause Analysis - February 3rd Recruiter connectivity interruption

Thu Apr 23 16:09:01 CDT 2015

Hi all - please see below for the root cause analysis related to the February 3rd Recruiter connectivity interruption.

[cid:image013.jpg at 01D07DE8.305C62B0]<http://app.learn.ellucian.com/e/er?utm_campaign=201301%20Global%20Inspire%20Awards%20Email%20ADD&utm_medium=email&utm_source=Eloqua&s=1855015449&lid=10&elq=e8c900450bcb466eb9e980e5be646947>

ELLUCIAN CLOUD SERVICES DATA CENTER

Root Cause Analysis

Incident Date: 2/3/15 9:30 a.m. (EST) - 2/3/15 5:15 p.m. (EST)

Confidentiality

Distribution of this document is limited. Access should only be granted to those with a business related need-to-know. If you have any questions pertaining to the distribution of this document, please contact your Engagement Manager.

Issue Summary/Time Frame
2/3/15 9:30 a.m. (EST) - 2/3/15 5:15 p.m. (EST):

On 2/3/15, at 9:30 a.m. (EST) Ellucian received the following Recruiter connectivity alert: "No connection could be made because the target machine actively refused it at 149.24.114.115:80." Technical teams were engaged to troubleshoot and determine the source of the issue.

The following troubleshooting steps were implemented:

9:30 a.m. (EST) - A quick Telnet 149.24.114.115 to port 80 identified that the port was blocked. Since all communication should have been going over port 443, initial investigation focused why port 80 was unreachable.

9:30 a.m. (EST) - 10:00 a.m. (EST) - Additional error messages were reported by more clients.

10:00 a.m. (EST) - Incident was escalated to global investigation.

10:00 a.m. (EST) - The infrastructure team was engaged along with Dimension Data.  Dimension Data reopened port 80 to the ADFS Load Balancer.

1:15 p.m. (EST) - Determined CRM Application was impacted (not just the WFE sites).

1:15 p.m. (EST)  - Reviewed Server Logs, CRM Logs, IIS Logs on the 20+ server deployment in AWS.  With port 80 open, we restarted the ADFS Services enabling the claim to be reestablished.  This appeared to mitigate the issue. After testing the system with positive results, Ellucian issued a client communication that the issue had been resolved.

2:00 p.m. (EST) - 3:00 p.m. (EST) - The issue reoccurred.

3:00 p.m. (EST) - Review of server logs did not identify the issue.  Started review of other possible blocked ports.  Contacted Dimension Data for health check from Load Balancer.  Dimension Data indicated that a health check had not been established.

4:00 p.m. (EST) - Still experiencing sporadic issues, we began removing servers from the configuration to see if we could narrow the scope of the issue.  Technicians forced all ADFS traffic to the secondary server, which seemed to fix the issue. The issue reappeared within 15 minutes.

4:30 p.m. (EST) - Forced all communication to the Primary ADFS server, which stabilized most clients. Two-three clients were still experiencing issue. We attributed the error to caching.

5:15 p.m. (EST) -The secondary ADFS and Proxy servers could not ping the Primary ADFS server. Technicians looked in ad.admin for Primary ADFS server and there was nothing there. The Secondary ADFS server was there.  Technician added an A record for Primary ADFS server then flushed the DNS on all four ADFS servers. Technicians restarted the ADFS Service.  Elluciancloud.com records were pinged to both the Primary and Secondary server and both were functional.

Follow up with clients indicated issues had been resolved. Solutions were tested against multiple clients through multiple browsers, and against the WFE and CRM URLs. We were unable to duplicate error.

NOTE: Not all clients were impacted by connectivity issues.
Resolution
Added the Primary ADFS server to internal DNS and opened port 80 to the ADFS Load Balancer.
Root CauseAlthough connectivity was restored, Ellucian created a problem ticket (PRB00050697)
to investigate the root cause:

At the time of the incident, Ellucian was migrating multiple clients to an upgraded application platform, which involved a number of necessary firewall port changes.  During the course of those changes, a port was inadvertently blocked.  Additionally, a backup port was not available at the application level to accommodate data traffic.

            Corrective Actions:

*                Added the Primary ADFS server to internal DNS.

*                Opened port 80 to the ADFS Load Balancer.

*                Application level changes have been made to allow for an alternate port, alleviating data traffic conditions.

*                Ellucian has reviewed our port blocking procedures with our vendor to help mitigate errors when managing access to ports.

Contact:        If you have any questions or concerns about this notification, please contact your engagement manager.
Thank You,
Your Ellucian Cloud Services Team

FOLLOW US
  [cid:image014.png at 01D07DE8.305C62B0] <http://twitter.com/@ellucianinc>   [cid:image015.png at 01D07DE8.305C62B0] <http://www.facebook.com/ellucianinc>   [cid:image016.png at 01D07DE8.305C62B0] <http://www.linkedin.com/company/ellucian>

[cid:image017.jpg at 01D07DE8.305C62B0]

Banner(r), Colleague(r), and Luminis(r) are trademarks of Ellucian Company L.P. or its affiliates and are registered in the U.S. and other countries. Ellucian(tm), Ellucian Advance(tm), Ellucian Degree Works(tm), Ellucian Course Signals(tm), Ellucian SmartCall(tm), and Ellucian Recruiter(tm) are trademarks of Ellucian Company L.P. or its affiliates. Other names may be trademarks of their respective owners.
(c) 2009-2013 Ellucian Company L.P. and its affiliates.

Ellucian
4375 Fair Lakes Court Fairfax, Virginia 22033
800-223-7036

Feel free to reach out to me directly with any questions.

Trent Schneider, PMP | Engagement Manager | ellucian(r)  | 4375 Fair Lakes Court, Fairfax, VA  22033 | O:+1.570.704.5634 | M:+1.610.858.0491 | trent.schneider at ellucian.com<mailto:trent.schneider at ellucian.com> | www.ellucian.com

Follow us: [cid:image002.jpg at 01CE82FD.CF04D420] <http://commons.ellucian.com/>  [cid:image004.jpg at 01CE82FD.CF04D420] <http://www.facebook.com/ellucianinc>  [cid:image006.jpg at 01CE82FD.CF04D420] <http://www.linkedin.com/company/ellucian>  [cid:image008.jpg at 01CE82FD.CF04D420] <http://twitter.com/@ellucianinc>

CONFIDENTIALITY: This email (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this email in error, please notify the sender and delete this email from your system. Thank you.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://newlist.semo.edu/pipermail/elluciansupport/attachments/20150423/19e3af34/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image013.jpg
Type: image/jpeg
Size: 7554 bytes
Desc: image013.jpg
Url : http://newlist.semo.edu/pipermail/elluciansupport/attachments/20150423/19e3af34/attachment-0006.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image014.png
Type: image/png
Size: 579 bytes
Desc: image014.png
Url : http://newlist.semo.edu/pipermail/elluciansupport/attachments/20150423/19e3af34/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image015.png
Type: image/png
Size: 694 bytes
Desc: image015.png
Url : http://newlist.semo.edu/pipermail/elluciansupport/attachments/20150423/19e3af34/attachment-0005.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image016.png
Type: image/png
Size: 506 bytes
Desc: image016.png
Url : http://newlist.semo.edu/pipermail/elluciansupport/attachments/20150423/19e3af34/attachment-0006.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image017.jpg
Type: image/jpeg
Size: 2518 bytes
Desc: image017.jpg
Url : http://newlist.semo.edu/pipermail/elluciansupport/attachments/20150423/19e3af34/attachment-0007.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image018.png
Type: image/png
Size: 167 bytes
Desc: image018.png
Url : http://newlist.semo.edu/pipermail/elluciansupport/attachments/20150423/19e3af34/attachment-0007.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image019.jpg
Type: image/jpeg
Size: 816 bytes
Desc: image019.jpg
Url : http://newlist.semo.edu/pipermail/elluciansupport/attachments/20150423/19e3af34/attachment-0008.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image020.jpg
Type: image/jpeg
Size: 810 bytes
Desc: image020.jpg
Url : http://newlist.semo.edu/pipermail/elluciansupport/attachments/20150423/19e3af34/attachment-0009.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image021.jpg
Type: image/jpeg
Size: 829 bytes
Desc: image021.jpg
Url : http://newlist.semo.edu/pipermail/elluciansupport/attachments/20150423/19e3af34/attachment-0010.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image022.jpg
Type: image/jpeg
Size: 781 bytes
Desc: image022.jpg
Url : http://newlist.semo.edu/pipermail/elluciansupport/attachments/20150423/19e3af34/attachment-0011.jpg 