An update on recent ART report outages
In the spirit of transparency, we are sharing the extent of the outage, the timeline of events, and what actions we are taking.
On June 4, ICA learned some ART reports were unable to successfully complete when users tried to run them. The down reports included Coordinated Entry reports, among them the report used for Priority Lists across the state, and state program reports for the LTH/HPH, HYA and THP programs.
What happened? Our path to investigation and resolution
ICA began to investigate when a few users contacted our Helpdesk to report the issue. Initially, we believed the issues were contained to a small number of frequently-run reports, and that they were related to updates made to our system by Wellsky, our software vendor, the day prior.
We implemented a temporary fix to the 254 Priority List report the day after the issues were reported, and held on any major report updates while we worked with Wellsky and continued our investigation. Our team checked all 100+ reports in our REPORTcollection to assess how widespread the issue might be. There are approximately 20 down reports.
What does "down reports" or "outage" mean in this case?
To download an Excel workbook or PDF of client data in HMIS, a user schedules a report, or selects the date range and provider(s) they would like to appear, in ART. The report is ready to download when its status shows "Completed" in the user's list of scheduled reports.
Sometimes an error can cause that report to get stuck and fail to run, so it cannot download. Many times, this is due a simple typo in the prompts. However, in this case, it is not user error.
How do I know if a report I schedule is down?
Instead of a "completed" status, the report will show a "failure":
The exact source of the report failure will display when clicking on the magnifying glass. If you see an error that references a column that does not exist, it is part of this outage.
What we know now
We believe the issues were caused by an unrelated update Wellsky made on June 3, due to an error in their code. We have also confirmed only reports containing outdated or retired data elements within subassessments were affected. To resolve the issue, ICA is working with Wellsky to identify remedies they can apply. At the same time, ICA is preparing to carefully extract the source of the issues in each report. If Wellsky is unable to remedy the underlying cause and we must manually fix all 20 reports, this will take some time.
We will triage reports based on their frequency of use and direct impact to clients, then based on reporting deadlines.
What you can do
ICA has created a resource to view a list of affected reports and to check on the progress toward resolution: click here to access this resource.
If you encounter a similar issue, where a scheduled report fails to complete and you see an error message containing the words "does not exist", please contact the Helpdesk to alert us of the issue.
Continue HMIS data entry as usual. ServicePoint, and most reports, continue to function normally.
Thank you
We know how important reports are to our users, and we are continuing to carefully and quickly resolve the issue. We apologize for the inconvenience caused by this unexpected event, and we thank our users for their understanding and patience.