ISSUE:
The FCC.today website was observed displaying FCC actions twice. There was a large amount of action activity as it was a renewal grant day.
ROOT CAUSE:
eLMS processing involves having to parse select FCC applications upon any action (filing, grant or dismissal). This is a time consuming process. During the day, LMSBOT3 is scheduled to operate once every 5 minutes. It appears that LMSBOT3 was still processing applications when the LMSBOT3 scripts were scheduled to operate again. This caused LMSBOT3 to pull a list of applications that were in the LMS real time feed but had not yet been processed by the applicaiton parser. This caused those items to get a second run thus creating the duplicate record. Today, there were a very high number of application actions (renewals) that were processed by the FCC Audio Division that caused the high demand. This was our first opportunity to test the system on a "renewal day".
RESOLUTION:
After stopping all LMSBOT and CDBSBOT services, a new "system lock" function was added to LMSBOT3 that would prevent more than one LMSBOT3 retriever and processor applications from operating at the same time. When the LMSBOT3 retriever runs, it will set a lock that will prevent the retriever and processor from launching again. Once the LMSBOT3 retriever is completed, it will partially unlock the system to permit the processor to start. Once the processor starts, the system is fully locked again. Once the processor has completed, the LMSBOT3 system will be fully unlocked.
If, on the schedule, LMSBOT3 is scheduled to run and encounters a locked system. The application will wait 5 or 15 minutes and then attempt to start the retriever and processor again.
Currently, LMSBOT3 operates every 5 minutes on Monday through Friday between 8AM and 8PM Eastern Time. At all other hours, LMSBOT3 runs every 15 minutes.
LMSBOT3 and CDBSBOT3 have been restored to the schedule. There were no software changes to CDBSBOT3 on this emergency fix.
To address with this and future "Renewal Days", we have extended the capacity of LMSBOT3 from 900 to 1,200 applications. We may go higher depending on load.
