I am running an import of data process that takes quite a long time to process. I am using the blocking Progress Bar within the action button settings and it is set as synchronous. The process creates numerous non-persistent objects based on a Json transformation and then processes these through into the apps persistent entities. The process takes quite a bit longer than I would expect - roughly 23 minutes to process 2500 lines of data (0.5Mb) and is causing a few issues.
1. The following error is thrown client side during the process which causes a pop up error being displayed
[error] 166#0: *2873 upstream timed out (110: Connection timed out) while reading response header from upstream,
2. The following is also thrown in the log
WARNING - WebUI: Feedback size of 14846 exceeds threshold of 5000 items which can result in severe performance problems.
I am currently cutting the size of the import files from 10,000 down to 2500 lines so that the process finishes, these could potentially be far bigger. Resolving the above errors would be a help and any advice on whether the timing to process is reasonable or not would be helpful too.
To help give an idea of the process, I am using a custom retrieve, currently set at 100 records in an outer loop, an inner loop processes the 100 records and on each inner iteration it does a Json transformation of the passed file creating all the non-persistent entities, processes the information based on business logic and creates the persistent objects required by the app with a commit on these and finally does a delete on the non-persistent top entity to help free up memory.