Automatic Batch processing

[Inf51]

I have a PHP script which regularly needs to process over 75000 records.

It takes around 30 minutes to run which is fine when it's running on my standalone LAMP box locally - I just set it running and leave it.

I would however like to be able to run it on other servers - most of which would probably timeout. I'd also like to be able to update the user during the process (showing number of records processes, time taken etc.) as at the moment I have no output until the script has finished.

It loops through the recordset of makes descisions and updates on other tables - I've tried using MySQL limit to process a batch at a time and then reload the page with the new starting record number to process the next batch but Firefox seems to see this as a perpetual loop and kills it.

Can anyone recommend any way of having a script run through a batch of MySQL records then maybe pause for 30 seconds (to be web/MySQL server friendly) before running another batch and show the current process status as it does it?

Thanks.

[Weedpacket]

Since it's regular it sounds like what you want is a cron job (search for that).

On the other hand, since you want to use a browser to view the progress, use set_time_limit to regularly reset the timeout limit while running through the entire batch and periodically produce output (to prevent the page loading from timing out).

Or, in a modification of the previous idea, have the browser use JavaScript to periodically query the server for the current status and use the response to update the progress page.

[Inf51]

Thanks, it's something that needs running regularly but only when a user manually wants to so scheduling it with a cron job isn't really needed.

I have set the PHP timeout which works on my own server but not all servers are going to take kindly to that. I'm just not sure how in PHP I can get it to run a batch and then reload the page and run another batch so the script runs for less than 30 seconds then waits 30 seconds before running another batch for 30 seconds.

My first try was to process a batch of 250 records which it can easily do in less than 30 seconds, at the end of the batch I used header("location: mypage.php?start_rec=250"); so that when the page reloaded it would look at the query string and use the start_rec in the MySQL limit to load the next batch or records.

I worked all the way through my script and the logic looked to me like it should work but as the page is redirecting to itself Firefox seemed to think this was an infinite loop and killed it.

I'm just not sure how I can get a PHP script to run and then reload itself to run the next batch without the browser objecting - or is a header redirect not the thing to use?

[dagon]

skip the browser issues by running the php script from the command line with

PHP Code:

exec('nohup php file.php >/dev/null 2>&1 &");


there's no timeout for php files run from the command line.