c# - Handle multiple threads, one out one in, in a timed loop -

i need process large number of files overnight, defined start , end time avoid disrupting users. i've been investigating there many ways of handling threading i'm not sure way go. files come exchange inbox attachments.

my current attempt, based on examples here , bit of experimentation, is:

 while (datetime.now < dtendtime.value)  {             var finished = new countdownevent(1);             (int = 0; < numthreads; i++)             {                   object state = offset;                  finished.addcount();                 threadpool.queueuserworkitem(delegate                 {                     try                     {                         startprocessing(state);                     }                                         {                         finished.signal();                     }                 });                  offset += numberoffilesperpoll;              }             finished.signal();             finished.wait();            } 

it's running in winforms app @ moment ease, core processing in dll can spawn class need windows service, console running under scheduler, easiest. have windows service set timer object kicks off processing @ time set in config file.

so question - in above code, initialise bunch of threads (currently 10), wait them process. ideal static number of threads, 1 finishes fire off another, , when end time wait threads complete. reason files i'm processing variable sizes - might take seconds process , might take hours, don't want whole application wait while 1 thread completes if can have ticking along in background. (edit)as stands, each thread instantiates class , passes offset. class gets next x emails inbox, starting @ offset (using exchange web services paging functionality). each file processed, it's moved separate folder. of replies far, i'm wondering if should grab e-mails in outer loop, , spawn threads needed. cloud issue, have backlog of e-mails i'm trying process through. once backlog has been cleared, it's nightly run have lower load.

on average there around 1000 files process each night.

update

i've rewritten large chunks of code can use parallel.foreach , i've come against issue thread safety. calling code looks this:

public bool startprocessing()         {              finditemsresults<item> emails = getemails();                var source = new cancellationtokensource(timespan.fromhours(10));              // process files in parallel, maximum thread count.             var opts = new paralleloptions { maxdegreeofparallelism = 8, cancellationtoken = source.token };              try             {                 parallel.foreach(emails, opts, processattachment);             }              catch (operationcanceledexception)             {                 console.writeline("loop cancelled.");             }             catch (exception err)             {                 writetologfile(err.message + "\r\n");                 writetologfile(err.stacktrace + "r\n");             }             return true;         } 

so far (excuse temporary error handling). have new issue fact properties of "item" object, email, not being threadsafe. example when start processing e-mail, move "processing" folder process can't grab - turns out several of threads might trying process same e-mail @ time. how guarantee doesn't happen? know need add lock, can add in foreach or should in processattachments method?

use tpl:

parallel.foreach( enumeratefiles(),                   new paralleloptions { maxdegreeofparallelism = 10 },                   file => processfile( file ) ); 

make enumeratefiles stop enumerating when end time reached, trivially this:

ienumerable<string> enumeratefiles() {     foreach (var file in directory.enumeratefiles( "*.txt" ))         if (datetime.now < _endtime)             yield return file;         else             yield break; }