README hrtxq_1113b On the SGI Origin300 at NYU I use the following compile line: f90 hrtxq_1113b.f -o hrtxq_1113b_O2_r8_mp -O2 -r8 -mp -lscs_mp -OPT:IEEE_arithmetic=1 ^^^ | ___________________________________________________| | | The scs library provides for some routines adapted from Cray libraries. If you do not have this library I will provide you with some substitute routines. I ran with the following OpenMP environment variables set: setenv OMP_SCHEDULE "dynamic" setenv OMP_NUM_THREADS 4 setenv OMP_DYNAMIC TRUE It may be that on a larger machine you can increase the number of threads. I have not tried this here since the Origin300 was enlarged. The basic files that the program READS are: heart.in (the initial configuration of the immersed boundary) markers.in (the initial positions of a set of fluid markers) In addition the program willl optionally look for checkpoint/restart files (suggestively named fort.1 and fort.2) which are written at an interval set in the code. The purpose of these files is to permit a computation to be continued after it has stopped. Computations stop typically for a number of reasons: because the user sets a stopping point and the executing code gets to that point; because the computation becomes unstable, which usually occurs following the onset of systole; because the computer itself stopped. ------------------------------------------------------------------------------- To start an experiment, edit hrtxq_1113b.f as follows: set the value of DENOVA to .TRUE. This indicates NOT to look for a checkpoint/restart file. set the value of EXPNUM. EXPNUM is the experiment number. The experiment number is encoded in the name of the output files. Any number greater than 999 and less than 10000 is acceptable. I suggest you start with 2000. set the value of NSTEPS. NSTEPS is the desired number of timesteps before stopping. In an environment where you have to compete with other users, this number should be large because the program needs a lot of resources and probably waits in a queue for a while. On the other hand it should not be too large, that is, you don't want the program to run for a long time before you look at results, just in case you made a mistake. I typically set NSTEPS=8193, and I typically increment EXPNUM by 1 every 8193 timesteps. I also create a new directory every time I increment EXPNUM, so that no single directory has too many files. set the value of INTWR. INTWR is the timestep interval at which checkpoint files are written. Make this as large as you feel comfortable with. If the machine crashes, a maximum of INTWR timesteps will be lost. I typically set INTWR=2048. Checkpoint information is written alternately to Fortran Unit 1 and Fortran Unit 2 (fort.1 and fort.2). This means that if the machine crashes in the middle of writing one of these files, (this happened to me) you still have the other file. When I create a new directory associated with the incremented EXPNUM, I COPY heart.in, markers.in, fort.1 and fort.2 from the previous directory. This means that there are permanent copies of these checkpoint files that you can go back to, if need be. set the value of INTOUT. INTOUT is the timestep interval at which displayable output is written. I typically set INTOUT somewhere between 64 and 512. If INTOUT is small you fill up your mass storage, if INTOUT is large your output may not look smooth in time. Output files have names like hrteeee_kkkkkk.ab where eeee is the experiment number kkkkkk is the timestep ab is: xf (boundary points) mk (fluid markers) vp (velocity & pressure on the grid) fl (flow through the 4 valves) To restart an experiment, change the value of DENOVA to .FALSE. and recompile. On restarting, the program will check if fort.1 or fort.2 is present. If both are present it uses the later one. If one is present it uses that one. If neither is present it complains and stops. The code checks the CFL condition and halts if it is being violated. This allows the output buffers to be flushed and no output is lost. If the program stops because of a CFL condition violation, you must restart with a smaller timestep. The code only permits refinement by powers of 2. On the first such restart, set NREFOLD to 1 and NREF to 2, and recompile. If the program needs to be restarted SUBSEQUENTLY from a checkpoint file written AFTER you set NREFOLD to 1 and NREF to 2, then you need to set NREFOLD to 2 and recompile. It is not unusual to have two CFL condition violations in an experiment. When restarting immediately after the second CFL condition violation, set NREFOLD to 2 and NREF to 4, and recompile. If the program needs to be restarted SUBSEQUENTLY from a checkpoint file written AFTER you set NREFOLD to 2 and NREF to 4, then you need to set NREFOLD to 4 and recompile. You will note that the main program loop (DO 5 KLOK=KLOK1,KLOKEND) has an inner loop (DO 4 NR=1,NREF). The result of this is that each increment of KLOK represents the same amount of physical time, irrespective of the amount of timestep refinement. Since output is written after the inner loop, output files are written at regular intervals of physical time, irrespective of the amount of timestep refinement. I am sure there is some way to automate all this, but I have not been able to figure one out. Possibly if more information were written into the checkpoint file to reflect the state of timestep refinement this could all be made automatic. The flow out of each source depends on the computed pressure at the location of that source. We use a second order method. When timestep refinement is required, the second order method needs a value of the pressure at a time AFTER the time of the checkpoint file in order to get a good estimate of the pressure at the new restart time. For this reason the value of NSTEPS must be 1 larger than an integer multiple i=of INTWR. For example, if you are using INTWR=2048, then use NSTEPS=4*2048+1=8193. This guarantees that at least one timestep passes after the checkpoint file time, and after this 1 additional timestep a file (fort.31 or fort.32) is written containing the required pressures. What this means to you the user is that hrtxq_1113b will write fort.1, fort.2, fort.31 and fort.32. Any subsequent restarts using one of these fort.1 or fort.2 will also require fort.31 and fort.32, as well as heart.in and markers.in. I am sure this is not completely clear, so after you read it feel free to email with questions. --Dave McQueen (mcqueen@cims.nyu.edu)