(presenter), Y. Omelchenko, J. Driscoll, N. Omidi,
R. Fujimoto
, and K.S. Perumalla
SciberNet, Inc.
Georgia Institute of Technology
email: <homa@san.rr.com>
H. Karimabadi (presenter), Y. Omelchenko, J. Driscoll, N. Omidi,
R. Fujimoto, and K.S. Perumalla
SciberNet, Inc.
Georgia Institute of Technology
email: <homa@san.rr.com>
Computer simulations of many important complex physical systems have reached a barrier as existing techniques are ill equipped to deal with the multi-physics, multi-scale nature of such systems. An example is the solar wind interaction with the Earth's magnetosphere. This interaction leads to a highly inhomogeneous system consisting of discontinuities and boundaries and involves coupled processes operating over spatial and temporal scales spanning several orders of magnitude. The brute force method of using full particle simulations for global simulations of the Earth's magnetosphere, with electron scale resolution everywhere in the simulation domain is an impossibility. It would take over 107 years on the fastest parallel computers. The ideal global code would have to take full advantage of the fact that there are regions within the magnetosphere with different modeling requirements: resolve electron physics only in the thin layers in the magnetosphere where reconnection is operational, resolve ion scales in the regions where the boundaries are formed and a lower resolution everywhere else. Such a code does not exist.
We have taken a new approach to simulation of such complex systems. The conventional time-stepping grid-based PIC models provide the sequential execution of synchronous (time-driven) field and particle updates. In a synchronous simulation the distributed field cells and particles undergo simultaneous state transitions at regular discrete time intervals. In contrast to this well known technique, we propose a new, asynchronous type of PIC simulations based on a discrete event-driven (as opposed to time-driven) approach, where particle and field time updates are carried out in space on a "need-to-be-done-only" basis. In these simulations particle and field information "events" are queued and continuously executed in time in a manner similar to that employed in the theory of cellular automata (CA). The rational of this novel approach is not to try to describe a complex plasma system by using instance differential equations but let the complexity emerge by modeling interaction of adjacent plasma cells following elementary rules that reflect the underlying physics laws. Event-driven PIC simulations automatically guarantee that the system progress in time captures all state changes without processing "idle" information. The parallelization of both synchronous and asynchronous PIC models is realized by decomposing the global geometric domain into subdomains. In each subdomain, the individual cells and particles are aggregated into data containers, which are mapped to the parallel processors. The parallel execution of the conventional (time-driven) simulations is commonly achieved by copying field information from the inner lattice cells to the ghost cells of the neighboring subdomains and exchanging out-of-bounds particles between the processors at the end of each update cycle. However, in parallel asynchronous PIC simulations both particle and field transitions (events) are not synchronized by a global clock, but occur at arbitrary time intervals, which potentially may introduce severe synchronization problems if processors are allowed to get ahead in time of other processors ("optimistic" simulation). As a result, a processor may receive an event message from a neighbor with a simulation time stamp that is in its own past, thus causing a causality error. We use an optimistic computing strategy to solve this problem. This approach allows causality errors to occur and provides a recovery mechanism for undoing such effects by rolling the local state of incorrect computations back in time and implementing anti-event messages (messages that cancel the originals when reaching the destination). We are exploring the use of reverse execution techniques to implement rollback in order to substantially reduce the time and memory required using state saving techniques.