2014-08-28

Profiling an high-performance actor application for metagenomics


I am currently in an improvement phase where I break, build and improve various components of the system.


The usual way of doing things is to have a static view of one node among all the nodes inside an actor computation. The graphs look like this:


512x16


1024x16



1536x16



2048x16




But with 2048 nodes, the one single selected node may not be an accurate representation of what is going on. This is why, using Thorium profiles, we are generating 3D graphs instead. They look like this:



512x16


1024x16


1536x16


2048x16


2014-08-02

The public datasets from the DOE/JGI Great Prairie Soil Metagenome Grand Challenge



I am working on a couple of very large public metagenomics datasets from the Department of Energy (DOE) Joint Genome Institute (JGI). These datasets were produced in the context of the Grand Challenge program.

Professor Janet Jansson was the Principal Investigator for the proposal named Great Prairie Soil Metagenome Grand Challenge ( Proposal ID: 949 ).


Professor C. Titus Brown wrote a blog article about this Grand Challenge.
Moreover, the Brown research group published at least one paper using these Grand Challenge datasets (assembly with digital normalization and partitioning).

Professor James Tiedje presented the Great Challenge at the 2012 Metagenomics Workshop.

Alex Copeland presented interesting work at Sequencing, Finishing and Analysis in the Future (SFAF) in 2012 related to this Grand Challenge.



Jansson's Grand Challenge included 12 projects. Below I made a list with colors (one color for the sample site and one for the type of soil).

  1. Great Prairie Soil Metagenome Grand Challenge: Kansas, Cultivated corn soil metagenome reference core (402463)
  2. Great Prairie Soil Metagenome Grand Challenge: Kansas, Native Prairie metagenome reference core (402464)
  3. Great Prairie Soil Metagenome Grand Challenge: Kansas, Native Prairie metagenome reference core (402464) (I don't know why it's listed twice)
  4. Great Prairie Soil Metagenome Grand Challenge: Kansas soil pyrotag survey (402466)
  5. Great Prairie Soil Metagenome Grand Challenge: Iowa, Continuous corn soil metagenome reference core (402461)
  6. Great Prairie Soil Metagenome Grand Challenge: Iowa, Native Prairie soil metagenome reference core (402462)
  7. Great Prairie Soil Metagenome Grand Challenge: Iowa soil pyrotag survey (402465)
  8. Great Prairie Soil Metagenome Grand Challenge: Wisconsin, Continuous corn soil metagenome reference core (402460)
  9. Great Prairie Soil Metagenome Grand Challenge: Wisconsin, Native Prairie soil metagenome reference core (402459)
  10. Great Prairie Soil Metagenome Grand Challenge: Wisconsin, Restored Prairie soil metagenome reference core (402457)
  11. Great Prairie Soil Metagenome Grand Challenge: Wisconsin, Switchgrass soil metagenome reference core (402458)
  12. Great Prairie Soil Metagenome Grand Challenge: Wisconsin soil pyrotag survey (402456)

I thank the Jansson research group for making these datasets public so that I don't have to look further for large politics-free metagenomics datasets.


Table 1: number of files, reads, and bases in the Grand Challenge datasets. Most of the sequences are paired reads.
Dataset
File count
Read count
Base count
Iowa_Continuous_Corn_Soil (details)
252 055 601 258196 708 830 076
Iowa_Native_Prairie_Soil (details)253 750 844 486326 986 888 235
Kansas_Cultivated_Corn_Soil (details)302 677 222 281272 276 185 410
Kansas_Native_Prairie_Soil (details)335 126 775 452597 933 511 278
Wisconsin_Continuous_Corn_Soil (details)181 912 865 700192 128 891 088
Wisconsin_Native_Prairie_Soil (details)202 098 317 886211 016 377 208
Wisconsin_Restored_Prairie_Soil (details)6347 778 67052 514 579 170
Wisconsin_Switchgrass_Soil (details)7448 382 76658 323 428 574
Total16418 417 788 4991 907 888 691 039


At Argonne we are using these datasets to develop a next-generation metagenomics assembler named "Spate" built on top of the Thorium actor engine. The word spate means a large number of similar things or events appearing or occurring in quick succession. With the actor model, every single message is an active message. Active messages are very neat and there is a lot of them with the actor model.


Similar posts:

2014-08-01

The Thorium actor engine is operational now, we can start to work on actor applications for metagenomics

I have been very busy during the last months. In particular, I completed my doctorate on April 10th, 2014 and we moved from Canada to the United States on April 15th, 2014. I started a new occupation on April 21st, 2014 at Argonne National Laboratory (a U.S. Department of Energy laboratory).

But the biggest change, perhaps, was not one listed in the enumeration above. The biggest change was to stop working on Ray. Ray is built on top of RayPlatform, which in turn uses MPI for the parallelism and distribution. But this approach is not an easy way of devising applications because message passing alone is a very leaky, not self-contained, abstraction. Ray usually works fine, but it has some bugs.

The problem with leaky abstractions is that they lack simplicity and are way too complex to scale out.

For example, it is hard to add new code to an existing code base without breaking anything. This is the case because MPI only offers a fixed number of ranks. Sure, the MPI standard has some features to spawn ranks, but it's not supported on most platforms and when it is ranks are spawned as operating system processes.

There are arguably 3 known methods to reduce the number of bugs. First is to (1) write a lot of tests. But it's better if you can have a lower number of bugs in the first place. The second one is to use pure (2) functional programming. The third is to use the (3) actor model.

If you look at what the industry is doing, Erlang, Scala (and perhaps D) use the actor model of computation. The actor model of computation was introduced by the legendary (that's my opinion) Carl Hewitt in two seminal papers (Hewitt, Bishop, Steiger 1973 and Hewitt and Baker 1977).

Erlang is cooler than Scala (this is an opinion, not a fact) because it enforces both the actor model and functional programming whereas Scala (arguably) does not enforce anything.

The real thing, perhaps, is to apply the actor model to high-performance computing. In particular, I am applying it to metagenomics because there is a lot of data. For example, Janet Jansson and her team generated huge datasets in 2011 in the context of a Grand Challenge.

So basically I started to work on biosal (biological sequence analysis library) on May 22th, 2014. The initial momentum for the SAL concept (Sequence Analysis Library) was created in 2012 at a workshop. So far, at least two projects (that I am aware of) are related to this workshop: KMI (Kmer Matching Interface) and biosal.

The biosal team is small: we are currently 6 people and we are only 2 that are pushing code.

Here is the current team:







Person (alphabetical order) Roles in biosal project
Pavan Balaji
  • MPI consultant
Sébastien Boisvert
  • Master branch owner
  • Actor model enthusiast
  • Metagenomics person
  • Scrum master
Huy Bui
  • PAMI consultant
  • Communication consultant
Rick Stevens
  • Supervisor
  • Metagenomics person
  • Stakeholder
  • Product owner
  • Exascale computing enthusiast
Venkatram Vishwanath
  • Actor model enthusiast
  • Exascale computing enthusiast
Fangfang Xia
  • Product manager
  • Actor model enthusiast
  • Metagenomics person





When I started to implement the runtime system in biosal, I did not plan to give a name to that component. But I changed my mind because the code is general and very cool. It is a distributed actor engine in C 1999, MPI 1.0, and Pthreads and it's named Thorium (like the atom).

Thorium uses the actor model, but does not use functional programming.

It is quite easy to get started with this. It is a two step process.

The first step is to create an actor script (3 C functions called init, destroy and receive). For a given actor script, you need to write 2 files (a H header file and a C implementation file).

The first step defines a actor script structure like this:

struct bsal_script hello_script = {
    .name = HELLO_SCRIPT,
    .init = hello_init,
    .destroy = hello_destroy,
    .receive = hello_receive,
    .size = sizeof(struct hello),
    .description = "hello"
};
The prototype for the 3 functions are:



Function Concrete actor function
init
void hello_init(struct bsal_actor *actor);
destroy
void hello_destroy(struct bsal_actor *self);
receive
void hello_receive(struct bsal_actor *self,
   struct bsal_message *message);


The functions init and destroy are called automatically by Thorium when an actor is spawned and killed, respectively. The function receive is called automatically by Thorium when the actor receives a message. Sending messages is the only way to interact with an actor.

There is only one (very simple) way to send a message to an actor:

void bsal_actor_send(struct bsal_actor *self, int destination, struct bsal_message *message);


The second step is to create a Thorium runtime node in a C file with a main function (around 10 lines).

After creating the code in two easy steps, you just need to compile and link the code.

After that, you can perform actor computations anywhere. A typical command to do so is:

mpiexec -n 1024 ./hello_world -threads-per-node 32
 
 

Obviously, you need more than just one actor script to actually something cool with actors.


On a final note, biosal is an object-oriented project. The current object is typically called self, like in Swift, Ruby, and Smalltalk.

2014-07-21

Is it required to use different priority in a high-performance actor system ?



I was reading a log file from an actor computation. In particular, I was looking at the outcome of a kmer counting computation performed with Argonnite, which runs on top of Thorium. Argonnite is an application in the BIOSAL project and Thorium is the engine of the BIOSAL project (which means that all BIOSAL applications run on top of Thorium).




In BIOSAL, everything is an actor or a message. And these are handled by the Thorium engine. Thorium is a distributed engine. A computation with Thorium is distributed across BIOSAL runtime nodes. Each node has 1 pacing thread and 1 bunch of worker threads (for example, with 32 threads, you get 1 pacing thread and 31 workers).




Each worker is responsible for a subset of the actors that live inside a given BIOSAL node. Obviously, you want each worker to have their own actors to keep every worker busy. Each worker has a scheduling queue with 4 priorities: max, high, normal, and low (these are the priority used by the Erlang ERTS called BEAM). An actor with max priority always wins. Otherwise, there is a ratio of N*N*N to N*N to N between high, normal, and low. This ratio protects against starvation.

In the current code, every actor is classified in normal by default.

If I put every actor in the same priority (BSAL_PRIORITY_NORMAL), I see this (for node 0 and worker 5) when I run a actor computation on one single physical machine (no latency when passing messages around):

node/0 worker/5 SchedulingQueue Levels: 4
node/0 worker/5 scheduling_queue: Priority Queue 1048576 (BSAL_PRIORITY_MAX), actors: 0
node/0 worker/5 scheduling_queue: Priority Queue 128 (BSAL_PRIORITY_HIGH), actors: 0
node/0 worker/5 scheduling_queue: Priority Queue 64 (BSAL_PRIORITY_NORMAL), actors: 4
node/0 worker/5 [0] actor aggregator/1291935834 (1 messages)
node/0 worker/5 [1] actor kmer_store/1477943366 (511 messages)
node/0 worker/5 [2] actor aggregator/443747990 (1 messages)
node/0 worker/5 [3] actor aggregator/710261816 (1 messages)
node/0 worker/5 scheduling_queue: Priority Queue 4 (BSAL_PRIORITY_LOW), actors: 0
node/0 worker/5 SchedulingQueue... completed report !


Arguably, actor 1477943366 should be classified in a higher priority than NORMAL (such as HIGH or MAX). But is it required ? I think, at least in this case, that the answer is no. Here is the reason.


The only thing that counts at the end of the day is that you want to waste CPU cycles. As long as CPU cycles are not wasted (called efficiency), the order (that's a partial order right there) of events is unimportant as long as no worker starve (remember, wasting CPU cycles is like wasting money: it's bad.).

Below are the load values across the actor system for an actor computation that lasted  17 minutes and that had an efficiency of 94% (the computation wasted around 6% of the CPU cycles, which is not bad, but not perfect neither).

At the beginning, there is some I/O, which waits for the magnetic disk. So CPU cycles are wasted.

Then, actors flow their messages at full capacity which is shown with a load between 0.99 and 1.00.

Then at the end the load drops a little because of work scarcity.

The first step is the data counting. In this computation, there were only one data file. So only one worker is busy from 0 seconds to 45 seconds.

LOAD EPOCH 0 s node/0 0.00/31 (0.00) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
LOAD EPOCH 5 s node/0 0.05/31 (0.00) 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
LOAD EPOCH 10 s node/0 0.96/31 (0.03) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.96 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
LOAD EPOCH 15 s node/0 0.98/31 (0.03) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
LOAD EPOCH 20 s node/0 0.98/31 (0.03) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
LOAD EPOCH 25 s node/0 0.98/31 (0.03) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
LOAD EPOCH 30 s node/0 0.98/31 (0.03) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
LOAD EPOCH 35 s node/0 0.98/31 (0.03) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
LOAD EPOCH 40 s node/0 0.98/31 (0.03) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
LOAD EPOCH 45 s node/0 3.79/31 (0.12) 0.00 0.00 0.00 0.00 0.22 0.22 0.22 0.23 0.24 0.22 0.23 0.00 0.00 0.22 0.98 0.00 0.00 0.00 0.00 0.00 0.26 0.00 0.26 0.27 0.00 0.00 0.00 0.00 0.00 0.22 0.00


Data distribution from input_stream actors to the sequence_store actors happens between 50 seconds and 60 seconds.

LOAD EPOCH 50 s node/0 11.04/31 (0.36) 0.73 0.74 0.77 0.21 0.22 0.22 0.22 0.23 0.24 0.22 0.23 0.74 0.21 0.73 0.73 0.27 0.26 0.26 0.27 0.26 0.26 0.27 0.26 0.27 0.26 0.26 0.26 0.27 0.71 0.22 0.22
LOAD EPOCH 55 s node/0 24.96/31 (0.81) 0.73 0.74 0.77 0.73 0.73 0.75 0.74 0.75 0.76 0.73 0.75 0.72 0.67 0.73 0.73 0.93 0.91 0.92 0.91 0.91 0.91 0.92 0.92 0.93 0.91 0.91 0.90 0.76 0.71 0.74 0.74
LOAD EPOCH 60 s node/0 25.65/31 (0.83) 0.71 0.71 0.72 0.71 0.72 0.72 0.71 1.00 0.72 0.72 0.73 0.72 0.71 0.72 0.72 1.00 0.96 0.98 0.98 0.98 0.96 0.98 0.98 1.00 0.95 0.99 0.96 0.75 0.71 0.72 0.72


From 65 seconds to 950 seconds (most of the computation), the load of every worker reported by the Thorium runtime system is 99% or 100%. This is good enough.

LOAD EPOCH 65 s node/0 30.87/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00
LOAD EPOCH 70 s node/0 30.90/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 0.99 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 75 s node/0 30.84/31 (0.99) 0.99 1.00 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 1.00 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 80 s node/0 30.91/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 85 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 90 s node/0 30.84/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 95 s node/0 30.85/31 (1.00) 0.99 1.00 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 1.00 0.99 1.00 1.00 1.00 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 100 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 105 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 110 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 115 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 120 s node/0 30.86/31 (1.00) 1.00 1.00 1.00 1.00 0.99 1.00 0.99 1.00 0.99 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 0.99 1.00 0.99 1.00 1.00
LOAD EPOCH 125 s node/0 30.81/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 130 s node/0 30.83/31 (0.99) 0.99 0.99 1.00 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 135 s node/0 30.86/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00
LOAD EPOCH 140 s node/0 30.91/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 145 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 150 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 155 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 160 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 165 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 170 s node/0 30.94/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 175 s node/0 30.94/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 180 s node/0 30.87/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 0.99 0.95 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 0.99 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 185 s node/0 30.77/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.95 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 190 s node/0 30.80/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 195 s node/0 30.81/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 200 s node/0 30.82/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 205 s node/0 30.83/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 210 s node/0 30.84/31 (0.99) 0.99 1.00 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 215 s node/0 30.86/31 (1.00) 1.00 1.00 0.99 1.00 0.99 1.00 0.99 1.00 1.00 1.00 1.00 0.99 1.00 1.00 0.99 1.00 0.99 1.00 1.00 0.99 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 0.99
LOAD EPOCH 220 s node/0 30.88/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 225 s node/0 30.91/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 230 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 235 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 240 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 245 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 250 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 255 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 260 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 265 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 270 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 275 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 280 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 285 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 290 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 295 s node/0 30.94/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 300 s node/0 30.94/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 305 s node/0 30.94/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 310 s node/0 30.61/31 (0.99) 1.00 0.88 0.99 0.99 0.94 1.00 1.00 1.00 0.98 1.00 0.99 1.00 1.00 1.00 1.00 0.99 1.00 0.99 1.00 1.00 0.99 0.99 0.99 1.00 0.99 1.00 0.93 1.00 1.00 1.00 1.00
LOAD EPOCH 315 s node/0 30.76/31 (0.99) 0.99 0.99 0.99 0.99 0.98 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.98 1.00 0.99 0.99 0.99
LOAD EPOCH 320 s node/0 30.79/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 325 s node/0 30.80/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 330 s node/0 30.80/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 335 s node/0 30.80/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 340 s node/0 30.81/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 345 s node/0 30.81/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 350 s node/0 30.82/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 355 s node/0 30.83/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 360 s node/0 30.81/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 0.97 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 365 s node/0 30.81/31 (0.99) 0.99 1.00 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 0.98 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.98 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 370 s node/0 30.80/31 (0.99) 0.99 1.00 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.98 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.98 0.99 0.99 0.99
LOAD EPOCH 375 s node/0 30.80/31 (0.99) 0.99 1.00 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 1.00 0.99 0.99 1.00 0.99 1.00 0.99 1.00 0.98 0.99 0.99 1.00 0.99 0.99 1.00 1.00 0.99 0.98 0.99 0.99 0.99
LOAD EPOCH 380 s node/0 30.83/31 (0.99) 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 0.99 0.99 1.00 1.00 0.99 1.00 1.00 1.00 0.99 0.99 1.00 1.00
LOAD EPOCH 385 s node/0 30.86/31 (1.00) 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 390 s node/0 30.88/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 395 s node/0 30.89/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 400 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 405 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 410 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 415 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 420 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 425 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 430 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 435 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 440 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 445 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 450 s node/0 30.89/31 (1.00) 1.00 1.00 1.00 1.00 1.00 0.98 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.98 1.00 1.00 1.00
LOAD EPOCH 455 s node/0 30.86/31 (1.00) 1.00 1.00 1.00 1.00 1.00 0.97 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.98 1.00 1.00 1.00 0.97 1.00 1.00 1.00
LOAD EPOCH 460 s node/0 30.86/31 (1.00) 1.00 1.00 1.00 1.00 1.00 0.98 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.98 1.00 1.00 1.00 0.97 1.00 1.00 1.00
LOAD EPOCH 465 s node/0 30.85/31 (1.00) 1.00 1.00 1.00 1.00 1.00 0.97 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.98 1.00 1.00 1.00 0.97 1.00 1.00 1.00
LOAD EPOCH 470 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 475 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 480 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 485 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 490 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 495 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 500 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 505 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 510 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 515 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 520 s node/0 30.94/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 525 s node/0 30.94/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 530 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 535 s node/0 30.94/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 540 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 545 s node/0 30.94/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 550 s node/0 30.94/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 555 s node/0 30.94/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 560 s node/0 30.40/31 (0.98) 1.00 1.00 1.00 1.00 1.00 0.92 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.55 1.00 1.00 0.98 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 565 s node/0 30.02/31 (0.97) 1.00 1.00 1.00 1.00 1.00 0.92 1.00 0.65 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.55 1.00 1.00 0.98 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.96 1.00 1.00 1.00
LOAD EPOCH 570 s node/0 27.44/31 (0.89) 1.00 0.81 0.98 1.00 0.88 0.56 1.00 0.65 1.00 0.98 0.56 1.00 1.00 1.00 1.00 0.56 0.99 1.00 0.56 1.00 0.99 0.81 0.86 0.99 0.85 1.00 0.91 0.56 1.00 1.00 1.00
LOAD EPOCH 575 s node/0 30.71/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.98 0.99 0.99 0.99 0.99
LOAD EPOCH 580 s node/0 30.79/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 585 s node/0 30.79/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 590 s node/0 30.79/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 595 s node/0 30.79/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 600 s node/0 30.79/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 605 s node/0 30.79/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 610 s node/0 30.79/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 615 s node/0 30.79/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 620 s node/0 30.79/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 625 s node/0 30.80/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 630 s node/0 30.80/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 635 s node/0 30.80/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 640 s node/0 30.81/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 645 s node/0 30.81/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 650 s node/0 30.81/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 655 s node/0 30.81/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 660 s node/0 30.81/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 665 s node/0 30.82/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 670 s node/0 30.82/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 675 s node/0 30.82/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 680 s node/0 30.83/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 685 s node/0 30.83/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 690 s node/0 30.83/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 695 s node/0 30.84/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 700 s node/0 30.84/31 (0.99) 0.99 0.99 0.99 0.99 0.99 1.00 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 1.00 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 705 s node/0 30.85/31 (1.00) 0.99 1.00 0.99 0.99 0.99 1.00 0.99 1.00 0.99 1.00 1.00 0.99 0.99 0.99 0.99 1.00 0.99 1.00 1.00 0.99 0.99 1.00 0.99 1.00 0.99 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 710 s node/0 30.85/31 (1.00) 0.99 1.00 0.99 0.99 0.99 1.00 0.99 1.00 0.99 1.00 1.00 0.99 0.99 0.99 0.99 1.00 0.99 1.00 1.00 0.99 0.99 1.00 0.99 1.00 1.00 0.99 0.99 1.00 0.99 0.99 0.99
LOAD EPOCH 715 s node/0 30.85/31 (1.00) 1.00 1.00 0.99 1.00 0.99 1.00 0.99 1.00 0.99 1.00 1.00 1.00 0.99 1.00 0.99 1.00 1.00 1.00 1.00 0.99 0.99 1.00 0.99 1.00 1.00 1.00 1.00 1.00 0.99 1.00 0.99
LOAD EPOCH 720 s node/0 30.86/31 (1.00) 1.00 1.00 1.00 1.00 0.99 1.00 0.99 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 0.99 1.00 1.00 0.99 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00
LOAD EPOCH 725 s node/0 30.86/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00
LOAD EPOCH 730 s node/0 30.87/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 735 s node/0 30.87/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 740 s node/0 30.88/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 745 s node/0 30.88/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 750 s node/0 30.89/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 755 s node/0 30.89/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 760 s node/0 30.91/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 765 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 770 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 775 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 780 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 785 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 790 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 795 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 800 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 805 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 810 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 815 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 820 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 825 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 830 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 835 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 840 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 845 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 850 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 855 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 860 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 865 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 870 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 875 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 880 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 885 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 890 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 895 s node/0 30.92/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 900 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 905 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 910 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 915 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 920 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 925 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 930 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 935 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 940 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 945 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
LOAD EPOCH 950 s node/0 30.93/31 (1.00) 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00







Starting at 955 seconds, there is the fiinal phase which as a variable load across the system.


LOAD EPOCH 955 s node/0 28.11/31 (0.91) 1.00 0.54 0.99 1.00 0.82 1.00 1.00 1.00 0.81 0.88 1.00 0.84 1.00 1.00 1.00 1.00 0.90 0.81 1.00 1.00 1.00 0.71 0.77 1.00 0.75 0.92 0.72 1.00 1.00 0.96 0.72
LOAD EPOCH 960 s node/0 4.86/31 (0.16) 0.22 0.00 0.00 0.12 0.00 0.14 0.99 0.21 0.00 0.00 0.30 0.00 0.22 0.07 0.38 1.00 0.00 0.00 0.17 0.29 0.02 0.00 0.00 0.18 0.00 0.00 0.00 0.21 0.35 0.00 0.00
LOAD EPOCH 965 s node/0 4.86/31 (0.16) 0.22 0.00 0.00 0.12 0.00 0.14 0.99 0.21 0.00 0.00 0.30 0.00 0.22 0.07 0.38 1.00 0.00 0.00 0.17 0.29 0.02 0.00 0.00 0.18 0.00 0.00 0.00 0.21 0.35 0.00 0.00
LOAD EPOCH 970 s node/0 4.86/31 (0.16) 0.22 0.00 0.00 0.12 0.00 0.14 0.99 0.21 0.00 0.00 0.30 0.00 0.22 0.07 0.38 1.00 0.00 0.00 0.17 0.29 0.02 0.00 0.00 0.18 0.00 0.00 0.00 0.21 0.35 0.00 0.00
LOAD EPOCH 975 s node/0 4.86/31 (0.16) 0.22 0.00 0.00 0.12 0.00 0.14 0.99 0.21 0.00 0.00 0.30 0.00 0.22 0.07 0.38 1.00 0.00 0.00 0.17 0.29 0.02 0.00 0.00 0.18 0.00 0.00 0.00 0.21 0.35 0.00 0.00
LOAD EPOCH 980 s node/0 4.86/31 (0.16) 0.22 0.00 0.00 0.12 0.00 0.14 0.99 0.21 0.00 0.00 0.30 0.00 0.22 0.07 0.38 1.00 0.00 0.00 0.17 0.29 0.02 0.00 0.00 0.18 0.00 0.00 0.00 0.21 0.35 0.00 0.00
LOAD EPOCH 985 s node/0 4.86/31 (0.16) 0.22 0.00 0.00 0.12 0.00 0.14 0.99 0.21 0.00 0.00 0.30 0.00 0.22 0.07 0.38 1.00 0.00 0.00 0.17 0.29 0.02 0.00 0.00 0.18 0.00 0.00 0.00 0.21 0.35 0.00 0.00
LOAD EPOCH 990 s node/0 4.86/31 (0.16) 0.22 0.00 0.00 0.12 0.00 0.14 0.99 0.21 0.00 0.00 0.30 0.00 0.22 0.07 0.38 1.00 0.00 0.00 0.17 0.29 0.02 0.00 0.00 0.18 0.00 0.00 0.00 0.21 0.35 0.00 0.00
LOAD EPOCH 995 s node/0 26.64/31 (0.86) 0.95 0.95 0.95 0.95 0.95 0.95 0.99 0.21 0.95 0.95 0.95 0.95 0.95 0.95 0.95 1.00 0.95 0.00 0.95 0.29 0.95 0.95 0.95 0.95 0.95 0.95 0.95 0.95 0.35 0.95 0.95
LOAD EPOCH 1000 s node/0 30.53/31 (0.98) 0.99 0.99 0.99 0.99 0.99 0.99 0.95 0.95 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.95 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.95 0.99 0.99
LOAD EPOCH 1005 s node/0 30.68/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 1010 s node/0 30.68/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 1015 s node/0 30.68/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 1020 s node/0 30.68/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 1025 s node/0 30.68/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 1030 s node/0 30.68/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 1035 s node/0 30.70/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 1040 s node/0 30.71/31 (0.99) 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99
LOAD EPOCH 1045 s node/0 28.28/31 (0.91) 0.99 0.99 0.92 0.99 0.99 0.99 0.99 0.95 0.54 0.99 0.99 0.99 0.99 0.93 0.99 0.99 0.91 0.99 0.99 0.99 0.48 0.88 0.99 0.85 0.99 0.99 0.99 0.40 0.99 0.63 0.99
LOAD EPOCH 1050 s node/0 12.11/31 (0.39) 0.43 0.36 0.00 0.30 0.28 0.92 0.18 0.00 0.00 0.99 0.84 0.99 0.55 0.00 0.17 0.85 0.00 0.85 0.72 0.99 0.00 0.00 0.78 0.00 0.74 0.11 0.85 0.00 0.13 0.00 0.09
LOAD LOOP 1052 s node/0 29.20/31 (0.94) 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.98 0.95 0.94 0.94 0.94 0.95 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94


The overall load reported by Thorium appears below.


LOAD LOOP 1052 s node/0 29.20/31 (0.94) 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.98 0.95 0.94 0.94 0.94 0.95 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94
node 0 efficiency: 0.94


This means that 6% of the CPU cycles went into the garbage bin and were not used. Usually, this is caused by unavailable operands. In an actor computation, unavailable operands are happening when a worker has 0 actors scheduled in its priority scheduling queue, which means that none of its actors have any message in their message inboxes.


Obviously, actors are cool. And BIOSAL will bring this coolness to genomics, at scale.

2014-06-08

The actor model of computation



This video has Professor Carl Hewitt in it. Hewitt is the original creator of the concept of actors.

Hewitt created the concept of actors in 1973 with Bishop, and Steiger.
Everything is in this paper and it's 11 page.


If you have little time, you can read only the bottom of page 12 of Gul Agha's PhD thesis.
http://dspace.mit.edu/bitstream/handle/1721.1/6952/AITR-844.pdf  page 12, section "2.1.3 Actors"


If you have more time:


Baker was a PhD student with Hewitt. He and Hewitt wrote the "laws of actors" in 1977 (see below).
Agha was also a PhD student with Hewitt and he reformulated the concepts in his thesis.

After that, Joe Armstrong (at Ericsson) advanced the subject (he also did his PhD on the topic).

Anyway, below are the 4 most important pieces of work in the literature (according to me).


Hewitt, Bishop, Steiger 1973    
Hewitt, C., Bishop, P. & Steiger, R. A universal modular ACTOR formalism for artificial intelligence. In Proceedings of the 3rd international joint conference on Artificial intelligence, IJCAI'73, 235-245 (Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1973). URL http://portal.acm.org/citation.cfm?id=1624804.    
http://ijcai.org/Past%20Proceedings/IJCAI-73/PDF/027B.pdf

Introduced concepts: actor, message, script, create actors, spawn actors

 
Hewitt & Baker 1977    
Hewitt, C. & Baker, H. Laws for communicating parallel processes. Tech. Rep., MIT Artificial Intelligence Laboratory (1977). URL http://dspace.mit.edu/handle/1721.1/41962.    
http://dspace.mit.edu/bitstream/handle/1721.1/41962/AI_WP_134A.pdf

Contains a lot of useful information for readers. In particular this paper underlines the importance of acquaintances.


Agha 1986
Agha, G. Actors: A Model of Concurrent Computation in Distributed Systems (MIT Press, Cambridge, MA, USA, 1986). URL http://dl.acm.org/citation.cfm?id=7929.    
http://dspace.mit.edu/bitstream/handle/1721.1/6952/AITR-844.pdf

On page 12 (2.1.3), the actor model is formally specified.


Armstrong 2003    
Armstrong, J. Making reliable distributed systems in the presence of software errors (2003). URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.3.408.    
http://www.erlang.org/download/armstrong_thesis_2003.pdf

Erlang is really a neat language.


Early prototype of our work with actors for genomics

Our work on biosal (bsal) is advancing well.

Here is an example of what you can do easily with the actor model.
This actor machine emulator contains 4 biosal nodes, each with 24 threads and 23 workers (23 worker threads and 1 communication thread). The actor machine executes actors, and the behavior of actor is specified with scripts (in C 1999).

The command I used on the supercomputer to launch the app:

# 4 physical computers, with a fast interconnect
# 4 bsal nodes
# 24 threads per bsal node

#       -d depth       Depth of each processor (number of threads)
#       -N pes         PEs per node
#       -n width       Number of processors needed

aprun -n 4 -N 1 -d 24 biosal/example_controller -threads-per-node 24 datasets/Iowa_Continuous_Corn/*


This is the standard output. This actor computation ran in 3 minutes, 39 seconds. The nice thing about running an actor computation is that the actor machine can collect easily a lot of metrics. These metrics are displayed at the end of the log.


 ############################# Beagle Job Start #############################
 #                  Job ID: 2716005  Project: CI-###########                  #
 #                 Start time: Sat Jun  7 23:00:28 CDT 2014                 #
 #                       Resources: walltime=04:00:00                       #
 ############################################################################

----------------------------------------------
biosal> node/0: 24 threads, 23 workers
actor 800941600 spawned controller 402725412
actor 800941600, starting controller 402725412
DEBUG bsal_input_controller_prepare_spawners
DEBUG bsal_input_controller_prepare_spawners
DEBUG bsal_input_controller_prepare_spawners
DEBUG bsal_input_controller_prepare_spawners
DEBUG bsal_input_controller_prepare_spawners
DEBUG all spawners are prepared
actor 800941600 add file -threads-per-node to actor 402725412
actor 800941600 add file 24 to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/foo to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/foo~ to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/foo.out to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.649-4.899.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.868-1.1053.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.868-2.1053.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.868-3.1053.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.868-4.1053.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.868-5.1053.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.868-6.1053.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.895-5.1070.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.895-6.1070.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/GPIC.895-7.1070.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/HTHO.1209-1.1231.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/HTHO.1209-2.1231.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/HTHO.1300-6.1364.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/HTHO.1429-2.1388.fastq to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/index.html to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/index.html?C=D;O=A to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/index.html?C=D;O=D to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/index.html?C=M;O=A to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/index.html?C=M;O=D to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/index.html?C=N;O=A to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/index.html?C=N;O=D to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/index.html?C=S;O=A to actor 402725412
actor 800941600 add file datasets/Iowa_Continuous_Corn/index.html?C=S;O=D to actor 402725412
actor 800941600 no more files to add
actor 800941600 asks actor 402725412 to distribute data
DEBUG actor 402725412 receives stream 1392161576 from spawner 800941600 for file -threads-per-node
----------------------------------------------
biosal> node/2: 24 threads, 23 workers
DEBUG actor 402725412 receives stream 1162079149 from spawner 1163079153 for file 24
DEBUG actor 402725412 receives stream 34225682 from spawner 139172306 for file datasets/Iowa_Continuous_Corn/foo
DEBUG actor 402725412 receives stream 1399876531 from spawner 1114128863 for file datasets/Iowa_Continuous_Corn/foo~
DEBUG actor 402725412 receives stream 148429560 from spawner 800941600 for file datasets/Iowa_Continuous_Corn/foo.out
----------------------------------------------
biosal> node/3: 24 threads, 23 workers
----------------------------------------------
biosal> node/1: 24 threads, 23 workers
DEBUG actor 402725412 receives stream 680978333 from spawner 1163079153 for file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq
DEBUG actor 402725412 receives stream 649752862 from spawner 139172306 for file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq
DEBUG actor 402725412 receives stream 122779715 from spawner 1114128863 for file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq
DEBUG actor 402725412 receives stream 1212714640 from spawner 800941600 for file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq
DEBUG actor 402725412 receives stream 1830160613 from spawner 1163079153 for file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq
DEBUG actor 402725412 receives stream 1144056826 from spawner 139172306 for file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq
DEBUG actor 402725412 receives stream 1288734375 from spawner 1114128863 for file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq
DEBUG actor 402725412 receives stream 547646904 from spawner 800941600 for file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq
DEBUG actor 402725412 receives stream 1630775601 from spawner 1163079153 for file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq
DEBUG actor 402725412 receives stream 602765326 from spawner 139172306 for file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq
DEBUG actor 402725412 receives stream 1882977711 from spawner 1114128863 for file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq
DEBUG actor 402725412 receives stream 721954240 from spawner 800941600 for file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq
DEBUG actor 402725412 receives stream 261732057 from spawner 1163079153 for file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq
DEBUG actor 402725412 receives stream 675296634 from spawner 139172306 for file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq
DEBUG actor 402725412 receives stream 468223423 from spawner 1114128863 for file datasets/Iowa_Continuous_Corn/GPIC.649-4.899.fastq
DEBUG actor 402725412 receives stream 94072032 from spawner 800941600 for file datasets/Iowa_Continuous_Corn/GPIC.868-1.1053.fastq
DEBUG actor 402725412 receives stream 609867641 from spawner 1163079153 for file datasets/Iowa_Continuous_Corn/GPIC.868-2.1053.fastq
DEBUG actor 402725412 receives stream 269542554 from spawner 139172306 for file datasets/Iowa_Continuous_Corn/GPIC.868-3.1053.fastq
DEBUG actor 402725412 receives stream 780701627 from spawner 1114128863 for file datasets/Iowa_Continuous_Corn/GPIC.868-4.1053.fastq
DEBUG actor 402725412 receives stream 920849820 from spawner 800941600 for file datasets/Iowa_Continuous_Corn/GPIC.868-5.1053.fastq
DEBUG actor 402725412 receives stream 572359065 from spawner 1163079153 for file datasets/Iowa_Continuous_Corn/GPIC.868-6.1053.fastq
DEBUG actor 402725412 receives stream 257301510 from spawner 139172306 for file datasets/Iowa_Continuous_Corn/GPIC.895-5.1070.fastq
DEBUG actor 402725412 receives stream 1823817815 from spawner 1114128863 for file datasets/Iowa_Continuous_Corn/GPIC.895-6.1070.fastq
DEBUG actor 402725412 receives stream 70674448 from spawner 800941600 for file datasets/Iowa_Continuous_Corn/GPIC.895-7.1070.fastq
DEBUG actor 402725412 receives stream 1774246797 from spawner 1163079153 for file datasets/Iowa_Continuous_Corn/HTHO.1209-1.1231.fastq
DEBUG actor 402725412 receives stream 1044554942 from spawner 139172306 for file datasets/Iowa_Continuous_Corn/HTHO.1209-2.1231.fastq
DEBUG actor 402725412 receives stream 696392135 from spawner 1114128863 for file datasets/Iowa_Continuous_Corn/HTHO.1300-6.1364.fastq
DEBUG actor 402725412 receives stream 363590824 from spawner 800941600 for file datasets/Iowa_Continuous_Corn/HTHO.1429-2.1388.fastq
DEBUG actor 402725412 receives stream 454036081 from spawner 1163079153 for file datasets/Iowa_Continuous_Corn/index.html
DEBUG actor 402725412 receives stream 746596374 from spawner 139172306 for file datasets/Iowa_Continuous_Corn/index.html?C=D;O=A
DEBUG actor 402725412 receives stream 716706071 from spawner 1114128863 for file datasets/Iowa_Continuous_Corn/index.html?C=D;O=D
DEBUG actor 402725412 receives stream 575385800 from spawner 800941600 for file datasets/Iowa_Continuous_Corn/index.html?C=M;O=A
DEBUG actor 402725412 receives stream 1079327317 from spawner 1163079153 for file datasets/Iowa_Continuous_Corn/index.html?C=M;O=D
DEBUG actor 402725412 receives stream 503721090 from spawner 139172306 for file datasets/Iowa_Continuous_Corn/index.html?C=N;O=A
DEBUG actor 402725412 receives stream 218874463 from spawner 1114128863 for file datasets/Iowa_Continuous_Corn/index.html?C=N;O=D
DEBUG actor 402725412 receives stream 1718717096 from spawner 800941600 for file datasets/Iowa_Continuous_Corn/index.html?C=S;O=A
DEBUG actor 402725412 receives stream 1847021925 from spawner 1163079153 for file datasets/Iowa_Continuous_Corn/index.html?C=S;O=D
DEBUG actor:402725412 receives from actor:602765326, file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:649752862, file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:675296634, file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq, 10002000 entries so far
DEBUG actor:402725412 receives from actor:1144056826, file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:269542554, file datasets/Iowa_Continuous_Corn/GPIC.868-3.1053.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:257301510, file datasets/Iowa_Continuous_Corn/GPIC.895-5.1070.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:1044554942, file datasets/Iowa_Continuous_Corn/HTHO.1209-2.1231.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:122779715, file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:1288734375, file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:1882977711, file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:780701627, file datasets/Iowa_Continuous_Corn/GPIC.868-4.1053.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:1823817815, file datasets/Iowa_Continuous_Corn/GPIC.895-6.1070.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:468223423, file datasets/Iowa_Continuous_Corn/GPIC.649-4.899.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:696392135, file datasets/Iowa_Continuous_Corn/HTHO.1300-6.1364.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:1830160613, file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:680978333, file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:261732057, file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:1630775601, file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:609867641, file datasets/Iowa_Continuous_Corn/GPIC.868-2.1053.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:1774246797, file datasets/Iowa_Continuous_Corn/HTHO.1209-1.1231.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:572359065, file datasets/Iowa_Continuous_Corn/GPIC.868-6.1053.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:721954240, file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:1212714640, file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:547646904, file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:920849820, file datasets/Iowa_Continuous_Corn/GPIC.868-5.1053.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:94072032, file datasets/Iowa_Continuous_Corn/GPIC.868-1.1053.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:70674448, file datasets/Iowa_Continuous_Corn/GPIC.895-7.1070.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:363590824, file datasets/Iowa_Continuous_Corn/HTHO.1429-2.1388.fastq, 10001000 entries so far
DEBUG actor:402725412 receives from actor:1288734375, file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:122779715, file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:1882977711, file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:1823817815, file datasets/Iowa_Continuous_Corn/GPIC.895-6.1070.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:780701627, file datasets/Iowa_Continuous_Corn/GPIC.868-4.1053.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:468223423, file datasets/Iowa_Continuous_Corn/GPIC.649-4.899.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:696392135, file datasets/Iowa_Continuous_Corn/HTHO.1300-6.1364.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:602765326, file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:649752862, file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:1144056826, file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:269542554, file datasets/Iowa_Continuous_Corn/GPIC.868-3.1053.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:675296634, file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq, 20004000 entries so far
DEBUG actor:402725412 receives from actor:257301510, file datasets/Iowa_Continuous_Corn/GPIC.895-5.1070.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:1044554942, file datasets/Iowa_Continuous_Corn/HTHO.1209-2.1231.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:261732057, file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:680978333, file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq, 20003000 entries so far
DEBUG actor:402725412 receives from actor:1830160613, file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:1630775601, file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:609867641, file datasets/Iowa_Continuous_Corn/GPIC.868-2.1053.fastq, 20003000 entries so far
DEBUG actor:402725412 receives from actor:1774246797, file datasets/Iowa_Continuous_Corn/HTHO.1209-1.1231.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:572359065, file datasets/Iowa_Continuous_Corn/GPIC.868-6.1053.fastq, 20003000 entries so far
DEBUG actor:402725412 receives from actor:1212714640, file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:721954240, file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:547646904, file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:920849820, file datasets/Iowa_Continuous_Corn/GPIC.868-5.1053.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:94072032, file datasets/Iowa_Continuous_Corn/GPIC.868-1.1053.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:70674448, file datasets/Iowa_Continuous_Corn/GPIC.895-7.1070.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:363590824, file datasets/Iowa_Continuous_Corn/HTHO.1429-2.1388.fastq, 20002000 entries so far
DEBUG actor:402725412 receives from actor:602765326, file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:649752862, file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:269542554, file datasets/Iowa_Continuous_Corn/GPIC.868-3.1053.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:675296634, file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq, 30005000 entries so far
DEBUG actor:402725412 receives from actor:257301510, file datasets/Iowa_Continuous_Corn/GPIC.895-5.1070.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:1144056826, file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:1044554942, file datasets/Iowa_Continuous_Corn/HTHO.1209-2.1231.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:122779715, file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq, 30004000 entries so far
DEBUG actor:402725412 receives from actor:1882977711, file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq, 30004000 entries so far
DEBUG actor:402725412 receives from actor:1288734375, file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:780701627, file datasets/Iowa_Continuous_Corn/GPIC.868-4.1053.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:1823817815, file datasets/Iowa_Continuous_Corn/GPIC.895-6.1070.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:468223423, file datasets/Iowa_Continuous_Corn/GPIC.649-4.899.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:696392135, file datasets/Iowa_Continuous_Corn/HTHO.1300-6.1364.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:547646904, file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:1212714640, file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:721954240, file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:920849820, file datasets/Iowa_Continuous_Corn/GPIC.868-5.1053.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:94072032, file datasets/Iowa_Continuous_Corn/GPIC.868-1.1053.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:70674448, file datasets/Iowa_Continuous_Corn/GPIC.895-7.1070.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:363590824, file datasets/Iowa_Continuous_Corn/HTHO.1429-2.1388.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:1830160613, file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:1630775601, file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:261732057, file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:680978333, file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq, 30004000 entries so far
DEBUG actor:402725412 receives from actor:609867641, file datasets/Iowa_Continuous_Corn/GPIC.868-2.1053.fastq, 30005000 entries so far
DEBUG actor:402725412 receives from actor:1774246797, file datasets/Iowa_Continuous_Corn/HTHO.1209-1.1231.fastq, 30003000 entries so far
DEBUG actor:402725412 receives from actor:572359065, file datasets/Iowa_Continuous_Corn/GPIC.868-6.1053.fastq, 30004000 entries so far
DEBUG actor:402725412 receives from actor:602765326, file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:649752862, file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq, 40005000 entries so far
DEBUG actor:402725412 receives from actor:269542554, file datasets/Iowa_Continuous_Corn/GPIC.868-3.1053.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:257301510, file datasets/Iowa_Continuous_Corn/GPIC.895-5.1070.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:675296634, file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq, 40006000 entries so far
DEBUG actor:402725412 receives from actor:1144056826, file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq, 40005000 entries so far
DEBUG actor:402725412 receives from actor:1044554942, file datasets/Iowa_Continuous_Corn/HTHO.1209-2.1231.fastq, 40004000 entries so far
DEBUG Actor 402725412 received from actor 696392135 for file datasets/Iowa_Continuous_Corn/HTHO.1300-6.1364.fastq: 39704380 entries
DEBUG actor:402725412 receives from actor:1882977711, file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq, 40005000 entries so far
DEBUG actor:402725412 receives from actor:1288734375, file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:122779715, file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq, 40005000 entries so far
DEBUG actor:402725412 receives from actor:780701627, file datasets/Iowa_Continuous_Corn/GPIC.868-4.1053.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:1823817815, file datasets/Iowa_Continuous_Corn/GPIC.895-6.1070.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:468223423, file datasets/Iowa_Continuous_Corn/GPIC.649-4.899.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:261732057, file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:680978333, file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq, 40005000 entries so far
DEBUG actor:402725412 receives from actor:1830160613, file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:1630775601, file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:609867641, file datasets/Iowa_Continuous_Corn/GPIC.868-2.1053.fastq, 40006000 entries so far
DEBUG actor:402725412 receives from actor:1774246797, file datasets/Iowa_Continuous_Corn/HTHO.1209-1.1231.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:572359065, file datasets/Iowa_Continuous_Corn/GPIC.868-6.1053.fastq, 40005000 entries so far
DEBUG actor:402725412 receives from actor:1212714640, file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:721954240, file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:547646904, file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:920849820, file datasets/Iowa_Continuous_Corn/GPIC.868-5.1053.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:94072032, file datasets/Iowa_Continuous_Corn/GPIC.868-1.1053.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:70674448, file datasets/Iowa_Continuous_Corn/GPIC.895-7.1070.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:363590824, file datasets/Iowa_Continuous_Corn/HTHO.1429-2.1388.fastq, 40004000 entries so far
DEBUG actor:402725412 receives from actor:602765326, file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:269542554, file datasets/Iowa_Continuous_Corn/GPIC.868-3.1053.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:649752862, file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq, 50006000 entries so far
DEBUG actor:402725412 receives from actor:257301510, file datasets/Iowa_Continuous_Corn/GPIC.895-5.1070.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:675296634, file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq, 50007000 entries so far
DEBUG actor:402725412 receives from actor:1144056826, file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq, 50007000 entries so far
DEBUG actor:402725412 receives from actor:1044554942, file datasets/Iowa_Continuous_Corn/HTHO.1209-2.1231.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:122779715, file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq, 50006000 entries so far
DEBUG actor:402725412 receives from actor:1882977711, file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq, 50006000 entries so far
DEBUG actor:402725412 receives from actor:1288734375, file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:780701627, file datasets/Iowa_Continuous_Corn/GPIC.868-4.1053.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:1823817815, file datasets/Iowa_Continuous_Corn/GPIC.895-6.1070.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:468223423, file datasets/Iowa_Continuous_Corn/GPIC.649-4.899.fastq, 50005000 entries so far
DEBUG Actor 402725412 received from actor 1774246797 for file datasets/Iowa_Continuous_Corn/HTHO.1209-1.1231.fastq: 47646564 entries
DEBUG Actor 402725412 received from actor 1044554942 for file datasets/Iowa_Continuous_Corn/HTHO.1209-2.1231.fastq: 51802070 entries
DEBUG actor:402725412 receives from actor:1830160613, file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:1630775601, file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:261732057, file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:680978333, file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq, 50006000 entries so far
DEBUG actor:402725412 receives from actor:609867641, file datasets/Iowa_Continuous_Corn/GPIC.868-2.1053.fastq, 50007000 entries so far
DEBUG actor:402725412 receives from actor:572359065, file datasets/Iowa_Continuous_Corn/GPIC.868-6.1053.fastq, 50007000 entries so far
DEBUG Actor 402725412 received from actor 468223423 for file datasets/Iowa_Continuous_Corn/GPIC.649-4.899.fastq: 54662170 entries
DEBUG actor:402725412 receives from actor:721954240, file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:547646904, file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:1212714640, file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:94072032, file datasets/Iowa_Continuous_Corn/GPIC.868-1.1053.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:920849820, file datasets/Iowa_Continuous_Corn/GPIC.868-5.1053.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:70674448, file datasets/Iowa_Continuous_Corn/GPIC.895-7.1070.fastq, 50005000 entries so far
DEBUG actor:402725412 receives from actor:363590824, file datasets/Iowa_Continuous_Corn/HTHO.1429-2.1388.fastq, 50005000 entries so far
DEBUG Actor 402725412 received from actor 257301510 for file datasets/Iowa_Continuous_Corn/GPIC.895-5.1070.fastq: 56963934 entries
DEBUG Actor 402725412 received from actor 1823817815 for file datasets/Iowa_Continuous_Corn/GPIC.895-6.1070.fastq: 56909532 entries
DEBUG actor:402725412 receives from actor:602765326, file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq, 60006000 entries so far
DEBUG actor:402725412 receives from actor:649752862, file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq, 60007000 entries so far
DEBUG actor:402725412 receives from actor:269542554, file datasets/Iowa_Continuous_Corn/GPIC.868-3.1053.fastq, 60006000 entries so far
DEBUG actor:402725412 receives from actor:675296634, file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq, 60008000 entries so far
DEBUG actor:402725412 receives from actor:1144056826, file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq, 60008000 entries so far
DEBUG actor:402725412 receives from actor:122779715, file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq, 60007000 entries so far
DEBUG actor:402725412 receives from actor:1288734375, file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq, 60006000 entries so far
DEBUG actor:402725412 receives from actor:1882977711, file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq, 60007000 entries so far
DEBUG actor:402725412 receives from actor:780701627, file datasets/Iowa_Continuous_Corn/GPIC.868-4.1053.fastq, 60006000 entries so far
DEBUG Actor 402725412 received from actor 269542554 for file datasets/Iowa_Continuous_Corn/GPIC.868-3.1053.fastq: 62814332 entries
DEBUG Actor 402725412 received from actor 780701627 for file datasets/Iowa_Continuous_Corn/GPIC.868-4.1053.fastq: 62313864 entries
DEBUG Actor 402725412 received from actor 70674448 for file datasets/Iowa_Continuous_Corn/GPIC.895-7.1070.fastq: 57067668 entries
DEBUG actor:402725412 receives from actor:261732057, file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq, 60006000 entries so far
DEBUG actor:402725412 receives from actor:680978333, file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq, 60007000 entries so far
DEBUG actor:402725412 receives from actor:1630775601, file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq, 60006000 entries so far
DEBUG actor:402725412 receives from actor:1830160613, file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq, 60007000 entries so far
DEBUG actor:402725412 receives from actor:609867641, file datasets/Iowa_Continuous_Corn/GPIC.868-2.1053.fastq, 60008000 entries so far
DEBUG actor:402725412 receives from actor:572359065, file datasets/Iowa_Continuous_Corn/GPIC.868-6.1053.fastq, 60008000 entries so far
DEBUG actor:402725412 receives from actor:602765326, file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq, 70007000 entries so far
DEBUG actor:402725412 receives from actor:649752862, file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq, 70008000 entries so far
DEBUG actor:402725412 receives from actor:1144056826, file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq, 70009000 entries so far
DEBUG actor:402725412 receives from actor:675296634, file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq, 70009000 entries so far
DEBUG Actor 402725412 received from actor 572359065 for file datasets/Iowa_Continuous_Corn/GPIC.868-6.1053.fastq: 62568148 entries
DEBUG Actor 402725412 received from actor 609867641 for file datasets/Iowa_Continuous_Corn/GPIC.868-2.1053.fastq: 62794138 entries
DEBUG actor:402725412 receives from actor:721954240, file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq, 60006000 entries so far
DEBUG actor:402725412 receives from actor:1212714640, file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq, 60006000 entries so far
DEBUG actor:402725412 receives from actor:547646904, file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq, 60006000 entries so far
DEBUG actor:402725412 receives from actor:94072032, file datasets/Iowa_Continuous_Corn/GPIC.868-1.1053.fastq, 60006000 entries so far
DEBUG actor:402725412 receives from actor:920849820, file datasets/Iowa_Continuous_Corn/GPIC.868-5.1053.fastq, 60006000 entries so far
DEBUG actor:402725412 receives from actor:363590824, file datasets/Iowa_Continuous_Corn/HTHO.1429-2.1388.fastq, 60006000 entries so far
DEBUG actor:402725412 receives from actor:1288734375, file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq, 70007000 entries so far
DEBUG actor:402725412 receives from actor:122779715, file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq, 70008000 entries so far
DEBUG actor:402725412 receives from actor:1882977711, file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq, 70008000 entries so far
DEBUG Actor 402725412 received from actor 920849820 for file datasets/Iowa_Continuous_Corn/GPIC.868-5.1053.fastq: 62681112 entries
DEBUG Actor 402725412 received from actor 94072032 for file datasets/Iowa_Continuous_Corn/GPIC.868-1.1053.fastq: 63471812 entries
DEBUG actor:402725412 receives from actor:261732057, file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq, 70007000 entries so far
DEBUG actor:402725412 receives from actor:680978333, file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq, 70008000 entries so far
DEBUG actor:402725412 receives from actor:1630775601, file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq, 70007000 entries so far
DEBUG actor:402725412 receives from actor:1830160613, file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq, 70008000 entries so far
DEBUG actor:402725412 receives from actor:649752862, file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq, 80009000 entries so far
DEBUG actor:402725412 receives from actor:602765326, file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq, 80008000 entries so far
DEBUG actor:402725412 receives from actor:1144056826, file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq, 80010000 entries so far
DEBUG actor:402725412 receives from actor:675296634, file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq, 80011000 entries so far
DEBUG actor:402725412 receives from actor:122779715, file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq, 80009000 entries so far
DEBUG actor:402725412 receives from actor:1288734375, file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq, 80008000 entries so far
DEBUG actor:402725412 receives from actor:1882977711, file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq, 80009000 entries so far
DEBUG actor:402725412 receives from actor:721954240, file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq, 70007000 entries so far
DEBUG actor:402725412 receives from actor:547646904, file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq, 70007000 entries so far
DEBUG actor:402725412 receives from actor:1212714640, file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq, 70007000 entries so far
DEBUG actor:402725412 receives from actor:363590824, file datasets/Iowa_Continuous_Corn/HTHO.1429-2.1388.fastq, 70007000 entries so far
DEBUG Actor 402725412 received from actor 363590824 for file datasets/Iowa_Continuous_Corn/HTHO.1429-2.1388.fastq: 70800844 entries
DEBUG actor:402725412 receives from actor:1288734375, file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq, 90009000 entries so far
DEBUG actor:402725412 receives from actor:122779715, file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq, 90010000 entries so far
DEBUG actor:402725412 receives from actor:1882977711, file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq, 90010000 entries so far
DEBUG actor:402725412 receives from actor:602765326, file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq, 90010000 entries so far
DEBUG actor:402725412 receives from actor:649752862, file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq, 90010000 entries so far
DEBUG actor:402725412 receives from actor:1144056826, file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq, 90012000 entries so far
DEBUG actor:402725412 receives from actor:675296634, file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq, 90012000 entries so far
DEBUG actor:402725412 receives from actor:261732057, file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq, 80008000 entries so far
DEBUG actor:402725412 receives from actor:1630775601, file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq, 80008000 entries so far
DEBUG actor:402725412 receives from actor:1830160613, file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq, 80009000 entries so far
DEBUG actor:402725412 receives from actor:680978333, file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq, 80009000 entries so far
DEBUG actor:402725412 receives from actor:721954240, file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq, 80008000 entries so far
DEBUG actor:402725412 receives from actor:1212714640, file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq, 80008000 entries so far
DEBUG actor:402725412 receives from actor:547646904, file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq, 80008000 entries so far
DEBUG Actor 402725412 received from actor 1144056826 for file datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq: 98895622 entries
DEBUG actor:402725412 receives from actor:122779715, file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq, 100011000 entries so far
DEBUG actor:402725412 receives from actor:1288734375, file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq, 100010000 entries so far
DEBUG actor:402725412 receives from actor:1882977711, file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq, 100011000 entries so far
DEBUG Actor 402725412 received from actor 649752862 for file datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq: 99214846 entries
DEBUG Actor 402725412 received from actor 122779715 for file datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq: 100163104 entries
DEBUG actor:402725412 receives from actor:602765326, file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq, 100011000 entries so far
DEBUG actor:402725412 receives from actor:675296634, file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq, 100013000 entries so far
DEBUG Actor 402725412 received from actor 1288734375 for file datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq: 101018426 entries
DEBUG Actor 402725412 received from actor 675296634 for file datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq: 102182450 entries
DEBUG Actor 402725412 received from actor 1882977711 for file datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq: 104209684 entries
DEBUG actor:402725412 receives from actor:261732057, file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq, 90010000 entries so far
DEBUG actor:402725412 receives from actor:1830160613, file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq, 90010000 entries so far
DEBUG actor:402725412 receives from actor:680978333, file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq, 90010000 entries so far
DEBUG actor:402725412 receives from actor:1630775601, file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq, 90009000 entries so far
DEBUG Actor 402725412 received from actor 680978333 for file datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq: 90613544 entries
DEBUG actor:402725412 receives from actor:721954240, file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq, 90009000 entries so far
DEBUG actor:402725412 receives from actor:547646904, file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq, 90009000 entries so far
DEBUG actor:402725412 receives from actor:1212714640, file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq, 90009000 entries so far
DEBUG Actor 402725412 received from actor 602765326 for file datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq: 108708372 entries
DEBUG Actor 402725412 received from actor 1830160613 for file datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq: 92773918 entries
DEBUG Actor 402725412 received from actor 547646904 for file datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq: 96424282 entries
DEBUG actor:402725412 receives from actor:261732057, file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq, 100011000 entries so far
DEBUG actor:402725412 receives from actor:1630775601, file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq, 100010000 entries so far
DEBUG actor:402725412 receives from actor:721954240, file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq, 100010000 entries so far
DEBUG actor:402725412 receives from actor:1212714640, file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq, 100010000 entries so far
DEBUG Actor 402725412 received from actor 1212714640 for file datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq: 103720072 entries
DEBUG Actor 402725412 received from actor 721954240 for file datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq: 103929788 entries
DEBUG Actor 402725412 received from actor 261732057 for file datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq: 104474188 entries
DEBUG Actor 402725412 received from actor 1630775601 for file datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq: 109812178 entries
DEBUG spawner 800941600 is on node 0
DEBUG spawner 800941600 (0) is on a node that has 23 workers
DEBUG spawner 1163079153 is on node 1
DEBUG spawner 1163079153 (1) is on a node that has 23 workers
DEBUG spawner 139172306 is on node 2
DEBUG spawner 139172306 (2) is on a node that has 23 workers
DEBUG spawner 1114128863 is on node 3
DEBUG spawner 1114128863 (3) is on a node that has 23 workers
DEBUG sequence stores are ready (92)
DEBUG sequence store 0 is 428109700
DEBUG sequence store 1 is 1859224832
DEBUG sequence store 2 is 1817726404
DEBUG sequence store 3 is 680362304
DEBUG sequence store 4 is 313912496
DEBUG sequence store 5 is 1500090952
DEBUG sequence store 6 is 478844432
DEBUG sequence store 7 is 39457720
DEBUG sequence store 8 is 1451755824
DEBUG sequence store 9 is 1507703764
DEBUG sequence store 10 is 416969480
DEBUG sequence store 11 is 1359529400
DEBUG sequence store 12 is 1764571444
DEBUG sequence store 13 is 1041620624
DEBUG sequence store 14 is 1285050912
DEBUG sequence store 15 is 1833356416
DEBUG sequence store 16 is 1271685380
DEBUG sequence store 17 is 85992512
DEBUG sequence store 18 is 88598184
DEBUG sequence store 19 is 1333884056
DEBUG sequence store 20 is 1330670444
DEBUG sequence store 21 is 237027748
DEBUG sequence store 22 is 399115052
DEBUG sequence store 23 is 578901097
DEBUG sequence store 24 is 119377981
DEBUG sequence store 25 is 1608519685
DEBUG sequence store 26 is 1943389781
DEBUG sequence store 27 is 162177185
DEBUG sequence store 28 is 1686897305
DEBUG sequence store 29 is 1001411021
DEBUG sequence store 30 is 130052141
DEBUG sequence store 31 is 62869621
DEBUG sequence store 32 is 695320025
DEBUG sequence store 33 is 1760490061
DEBUG sequence store 34 is 273749033
DEBUG sequence store 35 is 479885021
DEBUG sequence store 36 is 1771754613
DEBUG sequence store 37 is 936128949
DEBUG sequence store 38 is 1886405325
DEBUG sequence store 39 is 1800937401
DEBUG sequence store 40 is 1624599769
DEBUG sequence store 41 is 902000829
DEBUG sequence store 42 is 1290141489
DEBUG sequence store 43 is 639195269
DEBUG sequence store 44 is 1582979161
DEBUG sequence store 45 is 972818453
DEBUG sequence store 46 is 1844228354
DEBUG sequence store 47 is 593506374
DEBUG sequence store 48 is 1290312834
DEBUG sequence store 49 is 588448238
DEBUG sequence store 50 is 1601375534
DEBUG sequence store 51 is 1943565170
DEBUG sequence store 52 is 835468498
DEBUG sequence store 53 is 604027926
DEBUG sequence store 54 is 934214258
DEBUG sequence store 55 is 529642994
DEBUG sequence store 56 is 437485650
DEBUG sequence store 57 is 1194876174
DEBUG sequence store 58 is 1340557110
DEBUG sequence store 59 is 1516663006
DEBUG sequence store 60 is 1695883970
DEBUG sequence store 61 is 471874506
DEBUG sequence store 62 is 1104197718
DEBUG sequence store 63 is 1997076010
DEBUG sequence store 64 is 7164518
DEBUG sequence store 65 is 1243370022
DEBUG sequence store 66 is 1672340926
DEBUG sequence store 67 is 1893906550
DEBUG sequence store 68 is 1893122886
DEBUG sequence store 69 is 853876991
DEBUG sequence store 70 is 1094312951
DEBUG sequence store 71 is 93871123
DEBUG sequence store 72 is 1365671991
DEBUG sequence store 73 is 951943215
DEBUG sequence store 74 is 1887658259
DEBUG sequence store 75 is 628205727
DEBUG sequence store 76 is 935568603
DEBUG sequence store 77 is 867623855
DEBUG sequence store 78 is 1951284787
DEBUG sequence store 79 is 1803255419
DEBUG sequence store 80 is 1933999127
DEBUG sequence store 81 is 365042163
DEBUG sequence store 82 is 133888743
DEBUG sequence store 83 is 1546993019
DEBUG sequence store 84 is 1980942343
DEBUG sequence store 85 is 54395407
DEBUG sequence store 86 is 91126927
DEBUG sequence store 87 is 93293983
DEBUG sequence store 88 is 1021040619
DEBUG sequence store 89 is 386638991
DEBUG sequence store 90 is 1345686867
DEBUG sequence store 91 is 1143820331
DEBUG sequence files
actor:0, 0/42 -threads-per-node 0
actor:0, 1/42 24 0
actor:0, 2/42 datasets/Iowa_Continuous_Corn/foo 0
actor:0, 3/42 datasets/Iowa_Continuous_Corn/foo~ 0
actor:0, 4/42 datasets/Iowa_Continuous_Corn/foo.out 0
actor:0, 5/42 datasets/Iowa_Continuous_Corn/GPIC.1424-1.1371.fastq 90613544
actor:0, 6/42 datasets/Iowa_Continuous_Corn/GPIC.1424-2.1371.fastq 99214846
actor:0, 7/42 datasets/Iowa_Continuous_Corn/GPIC.1424-3.1371.fastq 100163104
actor:0, 8/42 datasets/Iowa_Continuous_Corn/GPIC.1424-4.1371.fastq 103720072
actor:0, 9/42 datasets/Iowa_Continuous_Corn/GPIC.1424-5.1371.fastq 92773918
actor:0, 10/42 datasets/Iowa_Continuous_Corn/GPIC.1424-6.1371.fastq 98895622
actor:0, 11/42 datasets/Iowa_Continuous_Corn/GPIC.1424-7.1371.fastq 101018426
actor:0, 12/42 datasets/Iowa_Continuous_Corn/GPIC.1425-1.1367.fastq 96424282
actor:0, 13/42 datasets/Iowa_Continuous_Corn/GPIC.1425-2.1367.fastq 109812178
actor:0, 14/42 datasets/Iowa_Continuous_Corn/GPIC.1425-3.1367.fastq 108708372
actor:0, 15/42 datasets/Iowa_Continuous_Corn/GPIC.1425-4.1367.fastq 104209684
actor:0, 16/42 datasets/Iowa_Continuous_Corn/GPIC.1425-5.1367.fastq 103929788
actor:0, 17/42 datasets/Iowa_Continuous_Corn/GPIC.1425-6.1367.fastq 104474188
actor:0, 18/42 datasets/Iowa_Continuous_Corn/GPIC.1425-7.1367.fastq 102182450
actor:0, 19/42 datasets/Iowa_Continuous_Corn/GPIC.649-4.899.fastq 54662170
actor:0, 20/42 datasets/Iowa_Continuous_Corn/GPIC.868-1.1053.fastq 63471812
actor:0, 21/42 datasets/Iowa_Continuous_Corn/GPIC.868-2.1053.fastq 62794138
actor:0, 22/42 datasets/Iowa_Continuous_Corn/GPIC.868-3.1053.fastq 62814332
actor:0, 23/42 datasets/Iowa_Continuous_Corn/GPIC.868-4.1053.fastq 62313864
actor:0, 24/42 datasets/Iowa_Continuous_Corn/GPIC.868-5.1053.fastq 62681112
actor:0, 25/42 datasets/Iowa_Continuous_Corn/GPIC.868-6.1053.fastq 62568148
actor:0, 26/42 datasets/Iowa_Continuous_Corn/GPIC.895-5.1070.fastq 56963934
actor:0, 27/42 datasets/Iowa_Continuous_Corn/GPIC.895-6.1070.fastq 56909532
actor:0, 28/42 datasets/Iowa_Continuous_Corn/GPIC.895-7.1070.fastq 57067668
actor:0, 29/42 datasets/Iowa_Continuous_Corn/HTHO.1209-1.1231.fastq 47646564
actor:0, 30/42 datasets/Iowa_Continuous_Corn/HTHO.1209-2.1231.fastq 51802070
actor:0, 31/42 datasets/Iowa_Continuous_Corn/HTHO.1300-6.1364.fastq 39704380
actor:0, 32/42 datasets/Iowa_Continuous_Corn/HTHO.1429-2.1388.fastq 70800844
actor:0, 33/42 datasets/Iowa_Continuous_Corn/index.html 0
actor:0, 34/42 datasets/Iowa_Continuous_Corn/index.html?C=D;O=A 0
actor:0, 35/42 datasets/Iowa_Continuous_Corn/index.html?C=D;O=D 0
actor:0, 36/42 datasets/Iowa_Continuous_Corn/index.html?C=M;O=A 0
actor:0, 37/42 datasets/Iowa_Continuous_Corn/index.html?C=M;O=D 0
actor:0, 38/42 datasets/Iowa_Continuous_Corn/index.html?C=N;O=A 0
actor:0, 39/42 datasets/Iowa_Continuous_Corn/index.html?C=N;O=D 0
actor:0, 40/42 datasets/Iowa_Continuous_Corn/index.html?C=S;O=A 0
actor:0, 41/42 datasets/Iowa_Continuous_Corn/index.html?C=S;O=D 0
DEBUG Partition Total: 2228341042, block_size: 4096, blocks: 544029
DEBUG Stores: 92, Blocks per store: 5913, entries per store: 24221098
Actor 800941600 is notified by actor 402725412 that the distribution is complete
actor 800941600 stops all other actors
----------------------------------------------
biosal> Counters for node/1
BSAL_COUNTER_SPAWNED_ACTORS 36
BSAL_COUNTER_KILLED_ACTORS 36
 balance BSAL_COUNTER_BALANCE_ACTORS 0
BSAL_COUNTER_RECEIVED_MESSAGES 1141458
  BSAL_COUNTER_RECEIVED_MESSAGES_FROM_SELF 1141368
  BSAL_COUNTER_RECEIVED_MESSAGES_NOT_FROM_SELF 90
BSAL_COUNTER_SENT_MESSAGES 1712114
  BSAL_COUNTER_SENT_MESSAGES_TO_SELF 1141368
  BSAL_COUNTER_SENT_MESSAGES_NOT_TO_SELF 570746
 balance BSAL_COUNTER_BALANCE_MESSAGES -570656
BSAL_COUNTER_RECEIVED_BYTES 699
  BSAL_COUNTER_RECEIVED_BYTES_FROM_SELF 24
  BSAL_COUNTER_RECEIVED_BYTES_NOT_FROM_SELF 675
BSAL_COUNTER_SENT_BYTES 2282996
  BSAL_COUNTER_SENT_BYTES_TO_SELF 24
 BSAL_COUNTER_SENT_BYTES_NOT_TO_SELF 2282972
 balance BSAL_COUNTER_BALANCE_BYTES -2282297
----------------------------------------------
biosal> Counters for node/0
BSAL_COUNTER_SPAWNED_ACTORS 37
BSAL_COUNTER_KILLED_ACTORS 37
 balance BSAL_COUNTER_BALANCE_ACTORS 0
BSAL_COUNTER_RECEIVED_MESSAGES 3345107
  BSAL_COUNTER_RECEIVED_MESSAGES_FROM_SELF 1674677
  BSAL_COUNTER_RECEIVED_MESSAGES_NOT_FROM_SELF 1670430
BSAL_COUNTER_SENT_MESSAGES 1674943
  BSAL_COUNTER_SENT_MESSAGES_TO_SELF 1674677
  BSAL_COUNTER_SENT_MESSAGES_NOT_TO_SELF 266
 balance BSAL_COUNTER_BALANCE_MESSAGES 1670164
BSAL_COUNTER_RECEIVED_BYTES 8917076
  BSAL_COUNTER_RECEIVED_BYTES_FROM_SELF 2235392
  BSAL_COUNTER_RECEIVED_BYTES_NOT_FROM_SELF 6681684
BSAL_COUNTER_SENT_BYTES 2237388
  BSAL_COUNTER_SENT_BYTES_TO_SELF 2235392
 BSAL_COUNTER_SENT_BYTES_NOT_TO_SELF 1996
 balance BSAL_COUNTER_BALANCE_BYTES 6679688
----------------------------------------------
biosal> Counters for node/2
BSAL_COUNTER_SPAWNED_ACTORS 35
BSAL_COUNTER_KILLED_ACTORS 35
 balance BSAL_COUNTER_BALANCE_ACTORS 0
BSAL_COUNTER_RECEIVED_MESSAGES 1161252
  BSAL_COUNTER_RECEIVED_MESSAGES_FROM_SELF 1161164
  BSAL_COUNTER_RECEIVED_MESSAGES_NOT_FROM_SELF 88
BSAL_COUNTER_SENT_MESSAGES 1741806
  BSAL_COUNTER_SENT_MESSAGES_TO_SELF 1161164
  BSAL_COUNTER_SENT_MESSAGES_NOT_TO_SELF 580642
 balance BSAL_COUNTER_BALANCE_MESSAGES -580554
BSAL_COUNTER_RECEIVED_BYTES 685
  BSAL_COUNTER_RECEIVED_BYTES_FROM_SELF 24
  BSAL_COUNTER_RECEIVED_BYTES_NOT_FROM_SELF 661
BSAL_COUNTER_SENT_BYTES 2322580
  BSAL_COUNTER_SENT_BYTES_TO_SELF 24
 BSAL_COUNTER_SENT_BYTES_NOT_TO_SELF 2322556
 balance BSAL_COUNTER_BALANCE_BYTES -2321895
----------------------------------------------
biosal> Counters for node/3
BSAL_COUNTER_SPAWNED_ACTORS 35
BSAL_COUNTER_KILLED_ACTORS 35
 balance BSAL_COUNTER_BALANCE_ACTORS 0
BSAL_COUNTER_RECEIVED_MESSAGES 1038052
  BSAL_COUNTER_RECEIVED_MESSAGES_FROM_SELF 1037964
  BSAL_COUNTER_RECEIVED_MESSAGES_NOT_FROM_SELF 88
BSAL_COUNTER_SENT_MESSAGES 1557006
  BSAL_COUNTER_SENT_MESSAGES_TO_SELF 1037964
  BSAL_COUNTER_SENT_MESSAGES_NOT_TO_SELF 519042
 balance BSAL_COUNTER_BALANCE_MESSAGES -518954
BSAL_COUNTER_RECEIVED_BYTES 684
  BSAL_COUNTER_RECEIVED_BYTES_FROM_SELF 24
  BSAL_COUNTER_RECEIVED_BYTES_NOT_FROM_SELF 660
BSAL_COUNTER_SENT_BYTES 2076180
  BSAL_COUNTER_SENT_BYTES_TO_SELF 24
 BSAL_COUNTER_SENT_BYTES_NOT_TO_SELF 2076156
 balance BSAL_COUNTER_BALANCE_BYTES -2075496
Application 4642195 resources: utime ~17998s, stime ~927s, Rss ~60224, inblocks ~984147715, outblocks ~50517

 ############################################################################
 #                  Job ID: 2716005  Project: CI-#########                  #
 #                  End time: Sat Jun  7 23:04:07 CDT 2014                  #
 #                             Resources Used:                              #
 #         cput=00:00:00,mem=5828kb,vmem=130864kb,walltime=00:03:39         #
 ############################# Beagle Job End ###############################