Cannot use mpco recorder in OpenSeesSP when running with multiple cores

Post Reply
jfhuang
Posts: 21
Joined: Thu Nov 26, 2020 5:21 am

Cannot use mpco recorder in OpenSeesSP when running with multiple cores

Post by jfhuang » Thu Nov 26, 2020 5:54 am

Dear STKO community,

I tried to use the mpco recorder in OpenSeesSP but it gave me the following error when I ran with multiple cores:

Code: Select all

root@5888f6ab4cdc:/home/ops# mpirun -np 2 OpenSeesPARALLEL main.tcl
Primary Process Running OpenSees Interpreter 0


         OpenSees -- Open System For Earthquake Engineering Simulation
                 Pacific Earthquake Engineering Research Center
                        Version 3.2.2 64-Bit

      (c) Copyright 1999-2016 The Regents of the University of California
                              All Rights Reserved
  (Copyright and Disclaimer @ http://www.berkeley.edu/OpenSees/copyright.html)


Secondary Process Running 1
MPCO recorder - Written by ASDEA Software Technology: M.Petracca, G.Camata
ASDEA Software Technology: https://asdeasoft.net 
STKO (Scientific ToolKit for OpenSees): https://asdeasoft.net/stko/ 
If you use this tool, please cite us:
Petracca, M., Candeloro, F., & Camata, G. (2017). "STKO user manual". ASDEA Software Technology, Pescara Italy.
Step 1 / 10
MPCORecorder sendSelf from: 0, send self count = 1
MPCORecorder recvSelf from: 1, send self count = -1
filename = mpcoRecorder.hdf5
freq: dt = 0
nodal results [1]
   0
elemental results [0]
node set:
[elem set:
[]
MPI_Channel::recvID() - incorrect number of entries for ID received: 1 exptected: 4
ActorSubdomain::run - error receiving msgData
Fatal error in MPI_Recv: Message truncated, error stack:
MPI_Recv(184)...........................: MPI_Recv(buf=0x556efda59c80, count=1, MPI_INT, src=0, tag=0, MPI_COMM_WORLD, status=0x7fff979b4770) failed
MPIDI_CH3_PktHandler_EagerShortSend(349): Message from rank 0 and tag 0 truncated; 16 bytes received but buffer size is 4

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
However, it is interesting to note that I would not get the error when running with only one core ("mpirun -np 1 OpenSeesPARALLEL main.tcl" or "OpenSeesPARALLEL main.tcl"), as shown below:

Code: Select all

root@5888f6ab4cdc:/home/ops# OpenSeesPARALLEL main.tcl
Primary Process Running OpenSees Interpreter 0


         OpenSees -- Open System For Earthquake Engineering Simulation
                 Pacific Earthquake Engineering Research Center
                        Version 3.2.2 64-Bit

      (c) Copyright 1999-2016 The Regents of the University of California
                              All Rights Reserved
  (Copyright and Disclaimer @ http://www.berkeley.edu/OpenSees/copyright.html)


MPCO recorder - Written by ASDEA Software Technology: M.Petracca, G.Camata
ASDEA Software Technology: https://asdeasoft.net 
STKO (Scientific ToolKit for OpenSees): https://asdeasoft.net/stko/ 
If you use this tool, please cite us:
Petracca, M., Candeloro, F., & Camata, G. (2017). "STKO user manual". ASDEA Software Technology, Pescara Italy.
Step 1 / 10
ID::resize() - size specified 0 <= 0
Step 2 / 10
Step 3 / 10
Step 4 / 10
Step 5 / 10
Step 6 / 10
Step 7 / 10
Step 8 / 10
Step 9 / 10
Step 10 / 10
Process Terminating 0
The error seems to be related to the send and recv functions but I could not figure it out. I compiled OpenSeesSP on ubuntu with mpich2 and hdf5-1.10.4 (I also tried hdf5-1.10.1 but still got the same error). Attached is a simple 1-story 1-bay model that I have tested if you'd like to try it out. It also contains the recorded hdf5 result file. Thanks in advance!

Regards,
Junfei
Attachments
1story1bay.zip
(66.54 KiB) Downloaded 303 times
Top
STKO Team
Posts: 3066
Joined: Tue Oct 29, 2019 8:45 am

Re: Cannot use mpco recorder in OpenSeesSP when running with multiple cores

Post by STKO Team » Thu Nov 26, 2020 11:23 am

Dear user,

Of course it is an error of the sendSelf and rcvSelf methods. It does not happen with a single core because in that case OpenSees will never call those methods.

Those methods are written correctly, but unfortunately we have seen that sometimes (probably due to the way opensees handles the MPI API for sending and receiving messages) the sent messages are not received in the same order. I've seen that it only happens when an object sends multiple vectors of ID, doubles and strings.

In fact the MPCORecorder sends and receives multiple data becasue it is a very complex class.

We are currently working in a workaround that will be available in few days. Basically instead of sending multiple vectors, we serialize everything into a formatted string. So that we send only 1 message (the string) and we de-serialize it in the recvSelf method.
We will let you know as soon as we test it and confirm it works.

In the meantime, have you tried to use OpenSeesMP instead? It does not suffer from the same problem, since the model is built directly on the different processors, so the sendSelf and recvSelf methods are not called at all.
Of course wrinting the TCL input file is not easy at all, but STKO can handle it for you.
Top
jfhuang
Posts: 21
Joined: Thu Nov 26, 2020 5:21 am

Re: Cannot use mpco recorder in OpenSeesSP when running with multiple cores

Post by jfhuang » Thu Nov 26, 2020 6:42 pm

Hello STKO Team,

Much appreciate your quick response and help! I look forward to trying the new implementation out!

I haven't tried OpenSeesMP and the partition feature in STKO yet as my models were built previously without STKO and it would take a long time to rebuild them in the STKO UI as the models are quite large. Actually I was wondering if there could be a workaround in the future to import existing models, say using a recorder like mpco to output the model info so that STKO can read and rebuild the model? That would be cool and very useful! In the meantime, it would still be really great if I can use STKO for post-processing!

Thanks again for your efforts and help!

Regards,
Junfei
Top
STKO Team
Posts: 3066
Joined: Tue Oct 29, 2019 8:45 am

Re: Cannot use mpco recorder in OpenSeesSP when running with multiple cores

Post by STKO Team » Mon Nov 30, 2020 9:55 am

On more question:
Did you try the fixed version of the MPCORecorder file that Dr. Jose Abell published in a recent Pull Request in GitHub? He says that ha does not have problems with that.
In the meantime you can check whether those fixed work for you
Top
jfhuang
Posts: 21
Joined: Thu Nov 26, 2020 5:21 am

Re: Cannot use mpco recorder in OpenSeesSP when running with multiple cores

Post by jfhuang » Tue Dec 01, 2020 6:19 pm

Thanks for looking into this!
Yes, to make sure I have used the fixed version, I tried twice to compile the OpenSees code but would still get the same error. Not sure why this would happen. I see the main changes was in the MPCORecorder::recvSelf() function.
Top
STKO Team
Posts: 3066
Joined: Tue Oct 29, 2019 8:45 am

Re: Cannot use mpco recorder in OpenSeesSP when running with multiple cores

Post by STKO Team » Thu Dec 03, 2020 9:30 am

Dear user,

I've seen that the PR made by J. Abell to fix that issue was not yet merged when you tried it.
It has been merged 18 hours ago, as you can see here:
https://github.com/OpenSees/OpenSees/pull/467

Download the source-code now, test it, and let us know. If your problem persists I will try to change the code.
Top
jfhuang
Posts: 21
Joined: Thu Nov 26, 2020 5:21 am

Re: Cannot use mpco recorder in OpenSeesSP when running with multiple cores

Post by jfhuang » Thu Dec 03, 2020 6:06 pm

Hello STKO Team,

Thanks for the reminder. Previously I was using the MPCORecorder.cpp file directly from Dr Abell's repo as that was the only file changed for the recorder. I have tried to use the latest code from the official OpenSees repo (pull request 475 got merged when I cloned the code. I changed "DomainPartitioner *thePartitioner = 0;//this->getPartitioner();" to "DomainPartitioner *thePartitioner = this->getPartitioner();" in the PartitionedDomain.cpp file to make the partition work, guessing it was just a typo) and the problem from MPCO still exist, as shown below:

Code: Select all

root@a62e97a2e5aa:/home/ops# mpirun -np 4 OpenSeesPARALLEL main.tcl 
Primary Process Running OpenSees Interpreter 0


         OpenSees -- Open System For Earthquake Engineering Simulation
                 Pacific Earthquake Engineering Research Center
                        Version 3.2.2 64-Bit

      (c) Copyright 1999-2016 The Regents of the University of California
                              All Rights Reserved
  (Copyright and Disclaimer @ http://www.berkeley.edu/OpenSees/copyright.html)


Secondary Process Running 1
Secondary Process Running 2
Secondary Process Running 3
        Start Gravity Analysis
          apply load factor: 0.5
DomainPartitioner::partition() - Start
DomainPartitioner::partition - Succesfull partition. Now redistributing data accordingly.
  * Identifying components to transfer.
     + Elements.
     + Nodes.
     + Boundary Nodes.
     + MP constraints.
     + MP constraints (2nd pass).
  * Sending nodes.
  * Sending elements.
  * Sending Load Patterns
  * Sending homogeneous SP Constraints`
  * Sending MP Constraints
DomainPartitioner::partition() - Done
          apply load factor: 1.0
        Gravity Analysis Has Been Finished! 

MPCO recorder - Written by ASDEA Software Technology: M.Petracca, G.Camata
ASDEA Software Technology: https://asdeasoft.net
STKO (Scientific ToolKit for OpenSees): https://asdeasoft.net/stko/
If you use this tool, please cite us:
Petracca, M., Candeloro, F., & Camata, G. (2017). "STKO user manual". ASDEA Software Technology, Pescara Italy.    
MPCORecorder sendSelf from: 0, send self count = 1
MPCORecorder sendSelf from: 0, send self count = 2
MPCORecorder sendSelf from: 0, send self count = 3
MPCORecorder recvSelf from: 1, send self count = -1
filename = mpcoRecorder.hdf5
freq: dt = 0
nodal results [2]
   0
   1
elemental results [0]
node set:
[]
elem set:
[]
MPCORecorder recvSelf from: 2, send self count = -1
filename = mpcoRecorder.hdf5
freq: dt = 0
nodal results [2]
   0
   1
elemental results [0]
node set:
[]
elem set:
[]
MPI_Channel::recvID() - incorrect number of entries for ID received: 2 exptected: 4
ActorSubdomain::run - error receiving msgData
MPI_Channel::recvID() - incorrect number of entries for ID received: 2 exptected: 4
ActorSubdomain::run - error receiving msgData
MPCORecorder recvSelf from: 3, send self count = -1
filename = mpcoRecorder.hdf5
freq: dt = 0
nodal results [2]
   0
   1
elemental results [0]
node set:
[]
elem set:
[]
MPI_Channel::recvID() - incorrect number of entries for ID received: 2 exptected: 4
ActorSubdomain::run - error receiving msgData
Fatal error in MPI_Recv: Message truncated, error stack:
MPI_Recv(184)...........................: MPI_Recv(buf=0x5651f8a15700, count=1, MPI_INT, src=0, tag=0, MPI_COMM_WORLD, status=0x7fff1b966150) failed
MPIDI_CH3_PktHandler_EagerShortSend(349): Message from rank 0 and tag 0 truncated; 16 bytes received but buffer size is 4
Fatal error in MPI_Recv: Message truncated, error stack:
MPI_Recv(184)...........................: MPI_Recv(buf=0x55c3333b1700, count=1, MPI_INT, src=0, tag=0, MPI_COMM_WORLD, status=0x7ffeae1bfd40) failed
MPIDI_CH3_PktHandler_EagerShortSend(349): Message from rank 0 and tag 0 truncated; 16 bytes received but buffer size is 4
Fatal error in MPI_Recv: Message truncated, error stack:
MPI_Recv(184)...........................: MPI_Recv(buf=0x562bb77bd700, count=1, MPI_INT, src=0, tag=0, MPI_COMM_WORLD, status=0x7ffe41460600) failed
MPIDI_CH3_PktHandler_EagerShortSend(349): Message from rank 0 and tag 0 truncated; 16 bytes received but buffer size is 4

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
As before, there is no problem when using 1 core:

Code: Select all

root@a62e97a2e5aa:/home/ops# mpirun -np 4 OpenSeesPARALLEL main.tcl 
Primary Process Running OpenSees Interpreter 0


         OpenSees -- Open System For Earthquake Engineering Simulation
                 Pacific Earthquake Engineering Research Center
                        Version 3.2.2 64-Bit

      (c) Copyright 1999-2016 The Regents of the University of California
                              All Rights Reserved
  (Copyright and Disclaimer @ http://www.berkeley.edu/OpenSees/copyright.html)


Secondary Process Running 1
Secondary Process Running 2
Secondary Process Running 3
        Start Gravity Analysis
          apply load factor: 0.5
DomainPartitioner::partition() - Start
DomainPartitioner::partition - Succesfull partition. Now redistributing data accordingly.
  * Identifying components to transfer.
     + Elements.
     + Nodes.
     + Boundary Nodes.
     + MP constraints.
     + MP constraints (2nd pass).
  * Sending nodes.
  * Sending elements.
  * Sending Load Patterns
  * Sending homogeneous SP Constraints`
  * Sending MP Constraints
DomainPartitioner::partition() - Done
          apply load factor: 1.0
        Gravity Analysis Has Been Finished!


-----------------------------------------------------------
START Transient Analysis
-----------------------------------------------------------
        CurrentTime =  0.002, TotalTime =  0.023
        CurrentTime =  0.005, TotalTime =  0.023
        CurrentTime =  0.007, TotalTime =  0.023
        CurrentTime =  0.009, TotalTime =  0.023
        CurrentTime =  0.011, TotalTime =  0.023
        CurrentTime =  0.014, TotalTime =  0.023
        CurrentTime =  0.016, TotalTime =  0.023
        CurrentTime =  0.018, TotalTime =  0.023
        CurrentTime =  0.020, TotalTime =  0.023
        CurrentTime =  0.023, TotalTime =  0.023
        CurrentTime =  0.025, TotalTime =  0.023

-----------------------------------------------------------
        FINISH Transient Analysis

-----------------------------------------------------------
Time for analyzing 10 steps: 15 s
Process Terminating 0
Process Terminating 1
Process Terminating 2
Process Terminating 3
This is quite puzzling as the new code doesn't work in my cases. Thanks for your attention!

Regards,
Junfei
Top
STKO Team
Posts: 3066
Joined: Tue Oct 29, 2019 8:45 am

Re: Cannot use mpco recorder in OpenSeesSP when running with multiple cores

Post by STKO Team » Thu Dec 03, 2020 6:36 pm

Ok, so I will do some tests on my side, and eventually, I will try the workaround for the send/recv self methods.
Top
jfhuang
Posts: 21
Joined: Thu Nov 26, 2020 5:21 am

Re: Cannot use mpco recorder in OpenSeesSP when running with multiple cores

Post by jfhuang » Fri Dec 04, 2020 12:28 am

Sounds great, thanks, let me know if I can be of any help!
Top
jfhuang
Posts: 21
Joined: Thu Nov 26, 2020 5:21 am

Re: Cannot use mpco recorder in OpenSeesSP when running with multiple cores

Post by jfhuang » Fri Jan 29, 2021 7:54 pm

Just wanted to check any update on this?
Top
Post Reply

Return to “OpenSees SP”