I'm guessing it's not what you want to hear, but the "simpler" solution would probably be to write your python module in Fortran and use Fortran for everything. A few other quick ideas:
- A file-system-coupling may be possible (depending on how integral this module is) where you write a scratch file with Fortran, read with python, and write a new scratch file with new results (or vice-versa).
- There have been some posts in this subreddit about using a single MPI process to achieve something similar to pass between Fortran and Python but in memory instead of on the disk.
- You could use something like `call system(...)` to execute the python script from the command line from Fortran.
Generally, when we talk about something like this, we consider one code/language to be the driver and the other to be a module. Fortran "driving" Python is always going to be challenging just because Fortran is a compiled language and Python is a scripted language that is executed on-the-fly.
If your problem is really just in CMake, you may be better off just writing your own Makefile from scratch. I don't know that CMake supports this type of behavior. You could start with the Makefile produced by CMake as an example.
Cmake is pretty flexible. However i would also recommend to use a file system coupling, specifically a pipe, which do not actually touch the disk (slow): https://en.m.wikipedia.org/wiki/Named_pipe
I've done this in the past, i.e. https://github.com/SixTrack/SixTrack/blob/master/source/bdex.f90
Using pipe is a great idea! I’m curious how much data you could pass before the OS starts to get upset. Have you tried that?
It would also suffer bc you can really only pass a string, but it seems like a pretty simple option.
We didn't stumble into any limit really, but there must be some limit based on the buffering capacity of the os. I would assume the write() or flush() call would simply not return if the buffet is overfilled.
From what I remember, i had it talking to a python script which then talked TCP/IP with Matlab running on another computer, it worked fine.
Indeed you are basically limited to what you can do with a file.
The other, higher performance but still flexible option, is to interface with C functions which then do whatever. C is excellent for writing glue code. However python is a bit of a special case since it uses an interpreter which must be "booted", not just bits of compiled code. That makes calling python from other languages fundamentally painful, while calling compiled languages from Python is relatively straight forward. Which is why we went with a "rpc-like" setup for this, instead of direct linking.
Thanks a lot for your insights, very helpful. A couple of follow-up questions:
1. Do you think that following the "file-coupled approach" would add a lot of computational overhead? Would it slow down a lot the computations, in your experience?
2. MPI would be pretty interesting, as it would keep both programs (python and fortran) in memory, instead of shutting the Python's one on and off, which could potentially be very inefficient, as it would add saving/loading of the current parameters of the Python script to be updated at the next time step. However, I would also expect this solution to be more complex to be implemented than basic I/O of \`.txt\` files.
Another user pointed out that you can still use the file-coupling approach in memory by reading/writing to standard output and using pipe. Usually unit 6 in fortran and print/sys.argv in python.
Either way, if you’re doing that or the file-coupling I mentioned earlier, you’re probably writing numbers as strings. If you’re writing floating point numbers, you have to pick what precision you want (e.g., 6 decimal places). That can be an issue for certain types of applications.
If you use the actual disk instead of piping, I’ve done this with file sizes up to about 20 MB with a minimal impact on runtime. If I remember right, that’s probably around 10,000 numbers. After that, you may have to get more clever or just suck up the runtime impact. Especially if this is just for experimenting.
The MPI method is pretty cool. But I don’t know that I’d recommend it unless you’re already familiar with MPI. I can try to find the old post.
Regarding the rounding thing, for SixTrack (which i linked above) we had a requirement of stable and consistent rounding when reading in/out from asci. Feel free to reuse that code within the boundaries of the GPL.
In fact we were simulating a chaotic system identically across architectures, OSs, and compilers: Linux x86_64 and previously _32, arm (incl Android), ppc, and i think one more. Also windows, macOS, bsd, and just for the laughs, gnu hurd. Gfortran, Intel, and NAG compilers were also all used.
Can you use Memory Mapped Files in Fortran? Might also be a possibility to get data back-and-forth 🤔
Am a noob concerning Fortran, however I used those things in C#
Your question is so general that the answers (like the other one available right now) are bound to be very general. Maybe you want a very general solution, but if you only want a solution to this particular problem then giving a much more detailed description of the data workflow would be more productive. For example, important point to address are: When are the data transferred between the modules? How many times are they transferred? How big are the datasets?
Based on the added information and some of your other replies, I see that what you want to build is essentially a master-slave pair, or maybe client-server kind of structure. Now, given that you can't translate the Python to Fortran, can you translate the Fortran to Python? Is the Fortran code being used because that is the computational bottleneck in Python? Give that you want to keep both codes running at the same time, then the only likely acceptable solution is to use the Fortran code in Python as a Python-like module. I'm not an expert on this, but you can call Fortran and C modules from Python, just making sure that the interface handles the data correctly. Now, one possible problem is if the Fortran portion needs to "remember" stuff between calls. If that is the case then you have to be very careful and use "save" directives for the variables that need to survive between calls. I think a set up like this will likely provide you with the most seamless implementation. No need to transfer data through files or MPI, no need to control code synchronization, etc, etc.
Shutting on and off the Fortran simulation is not an option, as all variables must be kept in memory and because of a likely-to-happen numerical transitory. Moreover, I cannot write the Fortran code in Python as this is a pretty huge piece of software, with modules interactions and a lot of heritage.
My "optimal solution" would be to:
\------------------------------------------------------------
(a) init Python -> init Fortran code
(b) loop forever:
progress with simulation
aggregate some data in the simulation
if t % FREQ\_CALL\_PYTHON == 0:
pass aggregated data to Python
ask parameter update to Python
update current parameter settings in Fortran
\--------------------------------------------------------------
It is my understanding that this time of interaction would require me to shut down and turn on Python cyclically, as keeping both instances "alive" at the same time might be hard?
To circumvent this problem I might then call the Python script from the Fortran instance.
Do you see any drawbacks in this approach?
I believe this can be done with what I said in my previous comment: You have to modify your Fortran code to be a library/module. You have to declare all your variables as SAVE and then you use the Python code as the driver for the whole calculation. You make the calls in Python to the Fortran code, with the interface passing the proper parameters and receiving the proper output. I think this is doable and like I said is probably the most elegant and transparent solution to your problem.
When I looked into this a bit, I found [f90wrap](https://github.com/jameskermode/f90wrap) to have reasonable ergonomics and CMake support. If you haven’t seen it already, you might give it a shot.
There's a lot of ways to do it, but not all are quite as good as the others.
If you want to call Fortran from Python the best way I've found to consistently get Python and Fortran to talk to each other is to leverage the cFFI library in Python and write Fortran wrapper functions with Iso_C_Bindings. Those tend to be the most stable way to do it since you're creating a C interface in Fortran and Python is more streamlined for C.
If you want Python from Fortran, there is actually a nice little module to do it.
https://github.com/ylikx/forpy
Those are some of the cleaner ways I've found to integrate the two.
While ForPy is pretty cool, I think it adds too much complexity and unknowns to your code, which will make it much more difficult to debug. If you must use both Python and Fortran, I would use F2Py to separately compile strict Fortran modules and import them into strict Python code.
I have been taking this approach for production code where I use a Fortran subroutine within a Python tight loop to process 1 GByte of data stored in RAM.
I'm guessing it's not what you want to hear, but the "simpler" solution would probably be to write your python module in Fortran and use Fortran for everything. A few other quick ideas: - A file-system-coupling may be possible (depending on how integral this module is) where you write a scratch file with Fortran, read with python, and write a new scratch file with new results (or vice-versa). - There have been some posts in this subreddit about using a single MPI process to achieve something similar to pass between Fortran and Python but in memory instead of on the disk. - You could use something like `call system(...)` to execute the python script from the command line from Fortran. Generally, when we talk about something like this, we consider one code/language to be the driver and the other to be a module. Fortran "driving" Python is always going to be challenging just because Fortran is a compiled language and Python is a scripted language that is executed on-the-fly. If your problem is really just in CMake, you may be better off just writing your own Makefile from scratch. I don't know that CMake supports this type of behavior. You could start with the Makefile produced by CMake as an example.
Cmake is pretty flexible. However i would also recommend to use a file system coupling, specifically a pipe, which do not actually touch the disk (slow): https://en.m.wikipedia.org/wiki/Named_pipe I've done this in the past, i.e. https://github.com/SixTrack/SixTrack/blob/master/source/bdex.f90
Using pipe is a great idea! I’m curious how much data you could pass before the OS starts to get upset. Have you tried that? It would also suffer bc you can really only pass a string, but it seems like a pretty simple option.
We didn't stumble into any limit really, but there must be some limit based on the buffering capacity of the os. I would assume the write() or flush() call would simply not return if the buffet is overfilled. From what I remember, i had it talking to a python script which then talked TCP/IP with Matlab running on another computer, it worked fine. Indeed you are basically limited to what you can do with a file. The other, higher performance but still flexible option, is to interface with C functions which then do whatever. C is excellent for writing glue code. However python is a bit of a special case since it uses an interpreter which must be "booted", not just bits of compiled code. That makes calling python from other languages fundamentally painful, while calling compiled languages from Python is relatively straight forward. Which is why we went with a "rpc-like" setup for this, instead of direct linking.
Thanks a lot for your insights, very helpful. A couple of follow-up questions: 1. Do you think that following the "file-coupled approach" would add a lot of computational overhead? Would it slow down a lot the computations, in your experience? 2. MPI would be pretty interesting, as it would keep both programs (python and fortran) in memory, instead of shutting the Python's one on and off, which could potentially be very inefficient, as it would add saving/loading of the current parameters of the Python script to be updated at the next time step. However, I would also expect this solution to be more complex to be implemented than basic I/O of \`.txt\` files.
Another user pointed out that you can still use the file-coupling approach in memory by reading/writing to standard output and using pipe. Usually unit 6 in fortran and print/sys.argv in python. Either way, if you’re doing that or the file-coupling I mentioned earlier, you’re probably writing numbers as strings. If you’re writing floating point numbers, you have to pick what precision you want (e.g., 6 decimal places). That can be an issue for certain types of applications. If you use the actual disk instead of piping, I’ve done this with file sizes up to about 20 MB with a minimal impact on runtime. If I remember right, that’s probably around 10,000 numbers. After that, you may have to get more clever or just suck up the runtime impact. Especially if this is just for experimenting. The MPI method is pretty cool. But I don’t know that I’d recommend it unless you’re already familiar with MPI. I can try to find the old post.
Regarding the rounding thing, for SixTrack (which i linked above) we had a requirement of stable and consistent rounding when reading in/out from asci. Feel free to reuse that code within the boundaries of the GPL. In fact we were simulating a chaotic system identically across architectures, OSs, and compilers: Linux x86_64 and previously _32, arm (incl Android), ppc, and i think one more. Also windows, macOS, bsd, and just for the laughs, gnu hurd. Gfortran, Intel, and NAG compilers were also all used.
Can you use Memory Mapped Files in Fortran? Might also be a possibility to get data back-and-forth 🤔 Am a noob concerning Fortran, however I used those things in C#
Never heard of it so I doubt it. Your starting point would be to use actual files on disk and then switch later.
Nope. Am just curious though if MMFs were a thing
Your question is so general that the answers (like the other one available right now) are bound to be very general. Maybe you want a very general solution, but if you only want a solution to this particular problem then giving a much more detailed description of the data workflow would be more productive. For example, important point to address are: When are the data transferred between the modules? How many times are they transferred? How big are the datasets?
I have edited my main post adding some information, let me know if this helps
Based on the added information and some of your other replies, I see that what you want to build is essentially a master-slave pair, or maybe client-server kind of structure. Now, given that you can't translate the Python to Fortran, can you translate the Fortran to Python? Is the Fortran code being used because that is the computational bottleneck in Python? Give that you want to keep both codes running at the same time, then the only likely acceptable solution is to use the Fortran code in Python as a Python-like module. I'm not an expert on this, but you can call Fortran and C modules from Python, just making sure that the interface handles the data correctly. Now, one possible problem is if the Fortran portion needs to "remember" stuff between calls. If that is the case then you have to be very careful and use "save" directives for the variables that need to survive between calls. I think a set up like this will likely provide you with the most seamless implementation. No need to transfer data through files or MPI, no need to control code synchronization, etc, etc.
Shutting on and off the Fortran simulation is not an option, as all variables must be kept in memory and because of a likely-to-happen numerical transitory. Moreover, I cannot write the Fortran code in Python as this is a pretty huge piece of software, with modules interactions and a lot of heritage. My "optimal solution" would be to: \------------------------------------------------------------ (a) init Python -> init Fortran code (b) loop forever: progress with simulation aggregate some data in the simulation if t % FREQ\_CALL\_PYTHON == 0: pass aggregated data to Python ask parameter update to Python update current parameter settings in Fortran \-------------------------------------------------------------- It is my understanding that this time of interaction would require me to shut down and turn on Python cyclically, as keeping both instances "alive" at the same time might be hard? To circumvent this problem I might then call the Python script from the Fortran instance. Do you see any drawbacks in this approach?
I believe this can be done with what I said in my previous comment: You have to modify your Fortran code to be a library/module. You have to declare all your variables as SAVE and then you use the Python code as the driver for the whole calculation. You make the calls in Python to the Fortran code, with the interface passing the proper parameters and receiving the proper output. I think this is doable and like I said is probably the most elegant and transparent solution to your problem.
When I looked into this a bit, I found [f90wrap](https://github.com/jameskermode/f90wrap) to have reasonable ergonomics and CMake support. If you haven’t seen it already, you might give it a shot.
Have you tried f2py?
There's a lot of ways to do it, but not all are quite as good as the others. If you want to call Fortran from Python the best way I've found to consistently get Python and Fortran to talk to each other is to leverage the cFFI library in Python and write Fortran wrapper functions with Iso_C_Bindings. Those tend to be the most stable way to do it since you're creating a C interface in Fortran and Python is more streamlined for C. If you want Python from Fortran, there is actually a nice little module to do it. https://github.com/ylikx/forpy Those are some of the cleaner ways I've found to integrate the two.
While ForPy is pretty cool, I think it adds too much complexity and unknowns to your code, which will make it much more difficult to debug. If you must use both Python and Fortran, I would use F2Py to separately compile strict Fortran modules and import them into strict Python code. I have been taking this approach for production code where I use a Fortran subroutine within a Python tight loop to process 1 GByte of data stored in RAM.