Refactoring of the pyam data backend


Daniel Huppmann
 

Dear users of the pyam package,

We have been working over the past weeks to refactor the data backend of pyam to improve performance: increase speed and reduce memory usage.

Because this was quite a major operation, it would be great if you could help us in testing if all features continue to work as before - in principle, everything is tested, but it could be that something goes wrong in corner cases.

To help us make pyam better for future users (and yourself), please take 30 minutes to do the following:

1) install the latest release version of pyam (via pypi or condo)
2) dig out any long, complicated script that you have done for scenario analysis or processing with pyam, and add the following
at the top:
```
import time
start_time = time.time()
```
at the end
```
print(time.time() - start_time)
```
3) run the script, save any outputs (excel tables, figures) and note how long the script took (see the print statement).
4) install the latest branch from GitHub
5) re-run the script (don’t forget to restart the kernel in a Jupyter notebook)
6) compare the outputs and the time to step 3
7) please add any comments about unexpected behaviour or errors as a new issue in the GitHub repo: https://github.com/IAMconsortium/pyam/issues/new

Expected timeline: it would be good to have some feedback until the end of this week (Sep 29) so that we continue with a new release in the first week of October.

Thank you so much,
Daniel

PS: shout-out to Florian Maczek who is currently an intern at IIASA and who has implemented a lot of the refactoring!

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@iiasa.ac.at
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann