|
Update to the Scenario Explorer passwords required
Daniel Huppmann
Dear pyam users, This message is relevant if you are using pyam to query data directly from any IIASA Scenario Explorer database. If you didn’t know that you could do that, check out this tutorial... # Update to the Scenario Explorer passwords We are constantly improving the database infrastructure and tools supporting the Scenario Explorer. To facilitate user/permission management and hosting many Scenario Explorer instances in parallel, we are in the process to migrate to a new "Scenario Services Manager". This will not require any changes to Scenario Explorer instances for users or for pyam-connection to the IIASA database API. However, for security reasons, every user will have to set a new password in the new system. The new service is available at https://manager.ece.iiasa.ac.at - you can go there and click "Reset password" to set a new password. Don’t forget to run the following line of Python code (only needed once) to update your local configuration. pyam.iiasa.set_config("<login>", "<password>")Feedback on the new service is welcome! Over the past months, we streamlined our workflows for scenario validation and region-processing that happens when submitting scenario results to the IIASA Scenario Explorer infrastructure. For this effort, we have completely separated the workflow into an installable, open-source Python package „nomenclature-iamc“ and project-specific GitHub repositories that hold the definitions and mappings. Maybe of interest for more advanced developers - it is now easily possible to run the validation and region-processing locally, before uploading the data file to the Scenario Explorer. This can increase efficiency and speed of debugging when you are implementing your workflows to export results to the common format and the variables and regions used in our project. Read the docs at https://nomenclature-iamc.readthedocs.io! # Access to the lists of variables regions used in the project The lists of variables (including expected units and descriptions) and the lists of regions used in a project hosted by the IIASA ECE Scenario Services team are available on Github (private or public depending on the project). Please reach out to us (with your GitHub user name) if you would like to have access to the repository of a project where you are participating. Best regards, Daniel — Dr. Daniel HUPPMANN Research Scholar Coordinator of the Research Theme „Scenario Services & Scientific Software" Energy, Climate, and Environment (ECE) Program email: huppmann@... Twitter: @daniel_huppmann International Institute for Applied Systems Analysis Schlossplatz 1, A-2361 Laxenburg, Austria | www.iiasa.ac.at
|
|
|
|
Update to the IIASA Scenario Explorer passwords
Daniel Huppmann
Dear pyam community, This is relevant if you are using pyam to query data directly from an IIASA Scenario Explorer database… (if you didn’t know you can do that, check out this tutorial). We are constantly improving the database infrastructure and tools supporting the Scenario Explorer. To facilitate user/permission management and hosting several Scenario Explorer instances in parallel, we will migrate to a new "Scenario Services Manager“. The migration is planned for this week. This will not require any changes to Scenario Explorer instances for users or for pyam-connection to the IIASA database API. However, for security reasons, every user will have to set a new password in the new system. Also, because we will not be able to migrate all config settings and permissions at once, there could be a period of a few days when you have to use the old password in some database instances and the new password in already-migrated instances. The new service is already available at https://manager.ece.iiasa.ac.at - you can go there and click "Reset password" to set a new password (which you can then use for login to any Scenario Explorer instance after the migration). We apologize for the inconvenience! Best regards, Daniel — Dr. Daniel HUPPMANN Research Scholar Coordinator of the Research Theme „Scenario Services & Scientific Software" Energy, Climate, and Environment (ECE) Program email: huppmann@... Twitter: @daniel_huppmann International Institute for Applied Systems Analysis Schlossplatz 1, A-2361 Laxenburg, Austria | www.iiasa.ac.at
|
|
|
|
#release pyam v1.5.0 - The IamSlice class
#release
Daniel Huppmann
Dear pyam user community, Happy to announce a new release v1.5 of the pyam package! # Highlight This release introduces an IamSlice class that allows faster filtering and inspection of an IamDataFrame - thanks to Jonas Hörsch for implementing this new feature! The release also adds an offset() method (docs) and improved support for a non-standard index # Join the Slack workspace? If you want to ask questions or read tips-and-tricks from time to time on how to use pyam more effectively in your scenario analysis or data-viz work, maybe consider joining our Slack workspace? For more information, please read the full release notes... Best regards, Daniel — Dr. Daniel HUPPMANN Research Scholar Coordinator of the Research Theme „Scenario Services & Scientific Software" Energy, Climate, and Environment (ECE) Program email: huppmann@... Twitter: @daniel_huppmann International Institute for Applied Systems Analysis Schlossplatz 1, A-2361 Laxenburg, Austria | www.iiasa.ac.at _._,_._,
|
|
|
|
#release pyam v1.4.0 - IPCC AR6 WG3 color palette & Python 3.10
#release
Daniel Huppmann
Dear pyam user community, A few days ago, the IPCC WG3 released its contribution to the Sixth Assessment report (AR6), including a public ensemble of scenarios and pathways supporting quantitative statements in the report (see here). And to mark the occasion and support subsequent work with the scenario data… I’m happy to announce a new release v1.4 of the pyam package! # Highlight - Add colors used for IPCC AR6 WGIII scenario analysis to the pyam plotting module (thanks Jarmo Kikstra!) - Support scenario data with mixed 'year' and 'datetime' domain (beta) - Add explicit support for Python 3.10 (including testing via GitHub Actions) # Dependency changes Following a change of the UNFCCC data inventory API, PR #647 updated the dependencies to require unfccc-di-api>=3.0.1. # Known issues The latest release of the pint package, which is used by the iam-units and the pyam packages, introduced a regression, see issue hgrecco/pin#498. For the time being, please stick with pint<=18.0. # API changes PR #598 added support for mixed time-domains, i.e., where the time column has both integer and datetime items. As part of the changes, filtering an IamDataFrame with yearly data using arguments that are only relevant for the datetime-domain (e.g., month, hour, time) returns an empty IamDataFrame. Previously, this raised an error. # Join the Slack workspace? If you want to ask questions or read tips-and-tricks from time to time on how to use pyam more effectively in your scenario analysis or data-viz work, maybe consider joining our Slack workspace? For more information, please read the full release notes... Best regards, Daniel — Dr. Daniel HUPPMANN Research Scholar Coordinator of the Research Theme „Scenario Services & Scientific Software" Energy, Climate, and Environment (ECE) Program email: huppmann@... Twitter: @daniel_huppmann International Institute for Applied Systems Analysis Schlossplatz 1, A-2361 Laxenburg, Austria | www.iiasa.ac.at
|
|
|
|
#release pyam v1.3.1 - compatibility patch
#release
Daniel Huppmann
Dear pyam user community, Happy to announce a new patch release v1.3.1 of the pyam package! # Incompatibility with pandas v1.4 fixed Jonas Hörsch implemented a fix for the incompatibility with the just-released pandas v1.4. Get the latest from pypi, GitHub or conda (well, conda is still working on it). # Join the Slack workspace? If you want to ask questions or read tips-and-tricks from time to time on how to use pyam more effectively in your scenario analysis or data-viz work, maybe consider joining our Slack workspace? Read the full release notes... Best regards, Daniel — Dr. Daniel HUPPMANN Research Scholar Coordinator of the Research Theme „Scenario Services & Scientific Software" Energy, Climate, and Environment (ECE) Program email: huppmann@... Twitter: @daniel_huppmann International Institute for Applied Systems Analysis Schlossplatz 1, A-2361 Laxenburg, Austria | www.iiasa.ac.at
|
|
|
|
#release pyam v1.3.0 - a "compute" module for derived indicators
#release
Daniel Huppmann
Dear pyam user community, Happy to announce a new release v1.3 of the pyam package! # Highlight A compute module and new functions to easily derive advanced indicators from any timeseries data in the IAMC format. # Usage of the new module We added a growth_rate() method, which can be called as df.compute.growth_rate({"Primary Energy|Wind": "Growth rate of Wind"})where df is an IamDataFrame, or pyam.timeseries.growth_rate(x) where x is a pd.Series indexed over an (integer) time domain. Read the Docs for more info! # Warning I noticed that the IIASA API does not work with the latest pandas release (v1.4) - I hope that this will be fixed by a pandas patch release, will investigate if the problem persists… # Join the Slack workspace? If you want to ask questions or read tips-and-tricks from time to time on how to use pyam more effectively in your scenario analysis or data-viz work, maybe consider joining our Slack workspace? Read the full release notes... Best regards, Daniel — Dr. Daniel HUPPMANN Research Scholar Coordinator of the Research Theme „Scenario Services & Scientific Software" Energy, Climate, and Environment (ECE) Program email: huppmann@... Twitter: @daniel_huppmann International Institute for Applied Systems Analysis Schlossplatz 1, A-2361 Laxenburg, Austria | www.iiasa.ac.at
|
|
|
|
Re: #question Upload scenarios to the scenario explorer via pyam
#question
mayrhofer@...
Hi Daniel,
thank you very much for the quick answer and the clarification. Best, Lukas
|
|
|
|
Re: #question Upload scenarios to the scenario explorer via pyam
#question
Daniel Huppmann
Thanks for this question, Lukas!
toggle quoted messageShow quoted text
Indeed, pyam currently only supports querying data from any IIASA Scenario Explorer database (also project-internal instances) - tutorial at [1]. We are also working on an improved database backend (i.e., the ixmp package) which will probably provide an option to upload data directly from your workflow to an IIASA database. But this will at the earliest be ready by summer next year. So for the time being, please upload your (ideally pyam-generated) data file via the project-specific instance of an IIASA Scenario Explorer. Best, Daniel
|
|
|
|
#question Upload scenarios to the scenario explorer via pyam
#question
mayrhofer@...
Dear pyam group,
is there any possibility to upload scenarios to the scenario explorer directly via pyam or do I have to upload it manually via the website? I heard that there is supposed to be some function or method within the package but I could only find ways to import data from the scenario explorer that is already on there via pyam.read_iiasa but not the other way around. Thank you very much for your help! All the best, Lukas Mayrhofer
|
|
|
|
#release pyam v1.2.0 - faster package, published manuscript
#release
Daniel Huppmann
Dear pyam user community, happy to announce a new release v1.2 of the pyam package! The new release has some improvements „under the hood“, aiming to make the package faster when reading large data files. We also started a profiler-module, to make it easier to work on further performance increases. Warning: Note that there is a change in the required version of the xlrd package - no worries if you install from pypi or conda, but make sure to install the latest version of xlrd if you are working with an editable installation in a GitHub-cloned folder. Read the full release notes... In other news, the manuscript in Open Research Europe was approved by the reviewers! The manuscript highlights how pyam relates to other tools used in the IAM, macro-energy and energy systems community, the design principles of the package, as well as several recent use cases… https://open-research-europe.ec.europa.eu/articles/1-74 See Figure 1 below of how we see pyam - a useful bridge between highly customized modelling frameworks and general-purpose data analysis and plotting packages.  Best regards, Daniel — Dr. Daniel HUPPMANN Research Scholar Coordinator of the Research Theme „Scenario Services & Scientific Software" Energy, Climate, and Environment (ECE) Program email: huppmann@... Twitter: @daniel_huppmann International Institute for Applied Systems Analysis Schlossplatz 1, A-2361 Laxenburg, Austria | www.iiasa.ac.at
|
|
|
|
#release pyam v1.1.0 - consistency with IPCC AR6 WG1 color palette
#release
Daniel Huppmann
Hello pyam community, To ensure consistency with the just-released IPCC AR6 WG1 report, there is a new release that makes some minor adjustments to the AR6 color palette embedded in pyam - so that you can easily apply the IPCC colors to any figures that you may be working on… Check out the #tips-and-tricks channel in the Slack workspace where I gave an example on how to use the pre-defined colors in your plots based on work by Matthew Gidden! The new release also makes the package compatible with pandas v1.3, and the pie-plot method now takes an explicit ‚colors' argument… Check out the complete v1.1 release notes! Best, Daniel -- Dr. Daniel Huppmann Research Scholar, Energy Program (ENE) International Institute for Applied Systems Analysis (IIASA) Schlossplatz 1, A-2361 Laxenburg, Austria huppmann@... +43(0) 2236 807-572 www.iiasa.ac.at/staff/huppmann
|
|
|
|
Re: timeseries data: calculate differences between two scenarios
#question
m.haller
Hi Daniel,
thanks for the quick reply! That did work. And also thanks for the link to the tutorial notebook. Is there a link to the tutorial notebook in the docs? I did not find one, and I find the notebook very helpful.
I would like to help improve the documentation, but at the moment I have too much work on my desk before the summer holidays. Maybe in September?
Thanks for the great package, it has been really helpful so far!
Best,
Markus
From: forum@pyam.groups.io <forum@pyam.groups.io>
On Behalf Of Daniel Huppmann
Sent: Tuesday, August 3, 2021 2:59 PM To: forum@pyam.groups.io Subject: Re: [pyam] timeseries data: calculate differences between two scenarios #question
Hi Markus,
Apologies, I guess that the tutorial was a bit too demanding for new users. See a more extensive tutorial at https://github.com/danielhuppmann/strommarkttreffen-pyam/blob/main/tutorial-notebook.ipynb
The short answer is that you only have to add `axis="scenario"` to the subtract-function and `a` and `b` are the names of the scenarios. Pyam will automatically calculate the difference for all variables that it finds.
It would be great if you could extend the tutorial notebook so that the next generation of users can find the answer more easily. Happy to assist if you want to take the lead?
Best, Daniel
|
|
|
|
Re: timeseries data: calculate differences between two scenarios
#question
Daniel Huppmann
Hi Markus,
toggle quoted messageShow quoted text
Apologies, I guess that the tutorial was a bit too demanding for new users. See a more extensive tutorial at https://github.com/danielhuppmann/strommarkttreffen-pyam/blob/main/tutorial-notebook.ipynb The short answer is that you only have to add `axis="scenario"` to the subtract-function and `a` and `b` are the names of the scenarios. Pyam will automatically calculate the difference for all variables that it finds. It would be great if you could extend the tutorial notebook so that the next generation of users can find the answer more easily. Happy to assist if you want to take the lead? Best, Daniel
|
|
|
|
timeseries data: calculate differences between two scenarios
#question
m.haller
Hi everybody,
how do I calculate differences between two scenarios , e.g. ``Difference = CO2 emissions of scenario A - CO2 emissions of scenario B`` ? I guess it should work with the ``subtract`` function, but the tutorials only show how to apply it to different variables of the same scenario, not vice versa. Best regards, Markus
|
|
|
|
#release pyam v1.0.0 (and v0.13.0) - and a manuscript in Open Research Europe
#release
Daniel Huppmann
Dear pyam user community, happy to announce a new release v1.0 of the pyam package! To celebrate that pyam is coming of age, we published a manuscript in Open Research Europe, a new open-access journal (well, maybe preprint server is a better description…) highlighting how pyam relates to other tools used in the IAM, macro-energy and energy systems community, the design principles of the package, as well as several recent use cases… https://open-research-europe.ec.europa.eu/articles/1-74 See Figure 1 below of how we see pyam - a useful bridge between highly customized modelling frameworks and general-purpose data analysis and plotting packages.
 Deprecation warning Warning: Release v1.0 removed all functions that were marked as deprecated. If you are worried about breaking your scripts, only update to v0.13 - which was also released yesterday and has all the new features added over the past weeks, and still supports all deprecated features (with the usual warnings). I apologize for any inconvenience! Take a look at the complete release notes on GitHub for a list of all methods and features that were removed - and the new, improved ways for working with pyam! Best regards, Daniel -- Dr. Daniel Huppmann Research Scholar, Energy Program (ENE) International Institute for Applied Systems Analysis (IIASA) Schlossplatz 1, A-2361 Laxenburg, Austria huppmann@... +43(0) 2236 807-572 www.iiasa.ac.at/staff/huppmann _._,_._,_
|
|
|
|
#release pyam v0.12.0 - Algebraic operations directly on IamDataFrame timeseries
#release
Daniel Huppmann
Dear pyam user community, happy to announce a new release v0.12.0 of the pyam package!
Sneak preview A manuscript describing the pyam package was just accepted in Open Research Europe, a new open-access journal by the European Commission to publish Horizon-2020-funded research. And we will celebrate the publication of this manuscript to officially mark pyam as a „mature“ project - so there will be a release v1.0 soon! Warning: we will also use release v1.0 to remove all functions that are currently marked as deprecated. I apologize for any inconvenience! Highlight of release v0.12 Thanks to an initiative by Patrick Jürgens (Fraunhofer ISE) and with some valuable input from core developers (hat tip to Zeb Nicholls), pyam can now perform algebraic operations (add, subtract, multiply, divide) directly on the timeseries data - and it will even handle units correctly for you! The screenshot below will give you a quick flavor of the power of this new feature - then check out this new tutorial for details!  And for power users, there is now also an IamDataFrame.apply() function very similar to the pandas apply() function, so that you can execute custom functions on the timeseries data. More highlights
Best regards, Daniel -- Dr. Daniel Huppmann Research Scholar, Energy Program (ENE) International Institute for Applied Systems Analysis (IIASA) Schlossplatz 1, A-2361 Laxenburg, Austria huppmann@... +43(0) 2236 807-572 www.iiasa.ac.at/staff/huppmann
|
|
|
|
#release pyam v0.11.0 - Connection to UNFCCC Data inventory and under-the-hood performance improvements
#release
Daniel Huppmann
Dear pyam user community, happy to announce a new release v0.11.0 of the pyam package! Highlights of the new release:
API changes PR #507 harmonizes the behavior of the aggregate() and aggregate_region() methods when performing "empty" aggregation, i.e., no components exist to perform the requested aggregation. In the new implementation, an empty IamDataFrame is returned if append=False (instead of None). PR #488 changes the default behavior when initializing an IamDataFrame from xlsx: now, all sheets names starting with data will be parsed for timeseries data. Best regards, Daniel -- Dr. Daniel Huppmann Research Scholar, Energy Program (ENE) International Institute for Applied Systems Analysis (IIASA) Schlossplatz 1, A-2361 Laxenburg, Austria huppmann@... +43(0) 2236 807-572 www.iiasa.ac.at/staff/huppmann
|
|
|
|
#release pyam v0.10.0 - An improved plotting library & out-of-the-box sankey diagrams
#release
Daniel Huppmann
Dear pyam user community, Happy to announce a new release v0.10.0 of the pyam package! Check out the detailed release notes on GitHub. Highlights of the new release: # Add an out-of-the-box Sankey diagram feature Thanks to a contribution by Maik Budzinski, we now have a Sankey feature from any IAMC-style variable hierarchy! Check out the brand-new example in our plotting gallery…  # Refactor the plotting API To improve the user experience and bring pyam behaviour as close as possible to the pandas/matplotlib behavior. There are now three ways to create a plot:
The third approach can use an IamDataFrame or directly a suitable pandas.DataFrame - this may be helpful for related-but-separate projects like open-scm, silicone, OpenEnergyPlatform, Sentinel, etc (@Zeb Nicholls @Robin Lamboll @Ludwig Hülk @Suavyu Ali) because you could use the plotting library without requiring casting the data to an IamDataFrame first… Current behavior will continue to work but raise a DeprecationWarning and will be removed for release 1.0 (hopefully in ~3 months). # Improve the tutorial illustrating how to read model results from a GAMS gdx file Many models use GAMS as their engine for mathematical computation, so we want to make it easy for modelers to get the results into a pyam workflow. Have a look at the tutorial to get your started! After a conference call with developers at GAMS, we refactored the GAMS-to-pyam-tutorial to use their new native gamstransfer module. Best regards, Daniel -- Dr. Daniel Huppmann Research Scholar, Energy Program (ENE) International Institute for Applied Systems Analysis (IIASA) Schlossplatz 1, A-2361 Laxenburg, Austria huppmann@... +43(0) 2236 807-572 www.iiasa.ac.at/staff/huppmann
|
|
|
|
Using pyam out in the wild...
Daniel Huppmann
Dear pyam user community,
The past weeks have seen quite some activity in the pyam GitHub repo, with several new colleagues starting to contribute! A shout-out to Jonas and Maik in particular - this is really motivating to see it used and gaining speed! I believe that we are getting close to a status where we can contemplate a release 1.0 of the package! For that, I would like to spend a few more hours on further improving the documentation and adding a section for „Using pyam in the wild“, ie links to other projects or manuscripts where pyam was used for the analysis or plotting. Please reach out bilaterally if you have something that could be relevant here! All the best, Daniel -- Dr. Daniel Huppmann Research Scholar, Energy Program (ENE) International Institute for Applied Systems Analysis (IIASA) Schlossplatz 1, A-2361 Laxenburg, Austria huppmann@... +43(0) 2236 807-572 www.iiasa.ac.at/staff/huppmann
|
|
|
|
Re: Importing hourly resolved data
#question
Daniel Huppmann
Hi Patrick,
toggle quoted messageShow quoted text
Thanks for reaching out! My hunch is that the filtering does not work as expected because of a mismatch between integer and strings - usually, pyam assumes that all index columns are strings. Try filtering by the following: ``` subannual=list(map(str, range(1, 186+1))) ``` One comment, though: pyam will sort the index and you will therefore see an order like [1, 10, 100, 11, …] in plots. A possible solution is to rename the subannual timesteps to `h 1` or `h0001`. The filtering could then be made using ``` subannual=[f'h{i:4}' for i in range(1, 186+1)] ``` Another alternative is to use a derivative the datetime-convention but without the year info, so use a format 01-01 00:00+01:00. See this description of the openENTRANCE nomenclature for details. The nomenclature package has a feature swap_time_for_subannual to translate between datetime format and two separate year/subannual columns. Best, Daniel
|
|
|
| 1 - 20 of 36 |