Date   

#release pyam v1.2.0 - faster package, published manuscript #release

Daniel Huppmann
 

Dear pyam user community, happy to announce a new release v1.2 of the pyam package!

The new release has some improvements „under the hood“, aiming to make the package faster when reading large data files. We also started a profiler-module, to make it easier to work on further performance increases.

Warning: Note that there is a change in the required version of the xlrd package - no worries if you install from pypi or conda, but make sure to install the latest version of xlrd if you are working with an editable installation in a GitHub-cloned folder.

Read the full release notes...

In other news, the manuscript in Open Research Europe was approved by the reviewers! The manuscript highlights how pyam relates to other tools used in the IAM, macro-energy and energy systems community, the design principles of the package, as well as several recent use cases… https://open-research-europe.ec.europa.eu/articles/1-74

See Figure 1 below of how we see pyam - a useful bridge between highly customized modelling frameworks and general-purpose data analysis and plotting packages.



Best regards,
Daniel



Dr. Daniel HUPPMANN
Research Scholar
Coordinator of the Research Theme „Scenario Services & Scientific Software"
Energy, Climate, and Environment (ECE) Program

emailhuppmann@...
webwww.iiasa.ac.at/staff/huppmann

International Institute for Applied Systems Analysis
Schlossplatz 1, A-2361 Laxenburg, Austria | www.iiasa.ac.at


#release pyam v1.1.0 - consistency with IPCC AR6 WG1 color palette #release

Daniel Huppmann
 

Hello pyam community,

To ensure consistency with the just-released IPCC AR6 WG1 report, there is a new release that makes some minor adjustments to the AR6 color palette embedded in pyam - so that you can easily apply the IPCC colors to any figures that you may be working on… Check out the #tips-and-tricks channel in the Slack workspace where I gave an example on how to use the pre-defined colors in your plots based on work by Matthew Gidden!

The new release also makes the package compatible with pandas v1.3, and the pie-plot method now takes an explicit ‚colors' argument…

Check out the complete v1.1 release notes!

Best,
Daniel

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@...
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann


Re: timeseries data: calculate differences between two scenarios #question

m.haller
 

Hi Daniel,

 

thanks for the quick reply! That did work. And also thanks for the link to the tutorial notebook. Is there a link to the tutorial notebook in the docs? I did not find one, and I find the notebook very helpful.

 

I would like to help improve the documentation, but at the moment I have too much work on my desk before the summer holidays. Maybe in September?

 

Thanks for the great package, it has been really helpful so far!

 

Best,

 

Markus

 

From: forum@pyam.groups.io <forum@pyam.groups.io> On Behalf Of Daniel Huppmann
Sent: Tuesday, August 3, 2021 2:59 PM
To: forum@pyam.groups.io
Subject: Re: [pyam] timeseries data: calculate differences between two scenarios #question

 

Hi Markus,

 

Apologies, I guess that the tutorial was a bit too demanding for new users. See a more extensive tutorial at https://github.com/danielhuppmann/strommarkttreffen-pyam/blob/main/tutorial-notebook.ipynb

 

The short answer is that you only have to add `axis="scenario"` to the subtract-function and `a` and `b` are the names of the scenarios. Pyam will automatically calculate the difference for all variables that it finds.

 

It would be great if you could extend the tutorial notebook so that the next generation of users can find the answer more easily. Happy to assist if you want to take the lead?

 

Best,

Daniel



Am 03.08.2021 um 14:33 schrieb m.haller <m.haller@...>:

 

Hi everybody,

how do I calculate differences between two scenarios , e.g.
``Difference = CO2 emissions of scenario A - CO2 emissions of scenario B``
?

I guess it should work with the ``subtract`` function, but the tutorials only show how to apply it to different variables of the same scenario, not vice versa.

Best regards,

Markus

 


Re: timeseries data: calculate differences between two scenarios #question

Daniel Huppmann
 

Hi Markus,

Apologies, I guess that the tutorial was a bit too demanding for new users. See a more extensive tutorial at https://github.com/danielhuppmann/strommarkttreffen-pyam/blob/main/tutorial-notebook.ipynb

The short answer is that you only have to add `axis="scenario"` to the subtract-function and `a` and `b` are the names of the scenarios. Pyam will automatically calculate the difference for all variables that it finds.

It would be great if you could extend the tutorial notebook so that the next generation of users can find the answer more easily. Happy to assist if you want to take the lead?

Best,
Daniel

Am 03.08.2021 um 14:33 schrieb m.haller <m.haller@...>:

Hi everybody,

how do I calculate differences between two scenarios , e.g.
``Difference = CO2 emissions of scenario A - CO2 emissions of scenario B``
?

I guess it should work with the ``subtract`` function, but the tutorials only show how to apply it to different variables of the same scenario, not vice versa.

Best regards,

Markus


timeseries data: calculate differences between two scenarios #question

m.haller
 

Hi everybody,

how do I calculate differences between two scenarios , e.g.
``Difference = CO2 emissions of scenario A - CO2 emissions of scenario B``
?

I guess it should work with the ``subtract`` function, but the tutorials only show how to apply it to different variables of the same scenario, not vice versa.

Best regards,

Markus


#release pyam v1.0.0 (and v0.13.0) - and a manuscript in Open Research Europe #release

Daniel Huppmann
 

Dear pyam user community, happy to announce a new release v1.0 of the pyam package!

To celebrate that pyam is coming of age, we published a manuscript in Open Research Europe, a new open-access journal (well, maybe preprint server is a better description…) highlighting how pyam relates to other tools used in the IAM, macro-energy and energy systems community, the design principles of the package, as well as several recent use cases… https://open-research-europe.ec.europa.eu/articles/1-74

See Figure 1 below of how we see pyam - a useful bridge between highly customized modelling frameworks and general-purpose data analysis and plotting packages.


Deprecation warning

Warning: Release v1.0 removed all functions that were marked as deprecated.

If you are worried about breaking your scripts, only update to v0.13 - which was also released yesterday and has all the new features added over the past weeks, and still supports all deprecated features (with the usual warnings). I apologize for any inconvenience!

Take a look at the complete release notes on GitHub for a list of all methods and features that were removed - and the new, improved ways for working with pyam!

Best regards,
Daniel

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@...
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann
_._,_._,_


#release pyam v0.12.0 - Algebraic operations directly on IamDataFrame timeseries #release

Daniel Huppmann
 

Dear pyam user community, happy to announce a new release v0.12.0 of the pyam package!

Sneak preview

A manuscript describing the pyam package was just accepted in Open Research Europe, a new open-access journal by the European Commission to publish Horizon-2020-funded research. And we will celebrate the publication of this manuscript to officially mark pyam as a „mature“ project - so there will be a release v1.0 soon!

Warning: we will also use release v1.0 to remove all functions that are currently marked as deprecated. I apologize for any inconvenience!

Highlight of release v0.12

Thanks to an initiative by Patrick Jürgens (Fraunhofer ISE) and with some valuable input from core developers (hat tip to Zeb Nicholls), pyam can now perform algebraic operations (add, subtract, multiply, divide) directly on the timeseries data - and it will even handle units correctly for you! 

The screenshot below will give you a quick flavor of the power of this new feature - then check out this new tutorial for details!

PastedGraphic-1.png

And for power users, there is now also an IamDataFrame.apply() function very similar to the pandas apply() function, so that you can execute custom functions on the timeseries data.

More highlights

  • Drop negative weights by default when performing weighted regional aggregation to avoid very „odd“ results (welcome to the developers team, Laura Wienpahl, recently joining the IIASA research software engineers squad).
  • Allow recursive aggregation when (some) intermediate variables exist and perform validation of the existing intermediate variables (again Patrick Jürgens).

More info

Take a look at the complete release notes on GitHub!

Best regards,
Daniel

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@...
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann


#release pyam v0.11.0 - Connection to UNFCCC Data inventory and under-the-hood performance improvements #release

Daniel Huppmann
 

Dear pyam user community, happy to announce a new release v0.11.0 of the pyam package!

Highlights of the new release:

  • Easily order data in the line-plot feature.
  • Add a module for reading data from the UNFCCC Data Inventory via the PIK-PRIMAP unfccc-di-api package.
  • Improved integration with any IIASA Scenario Explorer instance: read non-default versions from the connected database and obtain the "audit" info (scenario upload/edit timestamp and user) - thanks to Falk Benke for suggesting and testing the new feature.
  • Performance improvements when aggregating or concatenating data - thanks to Jonas Hörsch, Thorsten Burandt and Patrick Jürgens.
  • Refactor the entire code base to the Black code style.

API changes

PR #507 harmonizes the behavior of the aggregate() and aggregate_region() methods when performing "empty" aggregation, i.e., no components exist to perform the requested aggregation. In the new implementation, an empty IamDataFrame is returned if append=False (instead of None).

PR #488 changes the default behavior when initializing an IamDataFrame from xlsx: now, all sheets names starting with data will be parsed for timeseries data.

More info

Take a look at the complete release notes on GitHub!

Best regards,
Daniel

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@...
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann


#release pyam v0.10.0 - An improved plotting library & out-of-the-box sankey diagrams #release

Daniel Huppmann
 

Dear pyam user community,

Happy to announce a new release v0.10.0 of the pyam package! Check out the detailed release notes on GitHub.

Highlights of the new release:

Add an out-of-the-box Sankey diagram feature

Thanks to a contribution by Maik Budzinski, we now have a Sankey feature from any IAMC-style variable hierarchy! Check out the brand-new example in our plotting gallery


# Refactor the plotting API

To improve the user experience and bring pyam behaviour as close as possible to the pandas/matplotlib behavior. There are now three ways to create a plot:
  • IamDataFrame.plot.<kind>(**kwargs)
  • IamDataFrame.plot(kind='<kind>', **kwargs)
  • pyam.plotting.<kind>(df, **kwargs)
The third approach can use an IamDataFrame or directly a suitable pandas.DataFrame - this may be helpful for related-but-separate projects like open-scm, silicone, OpenEnergyPlatform, Sentinel, etc (@Zeb Nicholls @Robin Lamboll @Ludwig Hülk @Suavyu Ali) because you could use the plotting library without requiring casting the data to an IamDataFrame first…

Current behavior will continue to work but raise a DeprecationWarning and will be removed for release 1.0 (hopefully in ~3 months).

# Improve the tutorial illustrating how to read model results from a GAMS gdx file

Many models use GAMS as their engine for mathematical computation, so we want to make it easy for modelers to get the results into a pyam workflow. Have a look at the tutorial to get your started! After a conference call with developers at GAMS, we refactored the GAMS-to-pyam-tutorial to use their new native gamstransfer module.

Best regards,
Daniel

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@...
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann


Using pyam out in the wild...

Daniel Huppmann
 

Dear pyam user community,

The past weeks have seen quite some activity in the pyam GitHub repo, with several new colleagues starting to contribute! A shout-out to Jonas and Maik in particular - this is really motivating to see it used and gaining speed!

I believe that we are getting close to a status where we can contemplate a release 1.0 of the package! For that, I would like to spend a few more hours on further improving the documentation and adding a section for „Using pyam in the wild“, ie links to other projects or manuscripts where pyam was used for the analysis or plotting.

Please reach out bilaterally if you have something that could be relevant here!
All the best,
Daniel


--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@iiasa.ac.at
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann


Re: Importing hourly resolved data #question

Daniel Huppmann
 

Hi Patrick,

Thanks for reaching out!

My hunch is that the filtering does not work as expected because of a mismatch between integer and strings - usually, pyam assumes that all index columns are strings. Try filtering by the following:
```
subannual=list(map(str, range(1, 186+1)))
```

One comment, though: pyam will sort the index and you will therefore see an order like [1, 10, 100, 11, …] in plots. A possible solution is to rename the subannual timesteps to `h   1` or `h0001`.

The filtering could then be made using
```
subannual=[f'h{i:4}' for i in range(1, 186+1)]
```

Another alternative is to use a derivative the datetime-convention but without the year info, so use a format 01-01 00:00+01:00. See this description of the openENTRANCE nomenclature for details. The nomenclature package has a feature swap_time_for_subannual to translate between datetime format and two separate year/subannual columns.

Best,
Daniel

Am 23.10.2020 um 10:37 schrieb patrick.juergens@...:

Dear pyam-Group,

I'm having a question about how to best import hourly resolved timeseries, e.g. for energy consumption of different sectors. My goal is to read the data, be able to aggregate each timeseries for one year and be able to filter data for specific hours, e.g. for one week (hours 1 ... 168).

By now I imported the data following the style in the Tutorial "Aggregating subannual timeseries data", i.e. I used a column "subannual" with values 1 ... 8760 for each hour in one year. Aggregating the timeseries to a year value works perfectly as expected with aggregate_time(). However, the filter()-method doesn't support filtering 'subannual'-data like df.filter('subannual'=range(1, 168+1)). In the documentary however I saw that filtering by datetime is possible.

How would I import the timeseries in a way that filtering is possible? Would I need to put the hour-information in a time-column instead of a year and subannual-column? Does aggregate_time then still work?
Otherwise, could you add the possibility to filter by the column 'subannual'?

I hope to hear from you and I would be very pleased if you add a small tutorial about importing hourly resolved timeseries on your Website.

Greetings
Patrick Jürgens


--
Patrick Jürgens, M.Sc.
Energy Systems and Energy Economics
Energy System Analysis
Fraunhofer Institute for Solar Energy Systems ISE
Heidenhofstraße 2, 79110 Freiburg, Germany
Phone +49 761 4588-2292
patrick.juergens@...

Web: www.ise.fraunhofer.de
Blog:
blog.innovation4e.de
Social Media:
Facebook | Twitter | LinkedIn | YouTube

If you would like to remain informed and up-to-date on the latest news,
subscribe to our newsletter and receive all the relevant information about our latest research activities. -> Register here
--


Importing hourly resolved data #question

patrick.juergens@...
 

Dear pyam-Group,

I'm having a question about how to best import hourly resolved timeseries, e.g. for energy consumption of different sectors. My goal is to read the data, be able to aggregate each timeseries for one year and be able to filter data for specific hours, e.g. for one week (hours 1 ... 168).

By now I imported the data following the style in the Tutorial "Aggregating subannual timeseries data", i.e. I used a column "subannual" with values 1 ... 8760 for each hour in one year. Aggregating the timeseries to a year value works perfectly as expected with aggregate_time(). However, the filter()-method doesn't support filtering 'subannual'-data like df.filter('subannual'=range(1, 168+1)). In the documentary however I saw that filtering by datetime is possible.

How would I import the timeseries in a way that filtering is possible? Would I need to put the hour-information in a time-column instead of a year and subannual-column? Does aggregate_time then still work?
Otherwise, could you add the possibility to filter by the column 'subannual'?

I hope to hear from you and I would be very pleased if you add a small tutorial about importing hourly resolved timeseries on your Website.

Greetings
Patrick Jürgens


--
Patrick Jürgens, M.Sc.
Energy Systems and Energy Economics
Energy System Analysis
Fraunhofer Institute for Solar Energy Systems ISE
Heidenhofstraße 2, 79110 Freiburg, Germany
Phone +49 761 4588-2292
patrick.juergens@...

Web: www.ise.fraunhofer.de
Blog:
blog.innovation4e.de
Social Media:
Facebook | Twitter | LinkedIn | YouTube

If you would like to remain informed and up-to-date on the latest news,
subscribe to our newsletter and receive all the relevant information about our latest research activities. -> Register here
--


#release pyam v0.8.0 - Boxplot visualization, World Bank data API, GAMS integration, ... #release

Daniel Huppmann
 

Dear pyam user community,

Happy to announce a new release v0.8.0 of the pyam package! Check out the detailed release notes on GitHub.

Highlights of the new release:

Add an out-of-the-box boxplot visualization feature to the plotting toolbox

Thanks to a push by Edward Byers, we now have a boxplot feature! Check out the brand-new example in our plotting gallery...

# Implement an API to read data from World Bank Open Data Catalogue

When analyzing your scenario results or processing model input data, you want to have easy access to external reference data. Based on work by Maarten Brinkerink during his YSSP research (remote) visit at IIASA, we can now easily pull in data from the World Bank Data Catalogue. Read the docs for more info!

# Write a tutorial illustrating how to read model results from a GAMS gdx file

Many models use GAMS as their engine for mathematical computation, so we want to make it easy for modelers to get the results into a pyam workflow. Have a look at the tutorial to get your started!

# Easier overview of the index dimensions and coordinates of your scenario data

The new version has an  info()  function (which is also used when calling  print(df) ), which now shows a summary of the model/scenario names, variables, units and regions in an IamDataFrame. These dimensions or coordinates are now also directly available via modelscenario and similar attributes, which brings the user experience closer to the behavior one would expect in pandas and xarray (though it’s still different)...

# Refactor the timeseries data backend for improved performance

Last, but not least, and thanks to the heroic efforts by Florian Maczek (who did an internship at IIASA this summer), we have refactored the pyam timeseries data backend to use pandas.Series (instead of a DataFrame). In reports from a few users who have tested the new implementation, this shows an increase in speed and performance by up to 20%!

Best regards,
Daniel

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@...
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann


Refactoring of the pyam data backend

Daniel Huppmann
 

Dear users of the pyam package,

We have been working over the past weeks to refactor the data backend of pyam to improve performance: increase speed and reduce memory usage.

Because this was quite a major operation, it would be great if you could help us in testing if all features continue to work as before - in principle, everything is tested, but it could be that something goes wrong in corner cases.

To help us make pyam better for future users (and yourself), please take 30 minutes to do the following:

1) install the latest release version of pyam (via pypi or condo)
2) dig out any long, complicated script that you have done for scenario analysis or processing with pyam, and add the following
at the top:
```
import time
start_time = time.time()
```
at the end
```
print(time.time() - start_time)
```
3) run the script, save any outputs (excel tables, figures) and note how long the script took (see the print statement).
4) install the latest branch from GitHub
5) re-run the script (don’t forget to restart the kernel in a Jupyter notebook)
6) compare the outputs and the time to step 3
7) please add any comments about unexpected behaviour or errors as a new issue in the GitHub repo: https://github.com/IAMconsortium/pyam/issues/new

Expected timeline: it would be good to have some feedback until the end of this week (Sep 29) so that we continue with a new release in the first week of October.

Thank you so much,
Daniel

PS: shout-out to Florian Maczek who is currently an intern at IIASA and who has implemented a lot of the refactoring!

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@iiasa.ac.at
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann


#release pyam v0.7.0 - improved aggregation & downscaling features #release

Daniel Huppmann
 

Dear pyam user community,

Happy to announce a new release v0.7.0 of the pyam package! Check out the detailed release notes on GitHub.

Highlights of the new release:

# Additional feature for recursive aggregation & downscaling of timeseries data

Thanks to Thorsten Burandt (TU Berlin), the aggregate() function now supports „recursive“ behavior, automatically identifying a variable tree and then performing the aggregation iteratively.

And thanks to Maarten Brinkering (UCC, currently a YSSP at IIASA), the downscale_region() function now accepts an explicit weight-dataframe for disaggregating timeseries data at a regional level to sub-regional components - take a look at the extended tutorial!

# Compatibility with latest version of matplotlib

We updated the plotting library to work with the latest version of matplotlib.

# Reading data from IIASA scenario resources 

Reading timeseries data directly from an instance of the IIASA Scenario Explorer can be helpful to automate your analysis workflows. This release makes this interface more convenient. The tutorial has been rewritten to illustrate the new behavior - take a look here.

# Refactoring our continuous-integration infrastructure

We made some major changes to the continuous-integration infrastructure to ensure a more stable user experience across operating systems and Python versions.

Best regards,
Daniel

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@...
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann
_._,_._,_


Now supporting Python 3.8 and latest matplotlib version

Daniel Huppmann
 

Dear pyam user community,

We implemented a number improvements last week so that pyam is now fully compatible with Python 3.8 and the latest version of matplotlib! This is not yet included in a release (to identify any potential issues that weren’t caught by the tests over the next weeks), but you can check-out the GitHub repository and install locally. Please create an issue if this does not work as expected.

Also, we are in the process of migrating our continuous integration suite from specific tools (travis, appveyor, stickler) to GitHub Actions - this allows to test all operating systems and several Python versions in a more streamlined, efficient manner.

Best regards,
Daniel

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@...
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann



#release pyam v0.6.0 - aggregating data at sub-annual resolution and easier unit conversion (again) #release

Daniel Huppmann
 

Dear pyam user community,

Happy to announce a new release v0.6.0 of the pyam package! Check out the detailed release notes on GitHub.

Highlights:

# Add feature to aggregate timeseries at sub-annual time resolution

We added an aggregate_time() feature: it applies a function like sum (default), average, max, min, ... to data given at a sub-annual time resolution to a yearly or more aggregated sub-annual time slices (e.g., from months to seasons).

# Easier unit conversion (again)

The iam-units utility was turned into an installable package available via both pip and conda making maintenance of both packages pyam and iam-units easier going forward - and allowing you to easily get the latest version of the unit-conversion utility without having to wait for a new pyam release.

If you run into any issues when upgrading pyam, try running the following in a command prompt:

> pip install iam-units

Thanks to Paul Kishimoto (@khaeru) for developing the iam-units package and pushing this really nice utility forward!

# Improved documentation pages (again)

We (again) made an effort to rewrite and extend the documentation pages. Read the new docs!

Best regards,
Daniel

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@...
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann



#release pyam v0.5.0 - easier unit conversion and new tutorials #release

Daniel Huppmann
 

Dear pyam user community,

Happy to announce a new release v0.5.0 of the pyam package! Check out the detailed release notes on GitHub.

Highlights:

# Easier unit conversion

Improved feature for unit conversion using the pint package and the IAMconsortium/units repository, providing out-of-the-box conversion of units commonly used in integrated assessment research, energy systems modelling and related fields!

You will now be able to simply do 
> df.convert_unit('EJ/yr', 'GWh/yr')
without needing to specify the conversion factor (for standard units). See this tutorial for more information!

Thanks to our colleague Paul Kishimoto (@khaeru) for initiating the shared units repository that supports this new feature!

# New tutorial for working with various input data formats

Switching between long and wide table formats and adding relevant columns (like model and scenario name) is a nuisance in the processing of results. Take a look at this new tutorial to see how pyam how streamline your workflow!

# Increased support for timeseries data with continue time formats

The pyam package started as analysis tool for yearly data, but most functions support data using continuous-time formats. Thanks to Robin Lamboll (@Rlamboll) for refactoring the interpolate() function to correctly support both time formats!

# Improved documentation pages

We made an effort to rewrite and extend the documentation pages, in particular for the API docs. Read the new docs!

Best regards,
Daniel

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@...
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann


Improved (and changed?) feature for unit conversion

Daniel Huppmann
 

Dear pyam users,

We are continuously working on extending the features and tools provided by pyam and the number of contributors on GitHub is growing - but not all users are active on the GitHub repository, so let me reach out this way to gather feedback on a suggested API change.

## Current implementation

The current feature for unit conversion takes a dictionary of mappings:

df.convert_unit({'current_unit': ('target_unit', value)})

Advantage: multiple conversions possible with one command
Disadvantage: slightly cumbersome notation with a dictionary pointing to a tuple…

## Suggestion for new API

Playing around a bit, I suggest to change to the following:

df.convert_unit(current_unit, target_unit, factor=None)

where the conversion factor is used if given (similar to the current behaviour) - and we use the Python package pint to find the conversion factor for you for standard unit conversions like EJ to MWh, so that you don’t have to bother finding the right factor and inserting it in your script.

Advantage: easier notation, much better user experience for standard conversions
Disadvantage: the API change might break some already existing code in the future

## What do you think?

If you have any preference one way or another, please join the discussion on  https://github.com/IAMconsortium/pyam/issues/340!

Thank you,
Daniel


Re: #release pyam v0.4.1 - compatibility with pandas v1.0 & frictionless datapackage support #release

Matthew Gidden
 

Congrats Daniel, and thanks so much to Robert and Chris!!


On Tue, Feb 11, 2020 at 7:27 PM Daniel Huppmann <dh@...> wrote:
Dear pyam user community,

Happy to announce a patch-release of the pyam package! A few days ago, the pandas package released v1.0, so we needed to make some minor adaptations in the pyam internals to ensure compatibility. Get the latest version of pyam and you won’t notice a thing...

# openmod & frictionless datapackage

At the openmod workshop in Berlin last month, we organised a „do-a-thon“ on data standards and formats across the integrated-assessment and energy-system modelling communities - jointly chaired by the Horizon 2020 projects openENTRANCESENTINEL and SPINE as well as the Reiner Lemoine Institut (developers of the OpenEnergyPlatform). The slides presenting pyam (and the underlying data model) and ongoing & planned feature development are available here.

And as a direct result from the discussions at the do-a-thon, we added support for the „Frictionless“ datapackage format (input/ouput). Read the docs for more information!

# Acknowledgements

Thanks to Robert Gieseke and Chris Mutel for reviewing (and improving) the datapackage pull request!

Best regards,
Daniel

--

Dr. Daniel Huppmann
Research Scholar, Energy Program (ENE)

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
huppmann@...
+43(0) 2236 807-572
www.iiasa.ac.at/staff/huppmann

1 - 20 of 27