ModuleNotFoundError: No module named 'utils'

[meng25meng]

         When you open an issue on this repository there is a template ([this one](https://github.com/iiasa/message_ix/blob/main/.github/ISSUE_TEMPLATE/bug.md)) that asks you to give certain information—including "the output of `message-ix show-versions`".

ixmp:        3.8.0
message_ix:  3.8.0
message_ix_models: None
message_data: None

click:       8.1.7
dask:        2023.12.1
genno:       installed
graphviz:    None
jpype:       1.5.0
… JVM path:  D:\Anaconda3\envs\message_env\Library\lib\jvm\bin\server\jvm.dll
openpyxl:    3.1.2
pandas:      2.2.1
pint:        0.23
xarray:      2024.2.0
yaml:        6.0.1

iam_units:   installed
jupyter:     installed
matplotlib:  3.8.3
plotnine:    0.13.0
pyam:        1.6.0

GAMS:        45.7.0

python:      3.11.8 | packaged by conda-forge | (main, Feb 16 2024, 20:40:50) [MSC v.1937 64 bit (AMD64)]
python-bits: 64
OS:          Windows
OS-release:  10
machine:     AMD64
processor:   Intel64 Family 6 Model 142 Stepping 11, GenuineIntel
byteorder:   little
LC_ALL:      None
LANG:        None
LOCALE:      ('Chinese (Simplified)_China', '936')

To be very clear, without this information we cannot provide you with assistance.
Please update your issue description to include the items mentioned in the template.

Originally posted by @khaeru in #826 (comment)

I had a lot of nodes, so I used excel to read the data, but my excel expression was different than it is now. Ix_type_mapping is not used to add parameters, but all parameters are listed in the same table. If I want to get back to where I am now, there's going to be a lot of trouble. So, I still want to try the older version before messageix, but it lacks the util.py file when reading data. Can I download and install this tool in the current version?

import utils
ap = utils.add_par(scenario, horizon, vintage_years, firstyear, disp_error)
input_rows = utils.apply_filters(tecs, filters={'Parameter': 'input'})
ap.add_tec_input(input_rows)
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[23], line 1
----> 1 import utils
      2 ap = utils.add_par(scenario, horizon, vintage_years, firstyear, disp_error)

ModuleNotFoundError: No module named 'utils'

This code version should be before 2020.
Anxiously waiting for your reply, hope to get your professional guidance, thank you very much!

[meng25meng]

Hello, I found the source code in messageix. I don’t know if message_ix_models.util is a more advanced util version. Can I directly pip install the message_ix_models.util installation package in the virtual environment? Then can I call the functions in the installation package?

[glatterf42]

So if I understand correctly, you want to use the code snippet you posted here to read in a data file that used to work with a version of message_ix that is at least five years old. As your output of message-ix show-versions shows, you are currently using version 3.8.0, which was released a few months ago, so that explains why you can't import a module called utils -- it simply doesn't exist in this version of the code anymore.

So what can you do?
The best thing you can do is to transform your data file so that it can be read in with the functions that exist in the current code base. We offer tutorials on how to read in Scenario data from excel files. By following the current structure and using the latest code, we will be able to provide you with support and you will benefit from performance and security improvements as well as get more future-proof code.

Alternatively, you could try to make it work using an outdated version of the code. Please note that we generally do not provide support for these versions as their are enough issues in the current code base that we want to address.
You can look through the list of releases here and find one that looks sufficiently old, e.g. v3.0.0. You can then look at the files from that point in time, here's a link for v3.0.0 again. But even if you go to the file utils.py there and study all commits that have been made before v3.0.0 was released, you will find that the file at no point included a function called add_par(). You can do the same thing for message-ix-models and there you might find something like add_par_data(), but again, I would strongly advise you not to use this version of the code and can't tell you much more about where the utils module you want to import comes from.

Consider this alone: all Python versions that were actively supported during mid-2019 have reached end of life by now. We do not support these Python versions anymore, much less any dependency that relies on them. You yourself are using Python 3.11.8, which was released in February 2024! So please update your data to make it usable with the current version, which neither has nor needs a utils module with an add_par() function.

[yiyi1991]

Hi @meng25meng, let me know if you need further support to close this issue. 中文也可以。

Looking forward to seeing you at the MESSAGE capacity-building workshop!

[meng25meng]

Hi @meng25meng, let me know if you need further support to close this issue. 中文也可以。

Looking forward to seeing you at the MESSAGE capacity-building worksh感谢您的回复，我目前正在想用excel形式建立全国31节点的模型，许可证问题已经解决，但是在在excel的去读代码形式那里，以及使用脚本函数来对excel里的参数进行处理，一直没突破，因为节点很多，如果采用教程里给出的excel形式恐怕需要上万行数据，这个工作量有点大，因此我想把Excel格式转换成这种

把历史年份和活动年份这样排列，而不是vin-year act-year，这样明显减少了很多行，一种技术只需要几行就可以表达，同时较少了表格，但这样的表达，在参数传递的函数那里有点麻烦，而不像教程里那样简单了，因此，我在调用函数进行添加各种参数时，还没有突破，想利用这个函数进行添加：import pandas as pd
import os
from itertools import product
path_files = (r'C:\Users\Natalia\GitKraken\Brazil\Brazilian Electrical system models')

os.chdir(path_files)

A function for adding water supply technologies

def add_water(sc, setup_file):

# 1) Loading data from Excel
xls = pd.ExcelFile(setup_file)
setup = xls.parse('water')
demand = xls.parse('water_demand').set_index('time')
inflow = xls.parse('water_inflow').set_index('time')
setup = setup.loc[setup['active'] == 'yes']
setup = setup.set_index('technology')

# 2) Adding required sets for water technologies
sc.check_out()
model_yrs = [int(x) for x in sc.set('year') if int(x) >= sc.firstmodelyear]

# Water technologies
water_tecs = list(set(setup.index))
sc.add_set('technology', water_tecs)

# Water levels
water_lvls = list(set(setup['output_level'].dropna())
                  ) + list(set(setup['input_level'].dropna()))
sc.add_set('level', water_lvls)

# Water commodities
water_coms = list(set(setup['output_commodity'].dropna())
                  ) + list(set(setup['input_commodity'].dropna()))
sc.add_set('commodity', water_coms)

# 2) Adding required parameters for water technologies
for tec in water_tecs:
    node_locs = setup.loc[tec, 'node_loc'].split("/")
    node_dests = setup.loc[tec, 'node_dest'].split("/")

    # adding parameter output
    tec_from = setup.loc[tec, 'tec_from']
    for node1, node2 in zip(node_locs, node_dests):
        df = sc.par('output', {'technology': tec_from, 'node_loc': node1})
        df_new = df.copy()
        df_new['technology'] = tec
        df_new['commodity'] = setup.loc[tec, 'output_commodity']
        df_new['node_dest'] = node2
        df_new['value'] = 1
        df_new['level'] = setup.loc[tec, 'output_level']
        sc.add_par('output', df_new)

    # adding parameter input
    if not pd.isna(setup.loc[tec, 'input_level']):
        node_orgs = setup.loc[tec, 'node_origin'].split(",")
        for node1, node2 in zip(node_locs, node_orgs):
            df = sc.par('input', {'technology': tec_from,
                                  'node_loc': node1})
            df_new = df.copy()
            df_new['technology'] = tec
            df_new['commodity'] = setup.loc[tec, 'input_commodity']
            df_new['node_origin'] = node2
            df_new['value'] = 1
            df_new['level'] = setup.loc[tec, 'input_level']
            sc.add_par('input', df_new)这是一个脚本里的函数，还是不太会用，不知道如何使用函数，如果用这个函数添加各种参数的，该如何编代码呢？还是也需要按照他给出的函数代码，每次都需要上面函数代码？谢谢您的解答，如果可以，我们可以将您列为作者之一，论文将在6月前完成

[meng25meng]

Hi @meng25meng, let me know if you need further support to close this issue. 中文也可以。

Looking forward to seeing you at the MESSAGE capacity-building workshop!

上面是一个脚本里的函数，还是不太会用，不知道如何使用函数，如果用这个函数添加各种参数的，该如何编代码呢？还是也需要按照他给出的函数代码，每次都需要上面函数代码？谢谢您的解答，如果可以，我们可以将您列为作者之一，论文将在6月前完成

[yiyi1991]

It seems you met an error when trying to use .add_par.

sc.add_par('input', df_new)
This line looks fine.

Based on your description, the error seems to be caused by a mismatch between the structure of your df_new and what the parameter input requires.
You can still use sc.par("input") and check the head of it. And make sure that the df_new you feed into it follows the same structure.

根据您的描述，我猜测df_new的数据结构可能与参数input的所需结构不符。
您似乎采用了修改过的数据结构在excel中录入原始数据。
这不是问题。
我建议在脚本里读入excel后使用melt()或者wide_to_long()等功能将它还原成参数需要的数据结构。

[meng25meng]

Based on your description, the error seems to be caused by a mismatch between the structure of your df_new and what the parameter input requires.

就是读取表格不一定非要用教程里给出的格式是吗？原理就是只要把数据读出来，然后定义出来相关技术参数，添加到scenario中，把所需参数添加齐全，最后solve就可以了，不知道我的理解对不对，如果理解相符，我目前就想用上面那种excel形式，这样的话一种技术可以减少数据行的填充，读取代码形式可以在改进，对这个模型学习初学刚一个月，感觉有些东西还没理解透，期待您的回答

[yiyi1991]

sc.add_par('input', df_new)
最终这个df_new的数据结构是参数input所需要的数据结构就可以。

比如，input所需要的数据结构里，列包括'vintage', 'year_all'及其他，不包括'1965'等。
如果中途改变过数据结构，可以使用melt()或者wide_to_long()等变回原来的结构。

[meng25meng]

Hi @meng25meng, let me know if you need further support to close this issue. 中文也可以。

Looking forward to seeing you at the MESSAGE capacity-building workshop!

可以联系上你吗？对于在这个用excel建立中国电力系统模型的时候，如果在资源层面，input和output的作用是什么呢？在每一个年份上需要对input和output输入什么数据呢？我发现他的单位是百分比形式，

[yiyi1991]

input表示此项技术需要的各种commodity投入，output表示此项技术的产出的commodity。
其单位并不必须是百分比。

（图中技术为煤炭资源开采技术，ext_coal_TJ技术的input数值表示TJ地区开采的煤炭资源在全国一次煤炭能源中的占比，output数值表示TJ地区开采的煤炭资源用于二次能源转换的占比。）

请参照技术参数描述。
https://docs.messageix.org/projects/models/en/latest/pkg-data/codelists.html#technologies-technology-yaml

[meng25meng]

用输入表示域技术需要各种商品采用，用输出表示域技术产生的商品。 其单位并不兴奋。

（图中技术为煤炭资源开采技术，ext_coal_TJ技术输入数值表示TJ地区开采的煤炭资源在全国一次煤炭能源中的占比，输出数值表示TJ地区开采的煤炭资源用于二次能源转换的占比。）

请参照技术参数描述。 https://docs.messageix.org/projects/models/en/latest/pkg-data/codelists.html#technologies-technology-yaml

谢谢您的回复，感谢您解答，我最近一直困扰一个问题，我设置的煤电，运行完的结果他的电量一直显示为0，

我一开始以为是没有加开采技术导导致的，但是给一个省份加上煤炭开采技术他的电量还是一直显示为0，下面是我的输入数据，找了很久不知道问题出在哪里？

您能帮我指点一下吗？非常感谢

[yiyi1991]

同质性最终产品（Electricity|coal）可以使用同样的technology名，并在node（region）中设置区域。

关于“煤电的电量一直显示为0” ，从截图上看sub_ppl_*的output为electricity，可以检查electricity是否在commodity set内，以及能源需求侧的技术是否使用electricity作为input。

[meng25meng]

同质最终产品（电力|煤炭）可以使用相同的技术名，并在节点（区域）中设置区域。

关于“煤电的百分比一直显示为0”，从截图上看sub_ppl_*的输出为electricity，可以检查electricity是否在商品集合内，以及能源需求侧的​​数字是否用作输入electricity。

electricity在代码中读取到了，并在商品之中，在需求侧也是electrucity。

[yiyi1991]

从截图上看，已向commdodity添加了新的set（我假设对应的parameter也利用add_par成功添加了？），包含四种commodity。
至少有2种demand在使用electricity这项commodity。

为了排除是否为report功能问题。麻烦确认。
此excel截图是利用message的report功能生成的吗？
这些新添加的technology和commodity可以在output gdx file中找到吗？

[meng25meng]

从截图上看，已向商品集合（我假设相同的参数也利用add_par成功？）添加一项，包含四项商品。 至少有2种需求使用该电力是一项商品。

为了排除是否为report功能问题。麻烦确认。 此excel截图是利用message的report功能生成的吗？ 这些新添加的技术和商品可以在输出gdx文件中找到吗？

在gdx文件中商品也存在的，我的Excel貌似没用最新的report出的

[yiyi1991]

Let's try this first. EPS in the marginal can mean a lot of things.

How is the DEMAND (the variable) look in your output gdx file (I guess the demand shown in your previous screenshot refers to the exogenous demand parameters in the input gdx)? Are they using the desirable commodity at the desirable levels?

[meng25meng]

我们先来试试这个。边际 EPS 可能意味着很多事情。

（变量）在输出 gdx 文件中看起来怎么样DEMAND（我猜你之前的屏幕截图中显示的需求是指输入 gdx 中的外生需求参数）？他们是否在理想的水平上使用理想的商品？

您说的是对的，GDX里的demand是通过excel表格输入进去的，

[yiyi1991]

Can you then check the input and output in your output gdx file by commodity to make sure they are well linked? In your current chain "coal/gas/uranium -> electricity -> demand", it seems at least the last step is well built.

(e.g., do all sub_ppl_* have electricity as their output? Do other gas/nuclear power plants have electricity as their output and do these techs also show zeros in their ACT? Have you limited those commodities/techs to their regions? i.e., tech *_TJ only serving region TJ)

Once you are confident that these are all well built and but still see zero ACT level in coal techs, the next possible checks can be:

• try checking the prices of gas/uranium (namely the other commodities as the substitutes to coal) in PRICE_COMMODITY, are they very low? or are the prices of coal too high?
• try checking EXT to see if your extraction technologies have been successfully added.

[meng25meng]

那么，您能否按商品检查输出 gdx 文件中的input和output，以确保它们连接良好？在您当前的链条“煤炭/天然气/铀 -> 电力 -> 需求”中，似乎至少最后一步已经构建良好。

（例如，所有 sub_ppl_* 的输出是否都是电力？其他天然气/核电站的输出是否也是电力？这些技术的 中是否也显示为零ACT？您是否将这些商品/技术限制在其区域？即技术 *_TJ 仅服务于 TJ 区域）

一旦您确信这些都已构建完毕，但煤炭技术人员的 ACT 水平仍为零，则接下来可以进行的检查可以是：

• 尝试检查一下天然气/铀的价格（即作为煤炭替代品的其他商品）PRICE_COMMODITY，它们的价格是否很低？或者煤炭价格是否太高？
• 尝试检查EXT您的提取技术是否已成功添加。

谢谢您的提醒，都限制在一个区域是在node-loc这个参数吗？每个技术在每一个各地区denode-loc都设定了，关于煤电好像缺少了燃煤价格，只有一个投资成本、运维成本、

[meng25meng]

但是我今天在运行的时候遇到一个报错还没有出现过，
Exception Traceback (most recent call last)
File MsgScenario.java:1277, in at.ac.iiasa.ixmp.objects.MsgScenario.commit()

Exception: Java Exception

The above exception was the direct cause of the following exception:

java.lang.OutOfMemoryError Traceback (most recent call last)
Cell In[48], line 1
----> 1 scenario.solve()

File D:\Anaconda3\envs\message_env\Lib\site-packages\message_ix\core.py:675, in Scenario.solve(self, model, solve_options, **kwargs)
656 def solve(self, model="MESSAGE", solve_options={}, **kwargs):
657 """Solve MESSAGE or MESSAGE-MACRO for the Scenario.
658
659 By default, :meth:ixmp.Scenario.solve is called with 'MESSAGE' as the
(...)
673 :class:.MESSAGE_MACRO class and :class:.GAMSModel.
674 """
--> 675 super().solve(model=model, solve_options=solve_options, **kwargs)

File D:\Anaconda3\envs\message_env\Lib\site-packages\ixmp\core\scenario.py:878, in Scenario.solve(self, model, callback, cb_kwargs, **model_options)
876 # Iterate until convergence
877 while True:
--> 878 model_obj.run(self)
880 # Store an iteration number to help the callback
881 if not hasattr(self, "iteration"):

File D:\Anaconda3\envs\message_env\Lib\site-packages\message_ix\models.py:200, in GAMSModel.run(self, scenario)
190 """Execute the model.
191
192 GAMSModel creates a file named cplex.opt in the model directory containing
(...)
197 produce unexpected results.
198 """
199 # Ensure the data in scenario is consistent with the MESSAGE formulation
--> 200 self.enforce(scenario)
202 # If two runs are kicked off simultaneously with the same self.model_dir, then
203 # they will try to write the same optfile, and may write different contents.
204 #
(...)
207
208 # Write CPLEX options into an options file
209 optfile = Path(self.model_dir).joinpath("cplex.opt")

File D:\Anaconda3\envs\message_env\Lib\site-packages\message_ix\models.py:300, in MESSAGE.enforce(scenario)
297 continue # Contents are as expected; do nothing
299 # Not consistent; empty and then re-populate the set
--> 300 with scenario.transact(f"Enforce consistency of {set_name} and {par_name}"):
301 scenario.remove_set(set_name, existing)
302 scenario.add_set(set_name, expected)
File D:\Anaconda3\envs\message_env\Lib\contextlib.py:144, in _GeneratorContextManager.exit(self, typ, value, traceback)
142 if typ is None:
143 try:
--> 144 next(self.gen)
145 except StopIteration:
146 return False

File D:\Anaconda3\envs\message_env\Lib\site-packages\ixmp\core\timeseries.py:252, in TimeSeries.transact(self, message, condition, discard_on_error)
250 yield
251 finally:
--> 252 maybe_commit(self, condition, message)

File D:\Anaconda3\envs\message_env\Lib\site-packages\ixmp\util_init_.py:286, in maybe_commit(timeseries, condition, message)
283 return False
285 try:
--> 286 timeseries.commit(message)
287 except RuntimeError as exc:
288 log.info(f"maybe_commit() didn't commit: {exc}")

File D:\Anaconda3\envs\message_env\Lib\site-packages\ixmp\core\timeseries.py:203, in TimeSeries.commit(self, comment)
187 def commit(self, comment: str) -> None:
188 """Commit all changed data to the database.
189
190 If the TimeSeries was newly created (with version='new'), :attr:version
(...)
201 util.maybe_commit
202 """
--> 203 self._backend("commit", comment)

File D:\Anaconda3\envs\message_env\Lib\site-packages\ixmp\core\timeseries.py:108, in TimeSeries._backend(self, method, *args, **kwargs)
102 def _backend(self, method, *args, **kwargs):
103 """Convenience for calling method on the backend.
104
105 The weak reference to the Platform object is used, if the Platform is still
106 alive.
107 """
--> 108 return self.platform._backend(self, method, *args, **kwargs)

File D:\Anaconda3\envs\message_env\Lib\site-packages\ixmp\backend\base.py:55, in Backend.call(self, obj, method, *args, **kwargs)
49 def call(self, obj, method, *args, **kwargs):
50 """Call the backend method method for obj.
51
52 The class attribute obj.backend_prefix is used to determine a prefix for the
53 method name, e.g. 'ts{method}'.
54 """
---> 55 return getattr(self, method)(obj, *args, **kwargs)

File D:\Anaconda3\envs\message_env\Lib\site-packages\ixmp\backend\jdbc.py:737, in JDBCBackend.commit(self, ts, comment)
735 def commit(self, ts, comment):
736 try:
--> 737 self.jindex[ts].commit(comment)
738 except java.Exception as e:
739 arg = e.args[0]

java.lang.OutOfMemoryError: java.lang.OutOfMemoryError: Java heap space

[yiyi1991]

您好。我有一个提议。

鉴于这个open issue已经过长，建议考虑使用discussion板块，将其中的问题逐个清楚地描述出来，并发布独立的discussion，便于归类、回答、标明是否已经解决。其他有相似的问题的MESSAGE使用者，也能用相关的英文关键词搜索出来，避免重复提问。（同样的，您也可以在discussion中搜索以往的相关提问。）

MESSAGE开发团队人员有限，以志愿的形式回答问题，使用discussion板块的话，其他使用者也可能回答。您现在使用的是issue板块，一般用于针对MESSAGE_ix代码测试的讨论，具体应用问题并不在其范畴内。

您在discussion中提问后，可以@yiyi1991。我仍然能收到您的问题提醒。

如果这个建议可行的话，请您新建discussion并关闭此issue。