AttributeError: Can't pickle local object while loading ontology from local

[maxrosen]

Using Python 3.11.3, with hp.owl in my onto_path, I get

"AttributeError: Can't pickle local object 'BaseSubGraph.parse..do_parse'"

executing

hpo_ontology = hpo_world.get_ontology("http://purl.obolibrary.org/obo/hp.owl").load()

or

hpo_ontology = hpo_world.get_ontology("file:///Users/max/test/hp.owl").load()

If I use

hpo_ontology = hpo_world.get_ontology("http://purl.obolibrary.org/obo/hp.owl").load()

without setting onto_path, I get a 403 error. GitHub or OBO library at the time of writing seems to block requests from urllib requests. But this works:

hpo_ontology = hpo_world.get_ontology("https://github.com/obophenotype/human-phenotype-ontology/releases/download/v2023-04-05/hp.owl").load()

Any suggestions? Thanks.

[rsgoncalves]

What version of Owlready are you using?

In Owlready2 v0.41 and later versions I get the same "AttributeError: Can't pickle local object 'BaseSubGraph.parse.<locals>.do_parse'" while loading a locally stored copy of NCIT (obtained from http://purl.obolibrary.org/obo/ncit/releases/2022-08-19/ncit.owl). Using Python 3.9.

However in Owlready2 v0.40 the same ontology loads without problem for me.

If I specify the ontology URL "http://purl.obolibrary.org/obo/ncit/releases/2022-08-19/ncit.owl" I also get a "urllib.error.HTTPError: HTTP Error 403: Forbidden". GitHub links work for me. OBO Library ones don't.

[maxrosen]

I'm on 0.41. Maybe a problem with the new parallelized parsing?

There is already an issue on the purl.obolibrary.org tracker: https://github.com/OBOFoundry/purl.obolibrary.org/issues/923. EDIT: They seem to have fixed the problem.

[Jiba]

I successfully loaded the NCIT ontology from a local copy, and listed all classes, using Owlready2 0.43, as follows:

from owlready2 import *

onto_path.append("/tmp")

ncit = get_ontology("/tmp/ncit.owl").load()
for i, c in enumerate(default_world.classes()): print(i, c)

[Jiba]

It also works well with the HPO ontology, with Owlready2 0.43, as follows:

from owlready2 import *

onto_path.append("/tmp")

hpo  = get_ontology("http://purl.obolibrary.org/obo/hp.owl").load()


If the problem persists, could you send me the entire error traceback? Owlready does not use Pickle, so I'm quite surprised to see "Can't pickle local..." error.

[maxrosen]

I still get it in 0.43. Thanks for looking into this.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], line 5
      1 from owlready2 import *
      3 onto_path.append("/Users/max/development/ontology/")
----> 5 hpo  = get_ontology("http://purl.obolibrary.org/obo/hp.owl").load()

File /usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/owlready2/namespace.py:1090, in Ontology.load(self, only_local, fileobj, reload, reload_if_newer, url, **args)
   1088   fileobj = open(f, "rb")
   1089   #try:     new_base_iri = self.graph.parse(fileobj, default_base = self._base_iri, **args)
-> 1090   try:     new_base_iri = self.graph.parse(fileobj, default_base = self._orig_base_iri, **args)
   1091   finally: fileobj.close()
   1092 else:

File /usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/owlready2/driver.py:256, in BaseSubGraph.parse(self, f, format, delete_existing_triples, default_base)
    254 if parallel:
    255   queue = multiprocessing.Queue()
--> 256   multiprocessing.Process(target = do_parse).start()
    257   onto_base_iri = self.import_triples_from_queue(queue, getattr(f, "name", ""), delete_existing_triples)
    258 else:

File /usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py:121, in BaseProcess.start(self)
    118 assert not _current_process._config.get('daemon'), \
    119        'daemonic processes are not allowed to have children'
    120 _cleanup()
--> 121 self._popen = self._Popen(self)
    122 self._sentinel = self._popen.sentinel
    123 # Avoid a refcycle if the target function holds an indirect
    124 # reference to the process object (see bpo-30775)

File /usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py:224, in Process._Popen(process_obj)
    222 @staticmethod
    223 def _Popen(process_obj):
--> 224     return _default_context.get_context().Process._Popen(process_obj)

File /usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py:288, in SpawnProcess._Popen(process_obj)
    285 @staticmethod
    286 def _Popen(process_obj):
    287     from .popen_spawn_posix import Popen
--> 288     return Popen(process_obj)

File /usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py:32, in Popen.__init__(self, process_obj)
     30 def __init__(self, process_obj):
     31     self._fds = []
---> 32     super().__init__(process_obj)

File /usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_fork.py:19, in Popen.__init__(self, process_obj)
     17 self.returncode = None
     18 self.finalizer = None
---> 19 self._launch(process_obj)

File /usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py:47, in Popen._launch(self, process_obj)
     45 try:
     46     reduction.dump(prep_data, fp)
---> 47     reduction.dump(process_obj, fp)
     48 finally:
     49     set_spawning_popen(None)

File /usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/reduction.py:60, in dump(obj, file, protocol)
     58 def dump(obj, file, protocol=None):
     59     '''Replacement for pickle.dump() using ForkingPickler.'''
---> 60     ForkingPickler(file, protocol).dump(obj)

AttributeError: Can't pickle local object 'BaseSubGraph.parse..do_parse'

[Jiba]

Hello,

How are you starting your Python program? The traceback does not look like a usual Python traceback.

Could you also please try to run your program directly in a terminal, using the plain "python" (or "python3") command ?

In addition, what system are you using ?

I suspect that you use some kind of integrated development environment that, when forking, try to pickle some data.

Jiba

[maxrosen]

I used a jupyter notebook being run in Visual Studio Code 1.79.2 using Python 3.11 as kernel on macOS 13.4. Plain python gives the same error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/owlready2/namespace.py", line 972, in load
    try:     new_base_iri = self.graph.parse(fileobj, default_base = self._orig_base_iri, **args)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/owlready2/driver.py", line 256, in parse
    multiprocessing.Process(target = do_parse).start()
  File "/usr/local/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'BaseSubGraph.parse.<locals>.do_parse'

Interestingly, this works:

>>> from owlready2 import *
>>> onto_path.append("/Users/max/md/data/ontologies/")
>>> onto = get_ontology("http://purl.obolibrary.org/obo/hp_full.owl").load()

This does not:

>>> from owlready2 import *
>>> onto_path.append("/Users/max/md/data/ontologies/")
>>> onto = get_ontology("http://purl.obolibrary.org/obo/mondo.owl").load()

or

>>> from owlready2 import *
>>> onto_path.append("/Users/max/md/data/ontologies/")
>>> onto = get_ontology("http://purl.obolibrary.org/obo/hp.owl").load()

[Jiba]

Ok, I finally understand the problem: Python can use various methods for creating the auxiliary process, fork or spawn. It uses fork on Unix but spawn on Mac, however, spawn needs to pickles the function and its arguments, with is not easy to perform here.

I fixed the problem in the development version by allowing multiprocess loading only when fork is available.

Jiba

[maxrosen]

Awesome work :) Thanks for looking into this...