OverflowError: serializing a string larger than 4 GiB requires pickle protocol 4 or higher

[okojoalg]

When I was training, I encountered the below error.
I don't know how to give pickle protocol. Can anyone tell me, please?

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 644, in run_train
    self.train_loop.run_training_epoch()
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 564, in run_training_epoch
    self.trainer.run_evaluation(on_epoch=True)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 758, in run_evaluation
    self.evaluation_loop.on_evaluation_end()
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 100, in on_evaluation_end
    self.trainer.call_hook('on_validation_end', *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1101, in call_hook
    trainer_hook(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/trainer/callback_hook.py", line 183, in on_validation_end
    callback.on_validation_end(self, self.lightning_module)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 212, in on_validation_end
    self.save_checkpoint(trainer, pl_module)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 259, in save_checkpoint
    self._save_top_k_checkpoints(trainer, pl_module, monitor_candidates)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 566, in _save_top_k_checkpoints
    self._update_best_and_save(current, epoch, step, trainer, pl_module, metrics)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 608, in _update_best_and_save
    self._save_model(filepath, trainer, pl_module)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 335, in _save_model
    self.save_function(filepath, self.save_weights_only)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/trainer/properties.py", line 327, in save_checkpoint
    self.checkpoint_connector.save_checkpoint(filepath, weights_only)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 408, in save_checkpoint
    atomic_save(checkpoint, filepath)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/utilities/cloud_io.py", line 63, in atomic_save
    torch.save(checkpoint, bytesbuffer)
  File "/usr/local/lib/python3.8/site-packages/torch/serialization.py", line 372, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol)
  File "/usr/local/lib/python3.8/site-packages/torch/serialization.py", line 476, in _save
    pickler.dump(obj)
OverflowError: serializing a string larger than 4 GiB requires pickle protocol 4 or higher

[justusschock]

Hi.
We use torch.save to save the model checkpoints and torch.save internally uses pickle. For a long time pickle was limited to a maximum 4GB file size. But more recent pickle protocols should be able to overcome that issue.
What python version are you using, which pytorch and lightning-versions do you have installed?

[okojoalg]

Hi.
Thank you for your reply.
I am using the below environment:
Python 3.8.7
torch==1.7.1
pytorch-lightning==1.2.1

Regards,

[shizhediao]

Hi, I encountered the same issue and wonder how to solve this issue?

[N-Masi]

You should be able to use np.save instead following this PR. Alternatively, you could pickle the data directly