We’ve unfortunately had to delay the release of DeepChem 2.6.0 due to the fact that TensorFlow 2.5.0 doesn’t support Numpy 1.20. As a bit of background, numpy 1.20 was an important release that dropped support for python 3.6. Google uses python 3.6 for their internal CI which has made it hard for them to upgrade to numpy 1.20 (https://github.com/tensorflow/tensorflow/issues/50204). However, the PyTorch ecosystem and the python data science ecosystem more broadly has upgraded to numpy 1.20 which has created a painful dependency management problem.
DeepChem uses both TensorFlow and PyTorch extensively so we faced the challenge of trying to make both ecosystems work with one another. We attempted running TensorFlow models on numpy 1.20 but ran into mysterious errors (see https://stackoverflow.com/questions/58479556/notimplementederror-cannot-convert-a-symbolic-tensor-2nd-target0-to-a-numpy for examples). We attempted to downgrade to numpy 1.19, but found that we broke PyTorch/DGL/Pytorch-Geometric. Large portions of the ecosystem have updated and we couldn’t find a combination of pinned versions that enabled DeepChem to run stably (see https://github.com/deepchem/deepchem/pull/2531 for experiments and discussion). After considerable effort and discussion, we’ve made the call to upgrade to numpy 1.20 even though that will break our CI build on TensorFlow for the foreseeable future since this direction is more future-proof. The DeepChem CI is now broken on HEAD and will continue to be broken until TensorFlow upgrades to numpy 1.20. We are unfreezing DeepChem HEAD for community pull requests and will continue developing so as to unblock DeepChem Google Summer of Code student projects. We are unsure when we can release DeepChem 2.6.0, but will continue tracking the situation and will keep our community posted.
We’ve used TensorFlow in DeepChem for several years now, nearly from the alpha release of TensorFlow itself. TensorFlow has enabled tremendous amounts of innovation and has provided a powerful base for DeepChem to innovate and enable research (most of the papers/discoveries on the DeepChem Papers and Discoveries List ran TensorFlow models). But at the same time, TensorFlow support has become increasingly painful for us. Updating to TensorFlow 2.0 was very challenging and took months of effort. TensorFlow 2.4 broke a number of our models. TensorFlow 2.5.0/numpy 1.20 have now broken the CI and delayed our release.
For these reasons, we are planning to shift towards using PyTorch as our primary development framework. These changes will be gradual, spread over several releases. We don’t plan to remove TensorFlow support but will rather encourage PyTorch usage within DeepChem since we believe it provides a more stable base for our engineering efforts. We may rewrite some of our simpler models to run in PyTorch. If you are a DeepChem user, you should see little to no disruption as a result of this shift since no user facing APIs will change.