[PATCH 0/2] Add python-transformers

  • Open
  • quality assurance status badge
Details
2 participants
  • jgart
  • Nicolas Graves
Owner
unassigned
Submitted by
Nicolas Graves
Severity
normal
Blocked by

Debbugs page

N
N
Nicolas Graves wrote on 7 Sep 09:54 -0700
(address . guix-patches@gnu.org)(address . ngraves@ngraves.fr)
20240907165534.22367-1-ngraves@ngraves.fr
This patch adds the popular python-transformers package.

Nicolas Graves (2):
gnu: python-huggingface-hub: Update to 0.23.2.
gnu: Add python-transformers.

gnu/packages/machine-learning.scm | 60 +++++++++++++++++++++++++++++++
gnu/packages/python-web.scm | 10 ++++--
2 files changed, 67 insertions(+), 3 deletions(-)

--
2.45.2
N
N
Nicolas Graves wrote on 7 Sep 09:56 -0700
[PATCH 2/2] gnu: Add python-transformers.
(address . 73109@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20240907165703.23229-2-ngraves@ngraves.fr
* gnu/packages/machine-learning.scm (python-transformers): New variable.

Change-Id: Ifd7fa3a0f4611d3298ab76ceb44b3aea1397b824
---
gnu/packages/machine-learning.scm | 60 +++++++++++++++++++++++++++++++
1 file changed, 60 insertions(+)

Toggle diff (73 lines)
diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 412499d424..42842d7d61 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -5846,6 +5846,66 @@ (define-public python-tokenizers
tokenizers, @code{rust-tokenizers}.")
(license license:asl2.0)))
+(define-public python-transformers
+ (package
+ (name "python-transformers")
+ (version "4.44.2")
+ (source
+ (origin
+ (method url-fetch)
+ (uri (pypi-uri "transformers" version))
+ (sha256
+ (base32 "09h84wqqk2bgi4vr9d1m3dsliard99l53n96wic405gfjb61gain"))))
+ (build-system pyproject-build-system)
+ ;; The imported package contains ~60 more inputs, but they don't seem
+ ;; necessary to build a minimal version of the package.
+ (propagated-inputs
+ (list python-filelock
+ python-huggingface-hub
+ python-numpy
+ python-pytorch
+ python-pyyaml
+ python-regex
+ python-requests
+ python-safetensors
+ python-tokenizers
+ python-tqdm
+ tensorflow))
+ (home-page "https://github.com/huggingface/transformers")
+ (synopsis "Machine Learning for PyTorch and TensorFlow")
+ (description
+ "This package provides easy download of thousands of pretrained models to
+perform tasks on different modalities such as text, vision, and audio.
+
+These models can be applied on:
+@itemize
+@item Text, for tasks like text classification, information extraction,
+question answering, summarization, translation, and text generation, in over
+100 languages.
+@item Images, for tasks like image classification, object detection, and
+segmentation.
+@item Audio, for tasks like speech recognition and audio classification.
+@end itemize
+
+Transformer models can also perform tasks on several modalities combined, such
+as table question answering, optical character recognition, information
+extraction from scanned documents, video classification, and visual question
+answering.
+
+This package provides APIs to quickly download and use those pretrained models
+on a given text, fine-tune them on your own datasets and then share them with
+the community on our model hub. At the same time, each python module defining
+an architecture is fully standalone and can be modified to enable quick
+research experiments.
+
+Transformers is backed by the three most popular deep learning libraries —
+Jax, PyTorch and TensorFlow — with a seamless integration between them. It's
+straightforward to train your models with one before loading them for
+inference with the other.
+
+Note: This version doesn't support integration with JAX.")
+ (license license:asl2.0)))
+
(define-public python-hmmlearn
(package
(name "python-hmmlearn")
--
2.45.2
N
N
Nicolas Graves wrote on 7 Sep 09:56 -0700
[PATCH 1/2] gnu: python-huggingface-hub: Update to 0.23.2.
(address . 73109@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20240907165703.23229-1-ngraves@ngraves.fr
* gnu/packages/python-web.scm (python-huggingface-hub): Update to 0.23.2.

Change-Id: I06989e883f54d9f0fc8db002d0a580008f6c6b67
---
gnu/packages/python-web.scm | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)

Toggle diff (37 lines)
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index bb1b912c0c..2a7dc8f8b3 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -143,7 +143,7 @@ (define-module (gnu packages python-web)
(define-public python-huggingface-hub
(package
(name "python-huggingface-hub")
- (version "0.20.3")
+ (version "0.23.2")
(source
(origin
(method git-fetch)
@@ -152,7 +152,7 @@ (define-public python-huggingface-hub
(commit (string-append "v" version))))
(file-name (git-file-name name version))
(sha256
- (base32 "00indl9labvqvm4m0y5jbzl68cgj8i60a6qy498gpnjj2pqk4l6v"))))
+ (base32 "0hygxqcixkc1d9sr47j2km6z0p17aj4k1dzm4cvpddrvhrgqayq5"))))
(build-system pyproject-build-system)
(arguments
(list
@@ -185,7 +185,11 @@ (define-public python-huggingface-hub
"--ignore=tests/test_utils_git_credentials.py"
"--ignore=tests/test_utils_http.py"
"--ignore=tests/test_utils_pagination.py"
- "--ignore=tests/test_webhooks_server.py")
+ "--ignore=tests/test_webhooks_server.py"
+ "-k" (string-append
+ "not test_push_to_hub"
+ " and not test_from_pretrained_model_id_only"
+ " and not test_from_pretrained_model_id_and_revision"))
#:phases
'(modify-phases %standard-phases
(add-before 'check 'pre-check
--
2.45.2
N
N
Nicolas Graves wrote on 7 Sep 10:08 -0700
control message for bug #73109
(address . control@debbugs.gnu.org)
87tterie8a.fsf@ngraves.fr
block 73109 by 73106
quit


--
Best regards,
Nicolas Graves
N
N
Nicolas Graves wrote on 8 Sep 21:05 -0700
control message for bug #73115
(address . control@debbugs.gnu.org)
87r09t5v5q.fsf@ngraves.fr
block 73115 by 73109
quit


--
Best regards,
Nicolas Graves
N
N
Nicolas Graves wrote on 8 Sep 21:05 -0700
control message for bug #73109
(address . control@debbugs.gnu.org)
87o74x5v51.fsf@ngraves.fr
block 73109 by 73069
quit


--
Best regards,
Nicolas Graves
J
Re: [PATCH 0/2] Add python-transformers
(address . 73109@debbugs.gnu.org)(name . Nicolas Graves)(address . ngraves@ngraves.fr)
877cbkt7tq.fsf@dismail.de
Hi,

I applied the huggingface-hub update but the rest no longer applies.

Can you rebase on latest and send a v2 with just python-transformers?

Thanks!

--
all the best,
jgart
N
N
Nicolas Graves wrote on 10 Sep 01:25 -0700
Re: [bug#73109] [PATCH 0/2] Add python-transformers
87y13z7w4u.fsf@ngraves.fr
On 2024-09-10 00:07, jgart via Guix-patches via wrote:

Toggle quote (2 lines)
> Hi,

Hi jgart, thanks!

Toggle quote (6 lines)
>
> I applied the huggingface-hub update but the rest no longer applies.
>
> Can you rebase on latest and send a v2 with just python-transformers?
>

The python-tokenizers [73106] patches are also necessary before merging
python-transformers.

Herman Rimm has already reviewed the python-safetensors patches, these
are the same kind.

I'll resend a patch series for python-transformers once these blocking
patch series will be merged ;)

Toggle quote (2 lines)
> Thanks!

--
Best regards,
Nicolas Graves
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 73109@patchwise.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 73109
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch