[easy] Invalid call to aoti_torch_tensor_copy_ #123039

desertfire · 2024-03-30T21:08:32Z

Repro:
Comment out

pytorch/test/inductor/test_cuda_cpp_wrapper.py

Line 111 in e203aa9

"test_multi_device_cuda",

, and then run

TORCHINDUCTOR_ABI_COMPATIBLE=1 python test/inductor/test_cuda_cpp_wrapper.py -k test_multi_device_cuda

Error:

Output:
In file included from /data/users/binbao/pytorch/torch/include/torch/csrc/inductor/aoti_runtime/arrayref_tensor.h:3,
                 from /tmp/torchinductor_binbao/iw/ciwj3ypfsejr725rlcittslsy7aeekpezl7xqdln6yx6emhjd4ym.cpp:4:
/tmp/torchinductor_binbao/iw/ciwj3ypfsejr725rlcittslsy7aeekpezl7xqdln6yx6emhjd4ym.cpp: In function ‘void inductor_entry_impl(AtenTensorOpaque**, AtenTensorOpaque**)’:
/tmp/torchinductor_binbao/iw/ciwj3ypfsejr725rlcittslsy7aeekpezl7xqdln6yx6emhjd4ym.cpp:202:99: error: cannot convert ‘torch::aot_inductor::ArrayRefTensor<float>’ to ‘AtenTensorHandle’ {aka ‘AtenTensorOpaque*’}
  202 |     AOTI_TORCH_ERROR_CODE_CHECK(aoti_torch_tensor_copy_(expensive_copy_to_tensor_if_needed(buf0), buf1));
      |                                                                                                   ^~~~
      |                                                                                                   |
      |                                                                                                   torch::aot_inductor::ArrayRefTensor<float>
/data/users/binbao/pytorch/torch/include/torch/csrc/inductor/aoti_runtime/utils.h:34:8: note: in definition of macro ‘AOTI_TORCH_ERROR_CODE_CHECK’
   34 |   if ((call) != AOTI_TORCH_SUCCESS) {           \
      |        ^~~~
In file included from /data/users/binbao/pytorch/torch/include/torch/csrc/inductor/aoti_torch/generated/c_shim_cuda.h:4,
                 from /tmp/torchinductor_binbao/iw/ciwj3ypfsejr725rlcittslsy7aeekpezl7xqdln6yx6emhjd4ym.cpp:2:
/data/users/binbao/pytorch/torch/include/torch/csrc/inductor/aoti_torch/c/shim.h:382:64: note:   initializing argument 2 of ‘AOTITorchError aoti_torch_tensor_copy_(AtenTensorHandle, AtenTensorHandle)’
  382 | aoti_torch_tensor_copy_(AtenTensorHandle src, AtenTensorHandle dst);
      |                                               ~~~~~~~~~~~~~~~~~^~~
In file included from /data/users/binbao/pytorch/torch/include/torch/csrc/inductor/aoti_runtime/arrayref_tensor.h:3,
                 from /tmp/torchinductor_binbao/iw/ciwj3ypfsejr725rlcittslsy7aeekpezl7xqdln6yx6emhjd4ym.cpp:4:
/tmp/torchinductor_binbao/iw/ciwj3ypfsejr725rlcittslsy7aeekpezl7xqdln6yx6emhjd4ym.cpp:222:99: error: cannot convert ‘torch::aot_inductor::ArrayRefTensor<float>’ to ‘AtenTensorHandle’ {aka ‘AtenTensorOpaque*’}
  222 |     AOTI_TORCH_ERROR_CODE_CHECK(aoti_torch_tensor_copy_(expensive_copy_to_tensor_if_needed(buf4), buf5));
      |                                                                                                   ^~~~
      |                                                                                                   |
      |                                                                                                   torch::aot_inductor::ArrayRefTensor<float>
/data/users/binbao/pytorch/torch/include/torch/csrc/inductor/aoti_runtime/utils.h:34:8: note: in definition of macro ‘AOTI_TORCH_ERROR_CODE_CHECK’
   34 |   if ((call) != AOTI_TORCH_SUCCESS) {           \
      |        ^~~~
In file included from /data/users/binbao/pytorch/torch/include/torch/csrc/inductor/aoti_torch/generated/c_shim_cuda.h:4,
                 from /tmp/torchinductor_binbao/iw/ciwj3ypfsejr725rlcittslsy7aeekpezl7xqdln6yx6emhjd4ym.cpp:2:
/data/users/binbao/pytorch/torch/include/torch/csrc/inductor/aoti_torch/c/shim.h:382:64: note:   initializing argument 2 of ‘AOTITorchError aoti_torch_tensor_copy_(AtenTensorHandle, AtenTensorHandle)’
  382 | aoti_torch_tensor_copy_(AtenTensorHandle src, AtenTensorHandle dst);
      |                                               ~~~~~~~~~~~~~~~~~^~~

The text was updated successfully, but these errors were encountered:

This reverts commit 6e24cc0. Reverted #124037 on behalf of https://github.com/jeanschmidt due to seems to have introduced a regression in pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 3, 5, linux.4xlarge.nvidia.gpu) ([comment](#124037 (comment)))

fixes #123039 In abi mode, ExternKernelSchedulerNode generates code using `aoti_torch_tensor_copy_` which requires `AtenTensorHandle`, but the allocation generates ArrayRefTensor to allocate mem in stack. To fix this issue, this PR prevents ExternKernelSchedulerNode from using stack-mem-allocation in abi, and creates AtenTensorHandle instead. Pull Request resolved: #124037 Approved by: https://github.com/desertfire

This reverts commit f9379eb. Reverted #124037 on behalf of https://github.com/jeanschmidt due to introducing regressions in benchmark, see D56623194 for more details ([comment](#124037 (comment)))

…24037) fixes pytorch#123039 In abi mode, ExternKernelSchedulerNode generates code using `aoti_torch_tensor_copy_` which requires `AtenTensorHandle`, but the allocation generates ArrayRefTensor to allocate mem in stack. To fix this issue, this PR prevents ExternKernelSchedulerNode from using stack-mem-allocation in abi, and creates AtenTensorHandle instead. Pull Request resolved: pytorch#124037 Approved by: https://github.com/desertfire

…ytorch#124037)" This reverts commit f9379eb. Reverted pytorch#124037 on behalf of https://github.com/jeanschmidt due to introducing regressions in benchmark, see D56623194 for more details ([comment](pytorch#124037 (comment)))

…24037) fixes pytorch#123039 In abi mode, ExternKernelSchedulerNode generates code using `aoti_torch_tensor_copy_` which requires `AtenTensorHandle`, but the allocation generates ArrayRefTensor to allocate mem in stack. To fix this issue, this PR prevents ExternKernelSchedulerNode from using stack-mem-allocation in abi, and creates AtenTensorHandle instead. Pull Request resolved: pytorch#124037 Approved by: https://github.com/desertfire

…ytorch#124037)" This reverts commit 6e24cc0. Reverted pytorch#124037 on behalf of https://github.com/jeanschmidt due to seems to have introduced a regression in pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 3, 5, linux.4xlarge.nvidia.gpu) ([comment](pytorch#124037 (comment)))

…24037) fixes pytorch#123039 In abi mode, ExternKernelSchedulerNode generates code using `aoti_torch_tensor_copy_` which requires `AtenTensorHandle`, but the allocation generates ArrayRefTensor to allocate mem in stack. To fix this issue, this PR prevents ExternKernelSchedulerNode from using stack-mem-allocation in abi, and creates AtenTensorHandle instead. Pull Request resolved: pytorch#124037 Approved by: https://github.com/desertfire

…ytorch#124037)" This reverts commit f9379eb. Reverted pytorch#124037 on behalf of https://github.com/jeanschmidt due to introducing regressions in benchmark, see D56623194 for more details ([comment](pytorch#124037 (comment)))

…24037) fixes pytorch#123039 In abi mode, ExternKernelSchedulerNode generates code using `aoti_torch_tensor_copy_` which requires `AtenTensorHandle`, but the allocation generates ArrayRefTensor to allocate mem in stack. To fix this issue, this PR prevents ExternKernelSchedulerNode from using stack-mem-allocation in abi, and creates AtenTensorHandle instead. Pull Request resolved: pytorch#124037 Approved by: https://github.com/desertfire

…ytorch#124037)" This reverts commit 6e24cc0. Reverted pytorch#124037 on behalf of https://github.com/jeanschmidt due to seems to have introduced a regression in pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 3, 5, linux.4xlarge.nvidia.gpu) ([comment](pytorch#124037 (comment)))

fixes #123039 In abi mode, ExternKernelSchedulerNode generates code using `aoti_torch_tensor_copy_` which requires `AtenTensorHandle`, but the allocation generates ArrayRefTensor to allocate mem in stack. To fix this issue, this PR prevents ExternKernelSchedulerNode from using stack-mem-allocation in abi, and creates AtenTensorHandle instead. Pull Request resolved: #124037 Approved by: https://github.com/desertfire

…ytorch#124037)" This reverts commit f9379eb. Reverted pytorch#124037 on behalf of https://github.com/jeanschmidt due to introducing regressions in benchmark, see D56623194 for more details ([comment](pytorch#124037 (comment)))

desertfire mentioned this issue Mar 30, 2024

Cpp-wrapper mode issue tracker #117363

Open

40 tasks

desertfire added module: aotinductor aot inductor triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Mar 30, 2024

desertfire self-assigned this Mar 30, 2024

desertfire changed the title ~~test_multi_device_cuda~~ [easy] test_multi_device_cuda Mar 30, 2024

desertfire removed their assignment Apr 3, 2024

desertfire changed the title ~~[easy] test_multi_device_cuda~~ [easy] Invalid call to aoti_torch_tensor_copy_ Apr 3, 2024

trieuat assigned trieuat and unassigned trieuat Apr 3, 2024

trieuat self-assigned this Apr 14, 2024

trieuat added a commit to trieuat/pytorch that referenced this issue Apr 15, 2024

fix Invalid call to aoti_torch_tensor_copy_ pytorch#123039

0bd5658

trieuat mentioned this issue Apr 15, 2024

fix Invalid call to aoti_torch_tensor_copy_ #123039 #124037

Open

trieuat added a commit to trieuat/pytorch that referenced this issue Apr 15, 2024

fix Invalid call to aoti_torch_tensor_copy_ pytorch#123039

562d1ad

trieuat added a commit to trieuat/pytorch that referenced this issue Apr 15, 2024

fix Invalid call to aoti_torch_tensor_copy_ pytorch#123039

cfe719d

pytorch-bot bot pushed a commit that referenced this issue Apr 21, 2024

fix Invalid call to aoti_torch_tensor_copy_ #123039

0b94d0a

pytorchmergebot closed this as completed in 6e24cc0 Apr 22, 2024

pytorch-bot bot pushed a commit that referenced this issue Apr 24, 2024

fix Invalid call to aoti_torch_tensor_copy_ #123039

e7a7050

desertfire reopened this May 13, 2024

trieuat mentioned this issue May 20, 2024

fix invalid call to aoti_torch_tensor_copy_ #126668

Closed

pytorchmergebot closed this as completed in 082251e May 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[easy] Invalid call to aoti_torch_tensor_copy_ #123039

[easy] Invalid call to aoti_torch_tensor_copy_ #123039

desertfire commented Mar 30, 2024 •

edited

[easy] Invalid call to aoti_torch_tensor_copy_ #123039

[easy] Invalid call to aoti_torch_tensor_copy_ #123039

Comments

desertfire commented Mar 30, 2024 • edited

desertfire commented Mar 30, 2024 •

edited