torch.mul和*等价（attetion中可以用到）
每行乘上不同元素

  >>> a = torch.ones(3,4)
  >>> a
  tensor([[1., 1., 1., 1.],
          [1., 1., 1., 1.],
          [1., 1., 1., 1.]])
  >>> b = torch.Tensor([1,2,3]).reshape((3,1))
  >>> b
  tensor([[1.],
          [2.],
          [3.]])
  >>> a * b
  tensor([[1., 1., 1., 1.],
          [2., 2., 2., 2.],
          [3., 3., 3., 3.]])
每列乘上不同元素

  >>> a = torch.ones(3,4)
  >>> a
  tensor([[1., 1., 1., 1.],
          [1., 1., 1., 1.],
          [1., 1., 1., 1.]])
  >>> b = torch.Tensor([1,2,3,4]).reshape((1,4))
  >>> b
  tensor([[1., 2., 3., 4.]])
  >>> a*b
  tensor([[1., 2., 3., 4.],
          [1., 2., 3., 4.],
          [1., 2., 3., 4.]])
  >>> torch.mul(a,b)
  tensor([[1., 2., 3., 4.],
          [1., 2., 3., 4.],
          [1., 2., 3., 4.]])
带batch(mul和*会自动broadcaset到所以batch)

  >>> a=torch.ones(2,3,4)
  >>> a
  tensor([[[1., 1., 1., 1.],
           [1., 1., 1., 1.],
           [1., 1., 1., 1.]],

          [[1., 1., 1., 1.],
           [1., 1., 1., 1.],
           [1., 1., 1., 1.]]])
  >>> b = torch.Tensor([1,2,3,4]).reshape((1,4))
  >>> b
  tensor([[1., 2., 3., 4.]])
  >>> a*b
  tensor([[[1., 2., 3., 4.],
           [1., 2., 3., 4.],
           [1., 2., 3., 4.]],

          [[1., 2., 3., 4.],
           [1., 2., 3., 4.],
           [1., 2., 3., 4.]]])
  >>> torch.mul(a,b)
  tensor([[[1., 2., 3., 4.],
           [1., 2., 3., 4.],
           [1., 2., 3., 4.]],

          [[1., 2., 3., 4.],
           [1., 2., 3., 4.],
           [1., 2., 3., 4.]]])
此外还有针对矩阵的乘法如：torch.dot() torch.mm() torch.bmm() torch.dot()是针对一维的向量进行点积。

  In [252]: a=torch.randn(2,3)

  In [253]: b=torch.randn(3,2)

  In [254]: torch.dot(a,b)
  ---------------------------------------------------------------------------
  RuntimeError                              Traceback (most recent call last)
  <ipython-input-254-4939a8ae602a> in <module>()
  ----> 1 torch.dot(a,b)

  RuntimeError: dot: Expected 1-D argument self, but got 2-D

  In [255]: a=torch.randn(3)

  In [256]: b=torch.randn(3)

  In [257]: torch.dot(a,b)
  Out[257]: tensor(0.0967)
torch.mm是针对矩阵的点积(只针对2维)

  In [258]: a=torch.randn(2,3)

  In [259]: b=torch.randn(3,2)

  In [260]: torch.mm(a,b)
  Out[260]: 
  tensor([[-1.2849,  0.1272],
          [ 0.0600, -0.3183]])

  In [261]: torch.mm(a,b).size()
  Out[261]: torch.Size([2, 2])

torch.bmm()是针对一个batch的二维矩阵进行点积(假设batch_size=2)

  In [262]: a=torch.randn(2,2,3)

  In [263]: b=torch.randn(2,3,2)

  In [264]: torch.mm(a,b).size()
  ---------------------------------------------------------------------------
  RuntimeError                              Traceback (most recent call last)
  <ipython-input-264-6bdb1a27d804> in <module>()
  ----> 1 torch.mm(a,b).size()

  RuntimeError: matrices expected, got 3D, 3D tensors at /Users/soumith/code/builder/wheel/pytorch-src/aten/src/TH/generic/THTensorMath.cpp:2065

  In [265]: torch.bmm(a,b).size()
  Out[265]: torch.Size([2, 2, 2])
那么如果维度大于3，我们要针对数据集中单个的矩阵进行点积怎么办呢？比如在多头attention中，batch=2，head=2
torch.matmul只针对最后俩个维度进行点积。

  In [266]: b=torch.randn(2,2,3,2)

  In [267]: a=torch.randn(2,2,2,3)

  In [268]: torch.bmm(a,b).size()
  ---------------------------------------------------------------------------
  RuntimeError                              Traceback (most recent call last)
  <ipython-input-268-3084f0e99edf> in <module>()
  ----> 1 torch.bmm(a,b).size()

  RuntimeError: invalid argument 1: expected 3D tensor, got 4D at /Users/soumith/code/builder/wheel/pytorch-src/aten/src/TH/generic/THTensorMath.cpp:2304

  In [269]: torch.matmul(a,b).size()
  Out[269]: torch.Size([2, 2, 2, 2])

  In [270]: a=torch.randn(2,2,3)

  In [271]: b=torch.randn(2,3,2)

  In [272]: torch.matmul(a,b).size()
  Out[272]: torch.Size([2, 2, 2])
可以matmul同样可以实现bmm