Running PTv3 on a real scan, command by command

This is the long version of the AI section on the office study. It assumes you know what a point cloud is and nothing about GPU infrastructure. Copy the commands in order and you will get a label per point; skip a step and the failure mode is listed at the end so you can recognise it.

What segmentation means here. A point cloud is millions of XYZ coordinates with colour. Segmenting it means giving every point a class — wall, floor, ceiling, chair, clutter. The geometric method already on this site fits planes: excellent for walls, floors and ceilings, blind to furniture. PTv3 (Point Transformer V3) is a neural network trained on labelled indoor scans; it knows what a chair looks like, at least in the kind of room it was trained on.

Why your laptop cannot do it. Two dependencies, spconv and flash-attn, ship only as CUDA kernels, and CUDA is NVIDIA-only. On an Apple M2 the packages do not install at all — this is not a slowness problem. So the plan is: prepare a small copy locally, rent one NVIDIA GPU for an hour or two, run inference, come home.

Step 1 — make a small, network-friendly copy (on your laptop). The scan is 2.9 GB of LAS with 87.8 million points. Do not upload that: it takes 20–40 minutes on a home connection and the network cannot read LAS anyway. Voxel-downsample to 2 cm, centre the coordinates, keep RGB, and write a NumPy archive — 1.24 million points in 15 MB.

python3 scripts/prep-ptv3.py
# → .cache/office-ptv3.npz  (15 MB, 1,241,855 points)

Step 2 — rent one GPU. Runpod, one RTX 4090 with 24 GB VRAM, about US$0.9 per hour billed by the second, Jupyter Lab included. Deploy the PyTorch 2.8 + CUDA 12.8 image with 30 GB of container disk. Closing the browser tab does not stop the meter — you must Stop or Terminate the pod.

Step 3 — pin the code to the checkpoint's generation. This is the step that costs people a day. The S3DIS weights were published in 2024; the current main branch has renamed arguments, changed tensor shapes and even the config format. Check out the release that matches the weights instead of patching the newest code:

git clone --recursive https://github.com/Pointcept/Pointcept.git
cd Pointcept
git checkout ad653ee          # v1.5.2, the 2024 release
git submodule update --init --recursive

Step 4 — build the environment, in this order. spconv needs the wheel that matches your CUDA version (cu126 here). torch-scatter and torch-cluster must come from the PyG index built for your exact torch and CUDA tags. open3d needs system OpenGL libraries even though we render nothing with it. pointops is not on PyPI at all — compile it from the repository:

pip install spconv-cu126
pip install torch-scatter torch-cluster -f https://data.pyg.org/whl/torch-2.8.0+cu128.html
pip install addict einops plyfile termcolor timm ftfy regex tqdm h5py pyyaml tensorboard tensorboardx wandb yapf scipy matplotlib huggingface_hub
pip install --ignore-installed blinker open3d
apt-get update && apt-get install -y libsparsehash-dev ninja-build libegl1 libgl1 libglib2.0-0
cd libs/pointops && python setup.py install && cd ../..
python -c "import torch, spconv, pointops; print('environment ok')"

Step 5 — fetch the weights. They are on Hugging Face, which is much faster than the original Google Drive links:

python - <<'PY'
from huggingface_hub import snapshot_download
p = snapshot_download("Pointcept/PointTransformerV3",
                      allow_patterns=["s3dis-semseg-pt-v3m1-0-rpe/**"])
print(p)
PY

Step 6 — run inference. Upload office-ptv3.npz and the inference script, then point the config at the repository's own RPE config, because that is the one that matches these weights:

python ptv3-infer-v2.py \n        --config configs/s3dis/semseg-pt-v3m1-1-rpe.py \n        --weights <WEIGHTS_DIR>/model/model_best.pth \n        --input office-ptv3.npz \n        --out office-ptv3-labels.npy

Inside the script the parts that matter are: colour divided by 127.5 and shifted to the range −1…1; coordinates centred in X and Y with the floor moved to Z = 0; features built by concatenating coordinates and colour (six channels, what the checkpoint expects); and the cloud cut into 8-metre tiles, each passed through the model separately, because attention over 1.24 million points needs far more than 24 GB.

Step 7 — bring the labels home and stop the machine. Download the .npy (about 1.2 MB) through Jupyter, then Terminate the pod. Colouring, tiling decisions, figures and the website all happen back on the laptop.

What the result means. Zero-shot, the S3DIS weights put 46% of this office in “wall”, 0.7% in “floor” and 0.03% in “ceiling”. Nothing is broken — the model is doing what it was trained to do on American single-storey rooms, and this is a two-storey fitted office with glass partitions. That gap is the honest headline, and it is why the site shows a hybrid: planes from geometry, objects from PTv3.

Common mistakes, and what each one looks like. Every line below is an error I actually hit.

1. Cloning main instead of the pinned release. You get `TypeError: PointTransformerV3.__init__() got an unexpected keyword argument 'cls_mode'`, because the 2024 config passes arguments the 2026 class no longer accepts. Fix: `git checkout ad653ee`.

2. Installing a package without --no-deps. Something upgrades torch from 2.8 to a newer major version, and every compiled extension breaks. The symptom is usually `RuntimeError: operator torchvision::nms does not exist` or an import error from spconv. Fix: reinstall `torch==2.8.0` and `torchvision==0.23.0` from the CUDA 12.8 index with `--no-deps`, and rebuild pointops if needed.

3. Uploading the LAS. 2.9 GB, tens of minutes, and the training script cannot read it. Fix: upload the 15 MB .npz made in step 1.

4. A silently truncated upload. Jupyter can stop part-way through a large file and leave a 1 MB fragment that fails with `zipfile.BadZipFile`. Fix: check the size before running, or split the file and reassemble on the pod (`split -b 4000000 office-ptv3.npz part-`).

5. Running the script from the wrong directory. `ModuleNotFoundError: No module named 'pointcept'` happens when Python is given an absolute path to a script outside the repository. Fix: copy the script into the repo root and run it from there.

6. Missing or wrongly shaped tensors. Pointcept's older API wants a 1-D `batch`, an explicit `offset`, and a `feat` key. Miss any of them and you get `AssertionError`, `bincount only supports 1-d`, or a silent mis-read.

7. The wrong preprocessing. If colour stays 0–1, or Z is centred instead of the floor being moved to 0, the planes collapse: floor and ceiling come out near 0% and everything looks like “wall”. This is the failure that looks like a bad model but is actually a bad input.

8. One giant batch. Passing the whole cloud at once gives `CUDA out of memory` with a number in the tens of thousands of GiB, because the attention index overflows. Fix: tile the scene.

9. Leaving the pod running. The most expensive mistake, and the easiest: a forgotten pod costs about US$0.9 for every hour you forget it.

• Pin the code to the checkpoint's generation before writing any code — this caused most of the errors above.
• Match the training preprocessing exactly: colour to −1…1, floor at Z = 0, features = coordinates + colour.
• Tile the scene; never push a whole building through attention in one batch.
• Check file sizes after every upload — truncated transfers look like model failures.
• Stop the pod the moment the labels are downloaded.