Improving Parquet Dedupe on Hugging Face Hub

improving parquet dedupe on the hub is just a nice quality-of-life feature for storage. it saves a few gigabytes, which is nice for cost management, but don't get excited about it as a breakthrough. the real effort goes into ensuring the data you put on the hub is clean and properly versioned in the first place.