I also had this issue and found a solution after inspecting the FBX file with Autodesk FBX Converter. The issue is blender's FBX exporter is broken and you need to adjust some setting before you export it. Here are the steps to solve this annoying problem.
1. Apply all transforms
2. Scale your model by 100x
3. Apply all transforms again.
4. Use these settings in the FBX exporter.
Scale: 0.01
Apply Scalings: All local
Forward: Y Forward
Up: Z Up
Uncheck apply unit