Iconicity refers to a resemblance relationship between aspects of a form and aspects of its meaning. The perception of iconicity, measured, for instance, by degrees using iconicity ratings, is not objectively defined, however, but dependent on an individual’s language experience. This study investigates the perception of iconicity in the Hong Kong Sign Language (HKSL) lexicon by participants of different linguistic backgrounds: HKSL signers, German Sign Language (DGS) signers and sign-naïve Cantonese non-signers. Fifty-nine HKSL signs from three semantic categories (animal, human and tool) were selected for an iconicity rating task. Overall, HKSL signers rated the degree of iconicity the highest on average, followed by non-signers and DGS signers. Regardless of the differences, certain couplings of semantic category and mapping strategies of signs are preferred by all groups, which is in line with findings of previous studies. Interestingly, when signs are compared based on lexicon type (native vs. non-native), non-native signs (i.e., signs with borrowed elements via loan translation and/or with character signs) are found to have different effects on the overall rating for non-signers and DGS signers. The results further exemplify and support the notion that the perception of iconicity is subjective, mediated by one’s language- and modality-specific experiences with the spoken, signed and written modality.