Really no idea Pavel as no genetic studies have been done with orchids to this degree because generation of mutants and studying them is difficult considering that t a plant from seedling to flowering size needs atleast 3 years (one PhD tenure!!! against the model plant species 8 weeks LOL). What I can say for sure is that it is much more complex, if you remember my post about the ABC model of flower development on orchids, there are 3 classes of B genes (could be a total of 9 B genes) that regulate the formation of lip, 9 from each parent means there are 18 alleles, so you can think of everything that could happen right hehe. With the frilly lip I think it is more of dosage dependent dominant character rather than incomplete dominance. With the multiple gene single phenotype hypothesis (which is more likely) if the frilly lip was incompletely dominant its character would have washed down completely after a few generations as multiple genes would have been exchanged during crossing overs. But what we observe is that once you have hybridised a plant with digbyana the flowers will always give hints to its ancestry even if digyana was the great great grand parent of the hybrid. Dosage seems more probable, you have two copies of frilly genes so more frilly, one copy replaced by normal non frilly gene so less frilly. But yes incomplete dominance, co-dominance and most importantly epigenetics are all possibilities.