A textile image classifier may report 95% accuracy and still fail on the classes that matter most. The problem is not necessarily the model; it may be the way performance is being measured.
Computer vision is increasingly used for fabric-defect detection, fibre and weave identification, garment inspection, print recognition and saree classification. Accuracy is usually the first number reported because it is easy to understand: how many test images were classified correctly? Yet textile datasets are rarely perfectly balanced, and several classes may share colours, borders, motifs, yarn effects or surface textures. Under these conditions, one overall percentage can conceal serious weaknesses.
This question extends the earlier My Textile Notes discussion on deep learning for saree texture identification. Once a model has been built, the next challenge is deciding whether its performance is genuinely useful.
Why a High Accuracy Can Be Misleading
Accuracy is the number of correct predictions divided by the total number of predictions. It is valid, but it gives every image equal weight rather than every class equal importance. Imagine a hypothetical textile test set containing 700 images from a dominant catalogue class, 200 ikat images, 80 brocade images and only 20 images from a rare handloom class. Suppose the model correctly identifies 680, 170, 45 and 2 images respectively. Overall accuracy is 897 out of 1,000, or 89.7%.
That sounds respectable. However, recall for the rare class is only 10%, and recall for the 80-image class is about 56%. The model is performing well mainly because the large classes dominate the total. A merchandiser, museum archive or provenance-research project could therefore receive a reassuring accuracy figure while the less common textiles are being missed.
Strong performance on a majority class can hide failure on rare but important textiles.
Read the Confusion Matrix Before Celebrating
A confusion matrix places true classes in rows and predicted classes in columns. The diagonal contains correct predictions; off-diagonal cells show the direction of errors. This is especially valuable in textiles because not all mistakes are equally informative. If two classes are repeatedly confused, the pattern may point to similar motifs, inadequate image views, inconsistent labelling, insufficient samples or a taxonomy that cannot be resolved from appearance alone.
For example, images of two saree traditions may share floral zari, contrast borders or similar catalogue photography even though their weaving structures and provenance differ. The confusion matrix reveals the pairwise problem that accuracy suppresses. It should be examined in both raw counts and row-normalised form: counts show the workload represented by each error, while row normalisation shows what proportion of each true class is being recovered.
Off-diagonal clusters identify the textile pairs that require better data, labels or features.
The Minimum Metric Set
| Measure | What it answers | Why it matters in textiles |
|---|---|---|
| Per-class recall | Of all true images in a class, how many were found? | Exposes classes the model repeatedly misses. |
| Per-class precision | Of all images assigned to a class, how many really belong there? | Shows whether a label is being overused. |
| F1 score | How well are precision and recall balanced? | Useful when false positives and false negatives both matter. |
| Macro-F1 | What is the unweighted average of class-level F1 scores? | Prevents large classes from dominating the summary. |
| Balanced accuracy | What is the average recall across classes? | Makes minority-class weakness visible. |
The official scikit-learn model-evaluation guide defines precision, recall and F-measures and explains multiclass averaging. Its documentation defines balanced accuracy as the average recall obtained on each class. In the hypothetical example above, balanced accuracy is only about 62%, far below the 89.7% overall accuracy. Weighted-F1 should also be treated cautiously because weighting by class support can again allow large classes to dominate.
Prevent Leakage Before Calculating Any Metric
Metrics are trustworthy only when the test set is genuinely independent. Textile datasets often contain several photographs of the same physical saree, garment, fabric roll or defect: full view, pallu, border, close-up and reverse side. If these related images are randomly divided, one view can enter training while another view of the same item enters testing. The model may recognise the product, background or photo session rather than learning a general textile characteristic.
A group-aware split keeps all images from one physical item together. The GroupKFold documentation describes non-overlapping groups, with each group appearing in the test set once across the folds. Depending on the research claim, an appropriate group may be a physical saree, fabric roll, production lot, motif artwork, vendor or photography session. The grouping decision should be recorded in the paper or project report.
A trustworthy pipeline combines group-aware splitting, several metrics and expert review of difficult image pairs.
Match Evaluation to the Real Decision
The best metric depends on how the system will be used. For automatic catalogue tagging, precision may be important because a wrong public label damages trust. For archive retrieval, recall may matter more because missing a relevant textile is costly. For an expert-assistance tool, top-k accuracy can be useful: the correct class may be accepted if it appears among the model’s three most likely suggestions. Top-k performance, however, should complement rather than replace top-1 and class-wise results.
Error cost also varies. Confusing two colourways within one construction is not equivalent to assigning an unsupported provenance or craft identity. Visual appearance alone may not prove fibre composition, weaving technique, authenticity or geographical origin. A textile expert should review recurring error pairs and decide whether better images, microscopic structure, metadata or a revised label hierarchy is required.
This evaluation logic also applies beyond image classification. Readers interested in data-driven textile decisions may connect it with Python modelling for cotton yarn quality, optimising yarn quality through raw-material parameters, and marker efficiency as an optimisation problem. In every case, a model is useful only when its evaluation reflects the actual industrial or research decision.
A Practical Reporting Checklist
- Report the number of images and physical groups in each class.
- Explain how train, validation and test sets were separated.
- Provide accuracy, macro-F1, balanced accuracy and per-class precision and recall.
- Show raw and row-normalised confusion matrices.
- Review the most frequent error pairs with a textile expert.
- Test on images from new products, lots, vendors or capture conditions.
- State what the model cannot establish from an image alone.
Conclusion
Accuracy is a useful starting point, not a complete verdict. Textile image datasets combine class imbalance, fine visual differences, repeated views and domain-specific error costs. A reliable evaluation therefore needs class-wise metrics, a confusion matrix, group-aware splitting and expert interpretation. The most important question is not “How high is the accuracy?” but “Which textiles does the model recognise, which does it confuse, and would those errors be acceptable in the intended application?”
Sources and Acknowledgement
